A 15-second ad made only with local AI, with Strata (Qwen) reviewing the video and fixing it itself
Using only the two PCs on the LAN, we made an ad for a fictional ketchup in about 25 minutes. Strata handles planning, review, fix plans and choosing takes; Qwen-Image, MiniMax-H3 and VOICEVOX make the images, video and voice. Without any checklist Qwen found the big problems in the finished video (an empty bottle, a different boy, hands that were not in the picture), fixed them with shared reference pictures, and finally assembled the best take of each cut. Setup, tools, roles and results are on Video generation with Strata.
Moved to Strata 0.2.15: long prompts are read about 7% faster, with the same decode speed and answers
Since 0.2.14, Strata reads the prompt with its own kernels instead of code taken from llama.cpp, including the NVFP4 experts (0.2.15 changes only the app side; its engine is the same as 0.2.14's). We built 0.2.15 from source and compared it with 0.2.13 on our daily setup (one RTX 5060 Ti 16 GB, Qwen3.8-Flash-Next NVFP4), starting each version twice, alternately.
| Version | 31.1K tokens | 26.2K tokens | ~5K tokens | ~0.5K tokens | Decode |
|---|---|---|---|---|---|
| Strata 0.2.13 | 33.6 / 34.7 s | 33.1 / 33.3 s | 10.6 / 10.8 s | 4.2 / 4.3 s | 34.2 / 34.0 tok/s |
| Strata 0.2.15 | 31.5 / 31.5 s | 30.8 / 30.7 s | 10.3 / 10.4 s | 4.0 / 4.0 s | (28.4) / 33.8 tok/s |
Time to the first token, first / second engine start. Decode is the mean of 4 summaries of 6,500 characters. Temperature 0, no thinking.
- Long prompts came about 2.5 s sooner, by the same margin in both starts. The prompt is read in larger pieces (12,544 tokens instead of 10,496), and there are a few more expert slots. Start-up went from 45-60 s to 45 s.
- Decode speed did not change. The (28.4) in 0.2.15's first start comes from its last request overlapping another heavy job on the machine; the second start matched 0.2.13.
- RAG over tables (16 questions) gave the same answers. 15 were answered correctly from nearly the same evidence. One was lost because the answer came back without its JSON wrapper; asking that question 5 more times on each version, it happened once on 0.2.13 and once on 0.2.15, so it is not a version difference.
- Scope. One machine, one GPU, one model (NVFP4), two starts per version. Two-card setups and DeepSeek were not measured (they stay on 0.2.13).
The drone tests have moved to “Strata for physical AI”
The 3-run comparison with llama.cpp (Strata: 305 s on average, all 3 finished), reading images on the GPU, control through ROS 2, and the comparison with an embodied VLM (Cosmos-Reason2-8B) are on Strata for physical AI.
Against the latest llama.cpp, with the same model and the same GGUF, Strata is still faster: about 2–4× on prompt reads, about 1.7× on decode
On the i7-7800X rig with a single RTX 5060 Ti 16 GB, we ran Qwen3.8-Flash-Next on Strata 0.2.13 and on llama.cpp b12387 (released 11/14, the CUDA release build). llama.cpp read the UD-Q4_K_XL GGUF directly, and Strata was measured on a pack of the same GGUF. Our everyday NVFP4 setup is shown as well. llama.cpp used its default automatic placement (--fit on), with a 65,536-token context and flash attention. Every setup was measured the same way: the time to the first token, and the decode speed over four summaries of about 4K tokens.
| Engine and setup | 31.1K tokens | 26.2K tokens | About 4K tokens | Decode |
|---|---|---|---|---|
| Strata 0.2.13, NVFP4 (everyday setup) | 33.8 s | 33.2 s | 10.5 s | 31.9 tok/s |
| Strata 0.2.13, UD-Q4_K_XL | 40.9 s | 36.6 s | 11.9 s | 29.8 tok/s |
| llama.cpp b12387, UD-Q4_K_XL, default | 543.8 s | 388.9 s | 59.6 s | 17.3 tok/s |
llama.cpp b12387, UD-Q4_K_XL, -ub 2048 | 161.4 s | 131.6 s | 22.4 s | 17.5 tok/s |
llama.cpp b12387, UD-Q4_K_XL, --load-mode none | 429.7 s | 363.1 s | 55.2 s | 16.6 tok/s |
One start per setup. The 4K and decode columns are averages of four runs. Temperature 0, no thinking.
- Strata stays faster on the same GGUF. Against llama.cpp's fastest setup here (
-ub 2048), Strata read a 31K prompt about 4× faster and a 4K prompt about 1.9× faster, and decoded about 1.7× faster. - With llama.cpp's defaults, long prompts were very slow to read. About 70 tokens per second, so a 31K prompt took 9 minutes. The default batch is small, and we think each batch waits on the experts computed on the CPU. A larger batch (
-ub 2048) made reads about 3× faster, but did not change decode speed. --load-mode nonebarely helped. At startup llama.cpp warns that CPU tensor overrides with mmap are slow, but turning mmap off made reads only about 10% faster and left decode unchanged (we did not measure it together with-ub 2048).- Within Strata, NVFP4 stays a little faster than UD. This matches the direction of the two-GPU results from 11/05.
- Scope. One machine, one GPU, one model, one start per setup. Strata decodes with MTP speculative drafting; llama.cpp ran without drafting. llama.cpp's thread count was left at its default, and we did not hand-tune how many layers keep their experts on the CPU. Answer quality was not compared.
NVFP4 experts make long prompts about 14% and decode about 30% faster, on an older RTX 3060 paired with a Blackwell RTX 5060 Ti
Same i7-7800X rig (RTX 3060 Gen3 x16 + RTX 5060 Ti Gen3 x8 layer split, Strata 0.2.10, every expert in RAM). We swapped Qwen3.8-Flash-Next's experts from UD-Q4_K_XL to NVFP4, NVIDIA's 4-bit format, in the GPTQ pack made for Strata. The 3060 has no FP4 math units. Strata's default (w4a8) multiplies the NVFP4 weights on the 8-bit units, so both cards run the same math. The table shows the time to the first token and the decode speed, with the 16,384-token chunk cap and each GPU loading only its own layers' weights.
| Experts, layers on the 3060 | Auto chunk | 31.1K tokens | 26.2K tokens | Decode |
|---|---|---|---|---|
| UD-Q4_K_XL, 24 layers (this morning's best) | 10,752 | 29.0 s | 28.2 s | 29.5–37.1 tok/s |
| NVFP4, 24 layers | 13,824 | 28.9 s | 23.7 s | 40.4–45.0 tok/s |
| NVFP4, 21 layers | 14,848 | 29.8 s | 23.0 s | 41.3–45.6 tok/s |
| NVFP4, 18 layers | 15,872 | 24.8–25.2 s | 23.8–24.4 s | 39.1–44.1 tok/s |
| NVFP4, 16 layers | 16,128 | 26.3 s | 24.9 s | 38.8–41.7 tok/s |
One run each (NVFP4 with 18 layers: two engine starts). Temperature 0, no thinking.
- The gain comes from smaller weights, not faster math. An expert shrinks from 3.15 MB to 2.76 MB (−12%), so the same VRAM holds more of them. In decode, the experts read over PCIe fell to about a third. For long prompts, the freed VRAM gives a larger automatic chunk and so fewer passes. Two passes take 23–25 s and three take 29–30 s, so the pass count sets most of the time.
- The best split moved from 24 layers to 18. With UD, the x8 5060 Ti's receiving was the limit. With NVFP4 it receives less, and compute becomes the limit. The 3060 takes about 1.8x the 5060 Ti's time per layer, so fewer layers on the 3060 balance the two cards. That also frees VRAM on the 3060 for a chunk that reads 31K in two passes. This morning's formula, extended to per-layer expert sizes, picked 18 layers as the fastest (it predicts 1–3 s low).
- We saw no difference in answer quality. The 16 RAG questions scored the same and said the same things. The 10-table image-reading test scored the same (F1 0.87 and 0.87); errors on the hard staircase table stayed within the run-to-run spread. A hard question that turns on interpretation (with thinking, 5 runs each) went wrong for both, in the same way, most of the time. Upstream's measurement puts NVFP4 slightly further from an 8-bit reference than UD (KL 0.070 against 0.056). That gap is smaller than the run-to-run spread of these tasks and did not show.
- We do not use the FP4-math mode (w4a4). Only the Blackwell 5060 Ti can run it, and upstream's table shows the error of one product rising from 0.53% to 8.6%. When asked for it, the engine fell back to w4a8 card by card.
- UD and NVFP4 can also be mixed layer by layer. The engine reads experts whose format differs per layer. We built a mix with UD on the 3060's layers and NVFP4 on the 5060 Ti's, and it ran. On this rig it was not faster (31.1 s at 31K), because it lightened the side that was not the bottleneck. It may be useful for a model where NVFP4 loses quality.
- Our everyday setup now runs NVFP4 with 18 layers on the 3060. Instead of downloading the 51 GB NVFP4 PLE table, it uses the local UD one (NVFP4 changes only the experts). We reported this upstream as an NVFP4 measurement on an older-plus-Blackwell pair (#5279), and added it to #5265 as a case where the best split depends on the expert format.
- Scope. One machine, one model, every expert in RAM, few runs per setting. The quality checks were RAG, table reading and one hard question, not a detailed comparison such as KL.
Two-GPU prompt reads made 27–34% faster: the cause was re-sending experts over the x8 link
The rig is an i7-7800X (125 GiB RAM) with an RTX 3060 (PCIe Gen3 x16) + RTX 5060 Ti (Gen3 x8) layer split, running Strata 0.2.10 and Qwen3.8-Flash-Next UD-Q4_K_XL with every expert in RAM. We used nsys to find out why long prompts read slowly here, then changed the settings to make them faster. The table shows the time to the first token for a 31.1K-token prompt.
| Setting | 31.1K tokens |
|---|---|
| 0.2.10's automatic split (18 layers on the 3060) | 39.8–44.1 s |
24 layers on the 3060, each GPU loading only its own layers' weights (--layer-split 24 --trim-stage-weights) | 29.9 s |
The same + a 16,384-token chunk cap (STRATA_PREFILL_AUTO_MAX=16384) | 29.2 s |
| A prototype automatic split for the prompt read (picks 23 layers on the 3060 by itself) | 32.9 s |
| Both cards reading every layer on alternating chunks (upstream PR #5002) | 44.1 s |
| The CPU computing experts that few tokens route to | 30.8–46.5 s |
One run each; 24 layers on the 3060 was measured three times, at 29.9–30.0 s. The gap between 39.8 s and 44.1 s on the automatic split is the number of chunks: 5 against 6.
- The cause was the time the x8 5060 Ti spends receiving experts. In nsys, one 31K prompt made the 5060 Ti receive 263 GB of experts. Its copies ran 87% of the time and its compute only 36%. Most of the wait was 50,000 gaps of about 0.4 ms, each the time one 3.1 MB expert takes to arrive over x8.
- Every chunk re-sends nearly every expert of every layer. One chunk uses most of a layer's 512 experts, and the GPU's cache did not reduce what was sent. So the read takes about the number of chunks times the slower GPU's time per chunk. One cache slot more or less moved the automatic chunk between 6,400 and 6,144 tokens. That put the 31K prompt in 5 or 6 chunks, a difference of about 4 s.
- Strata's automatic split places the layers by decode speed alone. It ignores a link gap below 4x, so the 1.8x gap here (12.3 against 6.9 GB/s) went unseen. That is how the 5060 Ti on the slower link got 30 layers. With 24 layers on the 3060, the 3060's compute and the 5060 Ti's receiving take about the same time, which is the fastest split.
- We wrote a formula for the prompt read and checked it against the runs. It is: slowest GPU per chunk = layers × max(time to receive 512 experts, compute time). It fits 22 measurements within 1 s on average and predicts 24 layers as the best split. A prototype that places the layers by this formula took the automatic split from 44.1 s to 32.9 s at 31K (−26%). We proposed it upstream as #5265. A similar earlier proposal (#2385) had been closed in the 0.2.0-beta tracker reset.
- The answers did not change. On 16 RAG questions the retrieval was identical, and so was what the answers said (amounts, conditions, conclusions). Our everyday setup now runs 24 layers on the 3060, each GPU loading only its own layers' weights, with the 16,384-token chunk cap. Evening update: it has since moved to NVFP4 with 18 layers on the 3060 (details).
- What did not help. Reading every layer on both cards in turn (upstream PR #5002) cannot copy directly between two GeForce cards here, and the K/V setting it requires slows things down by itself. Having the CPU compute some experts made short, one-chunk prompts 15–18% faster but did nothing for long ones.
- Scope. This is one machine, one model, every expert in RAM, and the chunk-major read. With other links or GPUs, the best number of layers will differ.
Re-measured on the i7-7800X with more RAM: decode speed is set by RAM capacity, now confirmed
We raised the second PC (i7-7800X, AVX-512) to 125 GiB of RAM, moved the Xeon's two GPUs over (RTX 3060 + RTX 5060 Ti layer split), and measured with the same versions and the same config as on the Xeon (setup.py's default). Qwen3.8-Flash-Next (UD-Q4_K_XL) now fits entirely in RAM on the i7 too: experts (71.7 GiB) and the PLE table (26.8 GiB). The 3060 is on PCIe Gen3 x16 and the 5060 Ti on Gen3 x8 (on the Xeon both are Gen3 x8).
| Measure | i7-7800X (125 GiB RAM) | Xeon (252 GB RAM) |
|---|---|---|
| Short-prompt decode (0.2.9, median of 60) | 30.2 tok/s | 27.0 tok/s |
| Short-prompt time to first token (median) | 1.39 s | 1.81 s |
| 31K-token time to first token (0.2.10 default) | 39.2–39.5 s | 44.6–45.0 s |
| 49K-token time to first token (0.2.10 default) | 59.6–60.0 s | 64.4–65.3 s |
| Decode after a long prompt (0.2.9, median) | 33.5 tok/s | 31.9 tok/s |
- RAM capacity was what set decode speed. With 64 GB, the i7 decoded at about half the Xeon's speed. With every expert in RAM, it is 12% faster than the Xeon on the same version, config and GPUs (95% interval 1.10–1.14×, faster on all six prompt types). The Xeon's roughly 2× lead came from RAM capacity, not the CPU.
- Long-prompt waits got shorter only by the first ~3 seconds. Time per token is about the same (about 1.07 ms). We think the 3060's x16 link delivers the first experts sooner, but have not verified it.
- On two GPUs, layer-major did not make long prompts faster on the i7 either (0.2.10). Forcing it made 31K 13% slower and decode after long prompts 10% slower, the same result as on the Xeon. Whether it helps on a machine short of RAM is still untested.
- On one small card, 0.2.10's default prompt path got faster. With one CUDA card and a small expert cache, 0.2.10 by default reads prompts of two or more chunks with the layer-major walk. That walk includes my commits (such as the FP16 path for layers MMQ does not take). On the 3060 alone the walk ran by default and cut the time to first token by 11–15% for 12.8K–31K tokens (31K: 66.8 → 57.4 s). A check that asks for one sentence planted in the middle of a long document was answered correctly every time. A 5060 Ti alone (16 GB) has a larger cache, so it does not meet the condition for this path.
0.2.9 and 0.2.10: on the Xeon E5-2630 v2 machine, two GPUs did not make long prompts faster
We measured 0.2.9 and 0.2.10 on the Xeon (RTX 3060 + RTX 5060 Ti layer split, UD-Q4_K_XL). This time with setup.py's default config: all experts (71.7 GiB) in RAM, and no binding to a NUMA node. The table shows the time to the first token.
| Prompt (tokens) | 0.2.9 default | 0.2.9 layer-major forced | 0.2.10 default |
|---|---|---|---|
| 0.8–4.7K | 10.7–15.5 s | 11.2–16.0 s | 10.5–15.5 s |
| 7.6–10.3K | 19.5–21.7 s | 20.0–23.0 s | 19.8–21.9 s |
| 14.3–19.5K | 26.9–29.8 s | 27.6–32.3 s | 27.2–30.0 s |
| 31.3–31.5K | 44.5–44.7 s | 46.4–46.8 s | 44.6–45.0 s |
| 48.9–49.0K | 64.0–65.0 s | 62.1–62.9 s | 64.4–65.3 s |
3 rounds each, a different prompt every time, temperature 0, no thinking.
- With 0.2.9's defaults, layer-major never actually ran. It is 0.2.9's default for a two-card layer split, but the first long prompt logged "no VRAM for a super-chunk (619 MiB free)" once, and every prompt after that quietly used the usual chunk-major path.
- Forcing it on with more VRAM set aside did not change the wait. Up to 31K it was 0–8% slower, at 49K 3% faster. The experts sent to the GPUs fell to about a third, but the 3060's attention became the limit instead. Decode after long prompts was 13% slower (31.9 → 27.6 tok/s). Adding the K-quant prompt kernels (
STRATA_MMQ_KQ=1) changed nothing. - The usual path is limited by PCIe. While reading a prompt both cards receive at 5–7 GB/s, close to the Gen3 x8 limit: 10–15 s for the first 6,912-token chunk and about 7 s for each further chunk.
- 0.2.10 no longer uses layer-major by default. A layer split now reads prompts chunk-major by default as well (a single card with a small cache has its own rule). A note in the upstream code says a layer split "reads 32K prompts 37–55% sooner chunk-major", which matches this machine. 0.2.10's waits equal 0.2.9's, and short-prompt decode is 5% faster (27.0 → 28.5 tok/s).
- Why the earlier "about half" differs. The 10/24 and 10/31 results came from configs that read part of the experts from the drive (even on the Xeon on 10/31 the "off" arm read 39 GB for 32K). Layer-major cut those reads, so it was faster. With every expert in RAM there is nothing to cut. The tuned 10/31 run (30.9 s at 32K) used a different config (bound to NUMA node 0,
--prefill 4096, 24 / 24 layers) and was not re-measured under the same conditions. - Scope of this result. Only one combination was measured: 2 × Xeon E5-2630 v2 (both cards on PCIe Gen3 x8), RTX 3060 + RTX 5060 Ti, UD-Q4_K_XL, every expert in RAM. Other GPUs, configs that read part of the experts from the drive, and other layer splits were not tried on 0.2.9 or 0.2.10. On machines with faster PCIe or faster GPUs the result may differ, as the upstream note suggests. Added 11/05: the i7-7800X (125 GiB RAM, 3060 on Gen3 x16, 5060 Ti on Gen3 x8) gave the same result on 0.2.10 (details).
No more NUMA binding
- In early versions, one CPU was faster. On 0.1.45 (10/16), running on node 1 alone (the one the GPU is attached to) raised short-prompt decode from 10.5 to 16.2 tok/s (+54%) and cut the time to the first token from 5.0 to 2.4 s. Back then the CPU threads ran as one pool across both sockets and memory was scattered over both, so traffic between the sockets and waiting for slow threads cost time. The official docs also advised keeping RAM on one CPU.
- From 0.2.2 on, using both CPUs is faster. Strata now splits its CPU threads per NUMA node and puts half of the experts in each node's own memory. In re-tests on 11/03–11/04, Qwen (0.2.9) went from 18.81 to 21.99 tok/s (+17%, 95% interval +10% to +29%, 60 samples) and DeepSeek (0.2.10) from 12.96 to 18.71 tok/s on a repeated prompt (+44%, one run). In a partial round with varied prompts, binding made DeepSeek 30–40% slower.
- Bound to node 0, 0.2.10's DeepSeek ran out of memory. It put all 117 GB of the CPU lane's experts on node 0 (126 GiB), so a little more memory use elsewhere stopped it. 0.2.9 split them across the two nodes.
- Unbound, 0.2.10 is not faster than 0.2.9. DeepSeek decodes at 6.95 and 6.85 tok/s (36 prompts each). For Qwen the long-prompt waits are the same, and only short-prompt decode is 5% faster.
- What made Qwen faster still was RAM, not NUMA. The earlier config kept only 38 GiB of the experts in RAM and read the rest from the SSD. With setup.py's default config, which puts every expert in RAM, decode went from 21.99 to 27.03 tok/s (+23%). That config does not place the experts per node.
- Table-reading RAG measured again on 0.2.10, and compared with the models used before Strata, is on the RAG page.
Layer-major is now in Strata
Added 11/04: since 0.2.10, layer-major is not used by default (STRATA_PREFILL_LAYER_MAJOR=1 turns it on). On the Xeon with every expert in RAM it brought no gain on 0.2.9 or 0.2.10 (0.2.9 and 0.2.10). The results in this section come from configs that read part of the experts from the drive.
Layer-major prefill, a faster way to read long prompts, became the default for a layer split over two CUDA cards in Strata 0.2.7 (10/27). The release notes say: "From PR #3976 by @Norhiro, building on @vizdatom's #1670." Eight of my commits (Norihiro Nishimura) were merged into the Strata history and shipped in 0.2.7 (the main commit). They include the two-GPU layer split, even super-chunks and the FP16 path.
Until now, each layer's experts were read again for every 8,192 tokens. Layer-major works through a large piece one layer at a time, so they are read once per piece. vizdatom proposed the method for one GPU (#1670). I extended it to a two-GPU layer split and reported it in #3976. The table shows the time to the first word.
| Prompt | Mine: 3060 + 5060 Ti | Speedup | Official: RTX A4000 × 2 | Speedup |
|---|---|---|---|---|
| 8K | 28.9 s → 18.5 s | 1.6× | 41.8 s → 15.2 s | 2.8× |
| 20K | 54.2 s → 23.5 s | 2.3× | 98.8 s → 17.5 s | 5.6× |
| 32K | 65.0 s → 33.1 s | 2.0× | 141.8 s → 27.7 s | 5.1× |
My numbers compare stock 0.2.4.2 with the same version plus this change (UD-Q4_K_XL, the second run of each size). Drive reads at 32K fell from 75 GB to 35 GB. Correctness did not change: both scored needle 9/9, HumanEval 39 / 38, Japanese 11/12 and tool calls 8/8. Decode speed also matched, at 24.0 and 24.1 tok/s. The official numbers come from a different GPU and model, so they cannot be compared with mine.
0.2.8 measured on my two cards (10/31)
At the maintainer's request, I measured 0.2.8 on the 3060 + 5060 Ti and reported it in #3976. In short: at default settings layer-major does not run on this pair, and with more free VRAM the wait at 20K and 32K is about half. The split was 24 / 24 layers, with UD-Q4_K_XL. “Off” is STRATA_PREFILL_LAYER_MAJOR=0. “Tuned” is --vram-reserve-mib 1700 --vram-reserve-later-mib 1700 --prefill 4096 with STRATA_PREFILL_LM_MARGIN_MIB=96.
| Machine, arm | 8K | 20K | 32K | 32K reads | Decode (median) |
|---|---|---|---|---|---|
| i7-7800X, 62 GB RAM, off | 24.3 s | 42.7 s | 51.3 s | 87 GB | 26.85 tok/s |
| i7-7800X, tuned | 24.2 s | 23.4 s | 29.8 s | 39 GB | 21.8 tok/s |
| 2 × Xeon E5-2630 v2, 251 GB RAM, off | 24.8 s | 49.1 s | 62.7 s | 39 GB | 28.05 tok/s |
| Xeon, tuned | 22.7 s | 24.2 s | 30.9 s | 0.1 GB | 26.9 tok/s |
Times are to the first word, on the second run of each size (i7 two runs each, Xeon three). Decode is the median of 40 answers. Correctness matched between off and tuned on both machines: needle 9/9, HumanEval 39 / 38, Japanese 11/12, tool calls 8/8. The Xeon is an AVX-only CPU with both cards at PCIe Gen3 x8. The AVX build hit no illegal instruction.
On the i7, the tuning cost 19% of decode speed. On the Xeon the drop was 4%, but that is one run per arm with off first, so it may be noise. I have not isolated why more RAM shrinks it. The Xeon's near-zero reads when tuned include data already in the page cache. On the i7, layer-major did not run at default settings or with --layer-split auto in either card order, and 32K took 50–65 s. The Xeon did not run those arms. In #3976, others report it running slower on 3090s at PCIe Gen3 and on two V100s, so the effect depends on the cards and the link. To turn it off, set STRATA_PREFILL_LAYER_MAJOR=0.
Summary and contents
Summary
- A 15-second ad made only with local AI, reviewed and fixed by Strata (Qwen) itself (11/17). Without a checklist it found the big problems in the finished video, aligned people and products with shared reference pictures, and finally assembled the best take of each cut (video generation).
- Flying a simulated drone with an LLM, Strata was also faster and steadier (11/15). Three runs each with the same instruction: 305 s on average with all 3 finished, against 428 s and 2 finished for llama.cpp. Calls with an image went from 31.8 s to 10.6 s once Strata also read images on the GPU, faster than llama.cpp's 22.4 s. Flown through ROS 2 instead, the drone also completed the task. An embodied VLM (Cosmos-Reason2-8B) did not complete it (physical AI).
- Against the latest llama.cpp (b12387), with the same model and the same GGUF, Strata is still faster (11/15). On a single RTX 5060 Ti, compared with llama.cpp's fastest setup here, Strata read a 31K prompt about 4× faster and decoded about 1.7× faster (details).
- NVFP4 experts: long prompts about 14% faster, decode about 30% faster (11/05, evening). On an older 3060 paired with a Blackwell 5060 Ti, both cards run the same 8-bit math. The smaller weights moved the best split from 24 layers to 18, and a 31K prompt went from 29.0 s to 24.8 s. We saw no difference in answer quality (details).
- Two-GPU prompt reads 27–34% faster (11/05). The cause was the x8 5060 Ti receiving the experts again for every chunk. With 24 layers on the 3060 and each GPU loading only its own layers' weights, a 31K prompt went from 40–44 s to 29 s (details).
- On the Xeon E5-2630 v2 machine, two GPUs did not make long prompts faster on 0.2.9 and 0.2.10. This is from an RTX 3060 + RTX 5060 Ti with every expert in RAM (the default config here) only; on 11/05 the i7-7800X gave the same result (details). Under these conditions, 0.2.9's default layer-major never actually ran, and forcing it on left the wait unchanged; PCIe is the limit. 0.2.10 no longer uses layer-major by default. We also stopped binding to a NUMA node (details).
- Layer-major, a faster way to read long prompts, became the default in Strata 0.2.7 (since 0.2.10 it is no longer used by default). Eight commits from this two-GPU work were merged into Strata. On my two cards, with more free VRAM set aside, the wait at 32K was about half (in configs that read part of the experts from the drive; details).
- Strata runs correctly on an old AVX-only Xeon. From 0.1.45 to 0.2.10, the AVX builds never hit a single illegal instruction.
- On the Xeon, the GPU's PCIe slot mattered most. Moving from PCIe Gen2 x4 to Gen3 x8 made prompt processing about 3× faster. 0.2.0 also made decode 6–12% faster.
- 2-bit and 4-bit scored about the same on correctness, and 4-bit runs at about half the speed. On 0.2.0 with one GPU, 2-bit suited this machine. (Compared on 0.2.0 with one GPU. On 11/04, with 0.2.9 / 0.2.10, two GPUs and setup.py's default config, 4-bit decodes short prompts at 27–28.5 tok/s and reads 32K in about 45 s, about as fast as 2-bit on 0.2.0. 2-bit has not been re-measured on the newer versions.)
- RAM capacity sets 4-bit decode speed. On an i7-7800X with 64 GB, the experts do not fit in RAM, and decode ran at about half to 60% of the Xeon's, which holds every expert in one 128 GB node. A faster SSD path makes prompt processing about 1.27× faster but leaves decode unchanged. On 11/05, with the i7 raised to 125 GiB so every expert fits, it decoded 12% faster than the Xeon (details).
- With two GPUs, choosing the layer split by hand makes them faster than one (0.2.2, 3060 + 5060 Ti: with 24 / 24 layers and each card loading only its own layer weights, chat decode was about 1.4–2× one card and the agent-style wait shorter. On the Xeon, which holds every expert in RAM, 32K prompts were also about 1.5× one card. The automatic split helps little. The earlier “2.5–3.5× slower” was wrong. See the two-GPU re-test). All eight setups are ranked in the ranking.
Findings
Correct on an AVX-only CPU
From 0.1.45 to 0.2.10, the AVX builds never hit an illegal instruction (SIGILL). CPU-side math runs on 128-bit AVX1 kernels, and their parity tests pass.
The GPU slot mattered most
Just moving the GPU from a PCIe Gen2 x4 slot to a Gen3 x8 slot made prompt processing about 3× faster and cut the agent-style wait from about 22 s to about 6 s.
Even on old hardware, RAM capacity sets decode speed
The Xeon's DDR3 has half the i7's bandwidth per node. Because every expert fit in RAM, it still decoded about twice as fast as the i7. What matters is whether all experts fit, more than how fast the memory is. Prompt processing, on the other hand, is set by the GPU and its PCIe path. For a large model, get enough RAM for the experts first, then the fastest slot (confirmed on 11/05: with the i7 raised to 125 GiB so every expert fits, it decoded 1.12× as fast as the Xeon on the same version and config; details).
0.2.0 decodes faster
In the same slot, 0.2.0 decoded 6–12% faster than 0.1.45, close to the published figure (+10.2% on an RTX 3060).
2-bit suits this machine
4-bit (UD-Q4_K_XL) runs at about half the speed, while its correctness in these tests was about the same as 2-bit (IQ2_XS). (Compared on 0.2.0 with one GPU. On 11/04, with 0.2.9 / 0.2.10, two GPUs and setup.py's default config, 4-bit decodes short prompts at 27–28.5 tok/s and reads 32K in about 45 s, about as fast as 2-bit on 0.2.0. 2-bit has not been re-measured on the newer versions.)
On the second PC, RAM and the SSD path matter more than the instruction set
On an i7-7800X with AVX-512 (64 GB RAM), the AVX build and the regular build ran at the same speed. 4-bit does not fit in RAM, and most of the decode time goes to waiting for experts from the SSD. A direct x4 SSD did not speed up decode; RAM capacity is what matters (factor analysis).
What sets speed, across two PCs and eight setups
On the Xeon, just moving the GPU from a Gen2 x4 to a Gen3 x8 slot made it about 3× faster. On the i7, the RTX 5060 Ti gave ×1.28 and a direct x4 SSD ×1.27, reaching about 90% of the Xeon.
The Xeon, with every expert in RAM, decoded at 14.3 tok/s; the i7, where they do not fit, at 7–9 tok/s (as of 10/22, 4-bit, one GPU; on 11/04 the Xeon with two GPUs and the default config decodes at 27–28.5 tok/s). On 11/05 the i7, raised to 125 GiB, decoded at 30.2 tok/s under the same conditions, ahead of the Xeon. A new GPU, a faster SSD and AVX-512 barely narrowed the gap.
Speed: slot change and 0.2.0
Model IQ2_XS (2-bit), context 40,960. Medians of three runs of tools/strata_bench.py --long, each started after evicting the model files from memory (cold).
| Metric | 0.1.45, old slot | 0.1.45, new slot | 0.2.0, new slot |
|---|---|---|---|
| Chat decode | 17.1 tok/s | 23.6 tok/s | 25.1 tok/s |
| 4K prompt processing | 178 tok/s | 632 tok/s | 636 tok/s |
| 16K prompt processing | 342 tok/s | 951 tok/s | 987 tok/s |
| 32K prompt processing | 352 tok/s | 973 tok/s | 992 tok/s |
| Agent time to first token, per turn | 21.9 s | 6.17 s | 6.22 s |
| Agent decode | 20.5 tok/s | 27.5 tok/s | 30.9 tok/s |
Strata moves experts that do not fit on the GPU over PCIe. On dual-Xeon boards, a slot shaped like x16 may be wired as Gen2 x4. Check the actual speed and width in the LnkSta line of lspci -vv and use the fastest slot; that mattered more than any software setting.
Correctness
A needle-in-a-haystack test in long contexts, HumanEval problems 0–39 (scored with EvalPlus 0.3.1), 12 Japanese questions and 8 tool-calling tasks.
| Test | 0.1.45 (Oct 16) | 0.2.0 (Oct 20) |
|---|---|---|
| Needle (4K/16K/32K × depth 10/50/90%) | 9/9 | 9/9 |
| HumanEval 0–39 base / plus | 37/40 · 35/40 | 39/40 · 37/40 |
| Japanese, 12 questions | 11/12 | 11/12 |
| Tool calling, 8 tasks | 8/8 | 8/8 |
2-bit (IQ2_XS) vs 4-bit (UD-Q4_K_XL)
Both on 0.2.0, the new slot, context 40,960. The 4-bit experts (77 GB) also fit entirely in RAM, but each expert is about twice as large, so the number held on the GPU drops from 3,813 to 1,168.
| Metric | IQ2_XS (2-bit) | UD-Q4_K_XL (4-bit) |
|---|---|---|
| Chat decode | 25.1 tok/s | 14.3 tok/s |
| 32K prompt processing | 992 tok/s (32 s) | 445 tok/s (72 s) |
| Agent time to first token, per turn | 6.2 s | 14.0 s |
| Agent decode | 30.9 tok/s | 14.0 tok/s |
| Needle | 9/9 | 9/9 |
| HumanEval base / plus | 39/40 · 37/40 | 37/40 · 36/40 |
| Japanese / tools | 11/12 · 8/8 | 11/12 · 8/8 |
On this machine, 4-bit halves the speed, while these tests showed no difference in correctness. At the time (10/21) we chose 2-bit for daily use. As of 11/04, with two GPUs and setup.py's default config, 4-bit decodes at 27–28.5 tok/s, so we now use 4-bit (UD-Q4_K_XL). 2-bit has not been re-measured on the newer versions. The test sets are small, though, so finer quality differences were not measured.
Second PC: Core i7-7800X (AVX-512)
From 2026-10-21 we also measured on a PC with AVX2 and AVX-512. Same RTX 3060 (PCIe Gen3 x8), same Strata 0.2.0-beta (engine 0.1.45.3), with the 4-bit UD-Q4_K_XL model.
- CPU
- Core i7-7800X (6 cores, AVX2, AVX-512)
- RAM
- 64 GB DDR4-2666 (4 channels)
- GPU
- RTX 3060 12 GB
- OS
- Fedora 44
AVX build vs regular (AVX-512) build
We built the same source for old CPUs (STRATA_ISA_FLOOR=avx) and with the regular setting (-march=native) and compared them.
| Metric | AVX build | Regular build (AVX-512) |
|---|---|---|
| Chat decode | 6.6 tok/s | 7.4 tok/s |
| 4K prompt processing | 139 tok/s | 134 tok/s |
| 32K prompt processing | 274 tok/s | 272 tok/s |
| Agent time to first token, per turn | 27.6 s | 28.7 s |
Compared with the Xeon (4-bit)
| Metric | Xeon (252 GB RAM; used one node, about 128 GB, as set up until 10/31) | i7-7800X (64 GB RAM) |
|---|---|---|
| Chat decode | 14.3 tok/s | 7.4 tok/s |
| 4K prompt processing | 293 tok/s | 134 tok/s |
| 32K prompt processing | 445 tok/s (72 s) | 272 tok/s (118 s) |
| Agent time to first token, per turn | 14.0 s | 28.7 s |
The 4-bit experts (77 GB) do not fit in 64 GB of RAM, so only 38 GiB stay in RAM and the remaining ~30 GiB are read from the SSD for every token (--resident-budget-gib 38). Those reads set the speed; the instruction-set difference did not show. Comparing the CPUs themselves would need 2-bit, which fits in RAM on both machines.
RTX 5060 Ti 16 GB alone
On the same PC we swapped only the GPU for an RTX 5060 Ti (Blackwell, sm_120, PCIe Gen3 x8). Everything else stayed the same. The extra VRAM raises the experts held on the GPU from 1,374 to 2,731.
| Metric | RTX 3060 | RTX 5060 Ti |
|---|---|---|
| Chat decode | 7.4 tok/s | 8.9 tok/s |
| 4K prompt processing | 134 tok/s | 159 tok/s |
| 32K prompt processing | 272 tok/s (118 s) | 354 tok/s (91 s) |
| Agent time to first token, per turn | 28.7 s | 24.3 s |
| Agent decode | 11.3 tok/s | 13.9 tok/s |
The new GPU made things 20–30% faster, but still short of the Xeon with every expert in RAM.
The NVMe SSD holding the model (Solidigm P41 Plus) links at x4 itself, but it sits behind an M.2 expansion card (a PLX PEX 8747 switch) that connects to an x1 chipset port. Uncached reads ran at 782 MB/s, under a quarter of the SSD's own speed. Experts that miss RAM are read over this path, so at first we took it to be the main limit on 4-bit speed; the direct x4 test below showed it only affects prompt processing. You can check by walking /sys/bus/pci/devices/*/current_link_width from the SSD up through each bridge. We have not yet tried fixes such as moving the SSD to the motherboard's M.2 slot.
Moving the model to a direct x4 SSD
As a fix that needs no hardware changes, we copied the model to the /home SSD (KIOXIA BG4, Gen3 x4 direct). Uncached reads ran at 1.9 GB/s with one stream and 2.4 GB/s with four, 2.4–3× the original path (about 0.8 GB/s). GPU RTX 5060 Ti, everything else unchanged.
| Metric | x1 path | Direct x4 | For reference: Xeon |
|---|---|---|---|
| Chat decode | 8.9 tok/s | 7.8 tok/s | 14.3 tok/s |
| 4K prompt processing | 159 tok/s | 202 tok/s | 293 tok/s |
| 32K prompt processing | 354 tok/s (91 s) | 448 tok/s (72 s) | 445 tok/s (72 s) |
| Agent time to first token, per turn | 24.3 s | 17.2 s | 14.0 s |
| Agent decode | 13.9 tok/s | 15.0 tok/s | 14.0 tok/s |
Prompt processing got about 1.27× faster at every length and matched the Xeon at 32K. The agent-style wait before a reply starts dropped from 24 s to 17 s. Chat decode, though, stayed around 8 tok/s, still well behind the Xeon. Decode reads experts of about 4 MB a few at a time. The engine log shows each read waits about 3 ms on both x1 and direct x4, so bandwidth is not what limits it. To speed up decode as well, the sure fix is more RAM, enough to hold every expert.
Which factors matter (statistics)
We compared the six setups so far (Xeon; i7 with the AVX and AVX-512 builds; RTX 5060 Ti; direct x4 SSD; two GPUs), pairing setups that differ in one factor. Chat decode changed by ×0.47 from Xeon to i7, ×0.44 when split across two GPUs (this value was wrong; adding one card and two cards with a 24 / 24 split and trim on the Xeon with 0.2.2, two cards gave ×1.33; see the re-test), and ×1.36 with the RTX 5060 Ti. The direct x4 SSD and AVX-512 barely affected decode. The Xeon had its memory bound to one socket (126 GiB), so what mattered was one node with enough RAM for every expert (the setup until 10/31; nothing is bound since 11/04, details). Charts and 95% intervals are on the 4-bit speed factors page (English / 日本語). The eight setups, including those two, ranked by read (prompt processing) and write (decode) speed are in the effective speed ranking.
The GPU's PCIe link differed between test periods, as below. Transfer rates are the best host-to-GPU copy rate measured with CUDA under load.
| Period | RTX 3060 | RTX 5060 Ti |
|---|---|---|
| 3060 alone (10/21–22, all i7 tests) | Gen3 x8, about 6.2 GB/s | — |
| 5060 Ti alone (10/22, x1 path and direct x4) | — | Gen3 x8, 6.9 GB/s |
| Two GPUs (10/22 04:02–05:33) | Gen3 x8, about 6.2 GB/s | Gen2 x8, 3.5 GB/s |
| After reseating (10/22 09:44 on, two-GPU re-test) | Gen3 x16, 12.3 GB/s | Gen3 x8, 6.9 GB/s |
The 5060 Ti fell off the bus once at 03:47 on 10/22 and came back at Gen2 after the reboot, which we noticed only later. The two-GPU result from that time (2.5–3.5× slower) included this degraded link and also engine restarts after the card ran out of VRAM. After reseating, the link came back at Gen3, and we re-tested in both card orders (next note). Reseating also moved the 3060 to an x16 slot, so new 3060 results are not directly comparable with the earlier x8 ones. The 5060 Ti is an x8 card, so it runs at x8 in any slot, and its single-card results stay comparable.
We wrote here earlier that splitting across two GPUs was 2.5–3.5× slower. That was wrong. In that test the 5060 Ti link had dropped to Gen2, and the last card had only about 60 MiB of VRAM left. The engine stopped with out-of-memory errors and restarted during the run. With Strata 0.2.2 we added a VRAM reserve for the later card (--vram-reserve-later-mib 1400, the value the engine log suggests) and re-tested. No run stopped. All runs use the 0.2.2 defaults (route tail skip on), and the single-card numbers are from 0.2.2 too.
With the automatic layer split, a second GPU helped little. Strata splits the layers between the cards (the first card runs the first layers, the last card the rest and the output). The automatic split ("layer_split": "auto") gave the 3060 19 layers and the 5060 Ti 29. Reversing the card order (35 layers on the 5060 Ti) only made long prompts slower.
Choosing the layer counts by hand and loading only each card's own layer weights (--trim-stage-weights) made two GPUs clearly faster than one. The VRAM freed from the weights goes to the expert cache, so the two caches hold about 1.35× as many experts. All of these runs use the 3060 → 5060 Ti order.
| Median (cold / warm) | 5060 Ti alone | Two GPUs, automatic (19 / 29 layers) | Two GPUs, 24 / 24 layers + trim | Two GPUs, 30 / 18 layers + trim |
|---|---|---|---|---|
| Chat decode (tok/s) | 13.0 / 16.0 | 13.6 / 19.4 | 24.8 / 22.4 | 26.1 / 23.7 |
| 4K prompt (tok/s) | 268 / 268 | 203 / 200 | 236 / 239 | 281 / 239 |
| 16K prompt (tok/s) | 529 / 499 | 421 / 441 | 454 / 471 | 509 / 494 |
| 32K prompt (tok/s) | 588 / 588 | 538 / 541 | 639 / 653 | 606 / 620 |
| Agent-style wait (s) | 14.0 / 14.0 | 16.0 / 17.0 | 13.8 / 12.3 | 13.3 / 12.8 |
| Agent-style decode (tok/s) | 20.7 / 20.1 | 23.8 / 22.5 | 27.3 / 27.0 | 26.4 / 24.3 |
With 24 / 24 or 30 / 18 layers, chat decode was about 1.4–2× the single card (steady at 22–26 tok/s cold and warm), agent-style decode 20–35% faster, and the agent-style wait up to 1.7 s shorter. 32K prompts were 3–11% faster, and 4K and 16K prompts about the same as one card. Giving the 5060 Ti 36 layers (12 / 36) sped up decode but left prompt processing near the automatic split. The automatic split tends to give the faster card too many layers (the same as Strata issue #2644). We have not yet separated the effect of trimming from the effect of the layer counts.
Even so, the two caches hold only part of the commonly used experts, and the rest come from RAM and the SSD (about 900 GiB read from the SSD per test run). More RAM would still help on top of the second GPU. Our recommendation for this PC (64 GB of RAM): 3060 → 5060 Ti with "layer_split": "24" (or "30" for mostly short prompts and chat), --trim-stage-weights and --vram-reserve-later-mib 1400.
On the Xeon (E5-2630 v2, 252 GB installed; the engine was bound to one node of about 128 GB, which holds every expert; the setup until 10/31) with the same two cards, the second GPU helped even more. Same 0.2.2 (the Xeon's AVX build) and settings; we compared the 5060 Ti alone with the best split on the i7, 24 / 24 layers with trim. Both cards run at Gen3 x8. The Xeon never reads the SSD, so cold and warm came out almost the same.
| Median (cold / warm) | i7, one card | i7, 24 / 24 + trim | Xeon, one card | Xeon, 24 / 24 + trim |
|---|---|---|---|---|
| Chat decode (tok/s) | 13.0 / 16.0 | 24.8 / 22.4 | 21.7 / 21.3 | 28.1 / 30.0 |
| 4K prompt (tok/s) | 268 / 268 | 236 / 239 | 345 / 346 | 348 / 348 |
| 16K prompt (tok/s) | 529 / 499 | 454 / 471 | 689 / 689 | 848 / 845 |
| 32K prompt (tok/s) | 588 / 588 | 639 / 653 | 698 / 698 | 1,017 / 1,017 |
| Agent-style wait (s) | 14.0 / 14.0 | 13.8 / 12.3 | 11.6 / 11.6 | 11.3 / 11.3 |
| Agent-style decode (tok/s) | 20.7 / 20.1 | 27.3 / 27.0 | 23.6 / 27.2 | 35.3 / 35.2 |
On the Xeon, two cards made chat decode 30–40% and agent-style decode 30–50% faster than one. Unlike on the i7, prompts did not get slower: 16K was 23% and 32K 46% faster (4K and the agent-style wait about the same). The longer the prompt, the more the second card helps. On the i7 the SSD held speed back; with everything in RAM, the added GPU shows up directly. Even card for card, the old AVX-only Xeon beat the i7 (decode +33–67%, prompts +19–30%): again, what matters is RAM that holds every expert.
What follows is an estimate from the engine's own log figures and one or two measurements; it has not been tested. This PC (i7-7800X, X299) tops out at PCIe 3.0, and the 5060 Ti is an x8 card, so Gen3 x8 (6.9 GB/s measured) is its limit. Scaled the same way, Gen4 x8 would give about 14 GB/s and Gen5 x8 (32 GT/s × 8 lanes) about 28 GB/s.
Prompt processing sends the experts that are not in the GPU (about 68 GB) over PCIe once per roughly 8K-token chunk: about 10 s per chunk on Gen3, 2.5 s on Gen5. With 64 GB of RAM, though, reading the experts that miss RAM from the SSD takes longer (about 70 s for 32K) and hides the PCIe time. So we expect a faster PCIe link to change almost nothing at 64 GB. Once more RAM removes the SSD wait, PCIe starts to matter, but not 4×: compute and other work that does not overlap (about 6 s per chunk, estimated from the Xeon) still adds on.
| RAG prompt | 64 GB RAM (now) | 128 GB, Gen3 x8 | 128 GB, Gen5 x8 |
|---|---|---|---|
| 4–8K (one chunk) | about 20 s (4K measured) | about 15 s | about 8 s |
| 32K (four chunks) | about 72 s (measured) | 45–65 s | 20–35 s |
| Answer decode | about 8 tok/s | about 14 tok/s | about 14 tok/s |
RAG feeds different documents of thousands to tens of thousands of tokens each time, so earlier prompt work cannot be reused and this wait is the time until the answer starts. The hypothesis: RAM first (about 2× decode, a slightly shorter wait), then PCIe 4.0/5.0 (about half the wait). On 10/22 the Xeon, with every expert in RAM (5060 Ti alone, Gen3 x8), waited about 12 s at 4K and about 46 s at 32K and decoded about 22 tok/s. The CPU differs, so this is not a direct check, but the waits match or slightly beat the “128 GB, Gen3 x8” estimate in the table above, and decode is well above the estimate (the earlier Xeon + 3060 figure). To test it: with 128 GB of RAM, compare the 3060 at x16 (12.3 GB/s, about Gen4 x8 bandwidth) with the 5060 Ti at x8, using prompts of RAG length (8K, 16K and so on).
What follows is an estimate from our measurements (the i7 with 64 GB and the Xeon) and one official Strata benchmark (Ryzen 9 9900X3D, 96 GB DDR5-6000, RTX 5080, the same UD-Q4_K_XL); we have not tested it. It assumes Strata 0.2.2 with the 5060 Ti alone or the 3060 → 5060 Ti pair (24 / 24 layers + trim). The Xeon has 252 GB installed, but the engine was bound to one node, so it used about 128 GB (the setup until 10/31; since 11/04 it is unbound and uses both nodes). The DDR5 PC is assumed to have dual-channel DDR5-6000, a Gen4–5 NVMe, the 5060 Ti at Gen5 x8 and the 3060 at Gen4 x8.
| Estimate (except measured columns) | i7, 64 GB (measured) | i7, 128 GB | Xeon, one 128 GB node (measured) | DDR5, 64 GB | DDR5, 128 GB |
|---|---|---|---|---|---|
| All 4-bit experts in RAM | no | yes | yes | no | yes |
| Chat decode, one card (tok/s) | 13.0 / 16.0 | 24–30 | 21.7 / 21.3 | 16–22 | 30–40 |
| Chat decode, two cards (tok/s) | 24.8 / 22.4 | 30–38 | 28.1 / 30.0 | 25–32 | 38–50 |
| 32K prompt, one card (tok/s) | 588 / 588 | 700–750 | 698 / 698 | 600–850 | 800–1,100 |
| 32K prompt, two cards (tok/s) | 639 / 653 | 1,000–1,150 | 1,017 / 1,017 | 700–1,000 | 1,000–1,300 |
| Total time, one card (s) | about 90 | 60–68 | about 70 | 61–86 | 42–58 |
| Total time, two cards (s) | about 72 | 42–50 | about 49 | 48–67 | 35–46 |
We expect RAM capacity to matter most. Whether all 4-bit experts (71.7 GiB) fit in RAM moves the total time from about 72 s to 42–50 s on the i7 (two cards), and from 48–67 s to 35–46 s on a DDR5 PC. At the same 128 GB, the Xeon (DDR3, AVX only), the i7 (DDR4) and DDR5 differ by only 10–30%, set by the CPU and memory bandwidth. A DDR5 PC with 64 GB still waits on the SSD for experts, so we expect it to be about as fast as, or a little slower than, the i7 with 128 GB.
Caveats: 128 GB can slow the memory down (four DIMMs on dual-channel DDR5 often run at DDR5-4800–5600; eight on the i7 at DDR4-2400–2133). The DDR5 figures scale down one result that differs in GPU (5080), version (0.1.39) and OS. Adding RAM to the i7 and running the same two tests would check this hypothesis.
Reports to Strata
We submitted the results in Strata's official benchmark format. Bugs found during testing were reported with reproduction steps and proposed fixes.
tq_rows_avx512 is not on the list
Issue #3412ggml_tq_overlay fails on CMake 3.xIN_LIST is unavailable in -P mode
Comment · #3235NameError: port in strata_bench.pyReported that it happens on Linux as well as Windows
Benchmark · Issue #3703Unequal GPU pair (3060 + 5060 Ti) on the i7 and the XeonAutomatic split in both card orders and three hand-chosen splits with trim; one card and 24 / 24 on the Xeon with every expert in RAM
Issue #3702Automatic split leaves 68 MiB on the last card and the engine exitsOut of memory when the verify window is captured; fixed by --vram-reserve-later-mib 1400
Comment · #3470What happens when a later card fills upPointed to #3702: on this pair the engine exits
Timeline
Built 0.1.45 from source for AVX and ran the full test suite (speed, NUMA, needle, HumanEval, a 65-minute soak and more).
Submitted PR #2793 as an official benchmark.
Built and re-measured 0.2.0-beta. Noticed the GPU had moved to a Gen3 x8 slot, so re-measured 0.1.45 in the same slot to separate the slot's effect from the version's. Reported three bugs, updated the PR and reopened it.
Downloaded the 4-bit UD-Q4_K_XL and compared its speed and correctness with 2-bit. Then moved to a second PC (i7-7800X, AVX-512) and compared the AVX and regular builds on 4-bit.
Measured an RTX 5060 Ti alone on the same PC. Found that the model SSD's path was limited to PCIe Gen3 x1, moved the model to a direct x4 SSD and re-measured (prompt processing about 1.27× faster). Splitting across two GPUs (3060 + 5060 Ti) was 2.5–3.5× slower (later found to be wrong: out-of-VRAM restarts and a degraded link; re-tested on 0.2.2, the automatic split made prompts 8–31% slower, and a hand-chosen split made chat decode about 1.4–2× one card. See the re-test). Compared the factors that set speed across the six setups with statistics (factor analysis). Then moved the same two cards to the Xeon and measured one card and 24 / 24 layers with trim on 0.2.2, 4-bit (two cards: decode 30–50% faster, 32K prompts 46% faster; see the re-test). Reported the two-GPU results and the bug we found to Strata (#3703, #3702, a comment on #3470).
Extended the layer-major prefill of #1670 to a two-GPU layer split, compared speed and correctness on 0.2.4.2, and reported it in #3976.
Strata 0.2.7 made layer-major the default for a two-card layer split (details).
PR #2793 was closed. Its results are planned for a community benchmark page on the Strata website.
Measured 0.2.8 on the i7 and the Xeon and reported it in #3976 (details).
Measured 0.2.9 and 0.2.10 on the Xeon. Found that 0.2.10's DeepSeek runs out of memory when bound to one NUMA node.
Stopped NUMA binding (faster). Measured long prompts on two cards with 0.2.9 and 0.2.10: layer-major does not make them faster on this machine (details). Measured table-reading RAG again on 0.2.10 and compared it with the models used before Strata (RAG).
Raised the i7-7800X to 125 GiB of RAM, moved both GPUs over and re-measured. It decodes 12% faster than the Xeon, confirming that RAM capacity sets decode speed. Two-GPU layer-major is not faster on the i7 either; on a 3060 alone, 0.2.10's default walk was 11–15% faster (details). Found with nsys why two-GPU prompt reads were slow (the x8 5060 Ti receiving experts); 24 layers on the 3060 with each GPU loading only its own weights took a 31K prompt from 40–44 s to 29 s. Opened issue #5265 and PR #5255 upstream (details). In the evening, NVFP4 experts took a 31K prompt to 24.8 s and decode about 30% faster, with no visible quality difference; the everyday setup was switched, and benchmark #5279 went upstream (details).
On a single RTX 5060 Ti, compared Strata 0.2.13 with llama.cpp b12387 on the same UD-Q4_K_XL GGUF. Strata read prompts about 2–4× faster and decoded about 1.7× faster (details). Then flew a simulated drone on another PC on the LAN with each engine, 3 runs each: Strata averaged 305 s with all 3 finished, llama.cpp 428 s with 2. With Strata also reading images on the GPU, calls with an image went from 31.8 s to 10.6 s. With the drone flown through ROS 2 instead, it also completed the search and landing. Moved the drone tests to a “physical AI” page and compared one run each with an embodied VLM, Cosmos-Reason2-8B, which did not find the ball. One more run with Cosmos thinking in its official format also did not find the ball and hit the step limit (details).
Built Strata 0.2.15 and compared it with 0.2.13 on one RTX 5060 Ti. Long prompts were read about 7% faster, with the same decode speed and RAG answers, so the daily setup moved to 0.2.15 (details).
Made a 15-second ad for a fictional ketchup using only the two PCs on the LAN (Strata plans; Qwen-Image, MiniMax-H3 and VOICEVOX make the images, video and voice; about 25 min). Had Strata review the finished video, write fix plans from its findings and remake the cuts, and finally pick the best of all takes (details).
Conditions and limits
- Each version's speed was measured once, without alternating repeats. Except for the i7-7800X section, all measurements are from the one Xeon machine.
- The correctness tests are small (40 HumanEval problems, 12 Japanese questions) and cannot detect fine quality differences.
- We ran the engine and server from source. We did not try the 0.2.0 desktop app or installer.
- On the Xeon, everything was bound to the NUMA node the GPU is attached to until 10/31; since 11/04 nothing is bound (the i7-7800X has a single node).