Strata research · 2026-10–11

Running a large MoE on a Xeon without AVX2

Strata runs large MoE models such as Qwen3.8-Flash-Next on one GPU plus system RAM and CPU. It officially requires AVX2, but it has an experimental build for older CPUs (STRATA_ISA_FLOOR=avx). We measured how far it gets on a 2013-era Xeon with an RTX 3060.

CPU
Xeon E5-2630 v2 × 2 (AVX only)
RAM
252 GB DDR3
GPU
RTX 3060 12 GB (also with an RTX 5060 Ti 16 GB since 10/22)
OS
Fedora 44
News · 2026-11-17

A 15-second ad made only with local AI, with Strata (Qwen) reviewing the video and fixing it itself

Using only the two PCs on the LAN, we made an ad for a fictional ketchup in about 25 minutes. Strata handles planning, review, fix plans and choosing takes; Qwen-Image, MiniMax-H3 and VOICEVOX make the images, video and voice. Without any checklist Qwen found the big problems in the finished video (an empty bottle, a different boy, hands that were not in the picture), fixed them with shared reference pictures, and finally assembled the best take of each cut. Setup, tools, roles and results are on Video generation with Strata.

News · 2026-11-16

Moved to Strata 0.2.15: long prompts are read about 7% faster, with the same decode speed and answers

Since 0.2.14, Strata reads the prompt with its own kernels instead of code taken from llama.cpp, including the NVFP4 experts (0.2.15 changes only the app side; its engine is the same as 0.2.14's). We built 0.2.15 from source and compared it with 0.2.13 on our daily setup (one RTX 5060 Ti 16 GB, Qwen3.8-Flash-Next NVFP4), starting each version twice, alternately.

Version31.1K tokens26.2K tokens~5K tokens~0.5K tokensDecode
Strata 0.2.1333.6 / 34.7 s33.1 / 33.3 s10.6 / 10.8 s4.2 / 4.3 s34.2 / 34.0 tok/s
Strata 0.2.1531.5 / 31.5 s30.8 / 30.7 s10.3 / 10.4 s4.0 / 4.0 s(28.4) / 33.8 tok/s

Time to the first token, first / second engine start. Decode is the mean of 4 summaries of 6,500 characters. Temperature 0, no thinking.


News · 2026-11-15

The drone tests have moved to “Strata for physical AI”

The 3-run comparison with llama.cpp (Strata: 305 s on average, all 3 finished), reading images on the GPU, control through ROS 2, and the comparison with an embodied VLM (Cosmos-Reason2-8B) are on Strata for physical AI.

News · 2026-11-15

Against the latest llama.cpp, with the same model and the same GGUF, Strata is still faster: about 2–4× on prompt reads, about 1.7× on decode

On the i7-7800X rig with a single RTX 5060 Ti 16 GB, we ran Qwen3.8-Flash-Next on Strata 0.2.13 and on llama.cpp b12387 (released 11/14, the CUDA release build). llama.cpp read the UD-Q4_K_XL GGUF directly, and Strata was measured on a pack of the same GGUF. Our everyday NVFP4 setup is shown as well. llama.cpp used its default automatic placement (--fit on), with a 65,536-token context and flash attention. Every setup was measured the same way: the time to the first token, and the decode speed over four summaries of about 4K tokens.

Engine and setup31.1K tokens26.2K tokensAbout 4K tokensDecode
Strata 0.2.13, NVFP4 (everyday setup)33.8 s33.2 s10.5 s31.9 tok/s
Strata 0.2.13, UD-Q4_K_XL40.9 s36.6 s11.9 s29.8 tok/s
llama.cpp b12387, UD-Q4_K_XL, default543.8 s388.9 s59.6 s17.3 tok/s
llama.cpp b12387, UD-Q4_K_XL, -ub 2048161.4 s131.6 s22.4 s17.5 tok/s
llama.cpp b12387, UD-Q4_K_XL, --load-mode none429.7 s363.1 s55.2 s16.6 tok/s

One start per setup. The 4K and decode columns are averages of four runs. Temperature 0, no thinking.

News · 2026-11-05

NVFP4 experts make long prompts about 14% and decode about 30% faster, on an older RTX 3060 paired with a Blackwell RTX 5060 Ti

Same i7-7800X rig (RTX 3060 Gen3 x16 + RTX 5060 Ti Gen3 x8 layer split, Strata 0.2.10, every expert in RAM). We swapped Qwen3.8-Flash-Next's experts from UD-Q4_K_XL to NVFP4, NVIDIA's 4-bit format, in the GPTQ pack made for Strata. The 3060 has no FP4 math units. Strata's default (w4a8) multiplies the NVFP4 weights on the 8-bit units, so both cards run the same math. The table shows the time to the first token and the decode speed, with the 16,384-token chunk cap and each GPU loading only its own layers' weights.

Experts, layers on the 3060Auto chunk31.1K tokens26.2K tokensDecode
UD-Q4_K_XL, 24 layers (this morning's best)10,75229.0 s28.2 s29.5–37.1 tok/s
NVFP4, 24 layers13,82428.9 s23.7 s40.4–45.0 tok/s
NVFP4, 21 layers14,84829.8 s23.0 s41.3–45.6 tok/s
NVFP4, 18 layers15,87224.8–25.2 s23.8–24.4 s39.1–44.1 tok/s
NVFP4, 16 layers16,12826.3 s24.9 s38.8–41.7 tok/s

One run each (NVFP4 with 18 layers: two engine starts). Temperature 0, no thinking.

News · 2026-11-05

Two-GPU prompt reads made 27–34% faster: the cause was re-sending experts over the x8 link

The rig is an i7-7800X (125 GiB RAM) with an RTX 3060 (PCIe Gen3 x16) + RTX 5060 Ti (Gen3 x8) layer split, running Strata 0.2.10 and Qwen3.8-Flash-Next UD-Q4_K_XL with every expert in RAM. We used nsys to find out why long prompts read slowly here, then changed the settings to make them faster. The table shows the time to the first token for a 31.1K-token prompt.

Setting31.1K tokens
0.2.10's automatic split (18 layers on the 3060)39.8–44.1 s
24 layers on the 3060, each GPU loading only its own layers' weights (--layer-split 24 --trim-stage-weights)29.9 s
The same + a 16,384-token chunk cap (STRATA_PREFILL_AUTO_MAX=16384)29.2 s
A prototype automatic split for the prompt read (picks 23 layers on the 3060 by itself)32.9 s
Both cards reading every layer on alternating chunks (upstream PR #5002)44.1 s
The CPU computing experts that few tokens route to30.8–46.5 s

One run each; 24 layers on the 3060 was measured three times, at 29.9–30.0 s. The gap between 39.8 s and 44.1 s on the automatic split is the number of chunks: 5 against 6.

News · 2026-11-05

Re-measured on the i7-7800X with more RAM: decode speed is set by RAM capacity, now confirmed

We raised the second PC (i7-7800X, AVX-512) to 125 GiB of RAM, moved the Xeon's two GPUs over (RTX 3060 + RTX 5060 Ti layer split), and measured with the same versions and the same config as on the Xeon (setup.py's default). Qwen3.8-Flash-Next (UD-Q4_K_XL) now fits entirely in RAM on the i7 too: experts (71.7 GiB) and the PLE table (26.8 GiB). The 3060 is on PCIe Gen3 x16 and the 5060 Ti on Gen3 x8 (on the Xeon both are Gen3 x8).

Measurei7-7800X (125 GiB RAM)Xeon (252 GB RAM)
Short-prompt decode (0.2.9, median of 60)30.2 tok/s27.0 tok/s
Short-prompt time to first token (median)1.39 s1.81 s
31K-token time to first token (0.2.10 default)39.2–39.5 s44.6–45.0 s
49K-token time to first token (0.2.10 default)59.6–60.0 s64.4–65.3 s
Decode after a long prompt (0.2.9, median)33.5 tok/s31.9 tok/s
News · 2026-11-04

0.2.9 and 0.2.10: on the Xeon E5-2630 v2 machine, two GPUs did not make long prompts faster

We measured 0.2.9 and 0.2.10 on the Xeon (RTX 3060 + RTX 5060 Ti layer split, UD-Q4_K_XL). This time with setup.py's default config: all experts (71.7 GiB) in RAM, and no binding to a NUMA node. The table shows the time to the first token.

Prompt (tokens)0.2.9 default0.2.9 layer-major forced0.2.10 default
0.8–4.7K10.7–15.5 s11.2–16.0 s10.5–15.5 s
7.6–10.3K19.5–21.7 s20.0–23.0 s19.8–21.9 s
14.3–19.5K26.9–29.8 s27.6–32.3 s27.2–30.0 s
31.3–31.5K44.5–44.7 s46.4–46.8 s44.6–45.0 s
48.9–49.0K64.0–65.0 s62.1–62.9 s64.4–65.3 s

3 rounds each, a different prompt every time, temperature 0, no thinking.

No more NUMA binding


News · 2026-10-27

Layer-major is now in Strata

Added 11/04: since 0.2.10, layer-major is not used by default (STRATA_PREFILL_LAYER_MAJOR=1 turns it on). On the Xeon with every expert in RAM it brought no gain on 0.2.9 or 0.2.10 (0.2.9 and 0.2.10). The results in this section come from configs that read part of the experts from the drive.

Layer-major prefill, a faster way to read long prompts, became the default for a layer split over two CUDA cards in Strata 0.2.7 (10/27). The release notes say: "From PR #3976 by @Norhiro, building on @vizdatom's #1670." Eight of my commits (Norihiro Nishimura) were merged into the Strata history and shipped in 0.2.7 (the main commit). They include the two-GPU layer split, even super-chunks and the FP16 path.

Until now, each layer's experts were read again for every 8,192 tokens. Layer-major works through a large piece one layer at a time, so they are read once per piece. vizdatom proposed the method for one GPU (#1670). I extended it to a two-GPU layer split and reported it in #3976. The table shows the time to the first word.

PromptMine: 3060 + 5060 TiSpeedupOfficial: RTX A4000 × 2Speedup
8K28.9 s → 18.5 s1.6×41.8 s → 15.2 s2.8×
20K54.2 s → 23.5 s2.3×98.8 s → 17.5 s5.6×
32K65.0 s → 33.1 s2.0×141.8 s → 27.7 s5.1×

My numbers compare stock 0.2.4.2 with the same version plus this change (UD-Q4_K_XL, the second run of each size). Drive reads at 32K fell from 75 GB to 35 GB. Correctness did not change: both scored needle 9/9, HumanEval 39 / 38, Japanese 11/12 and tool calls 8/8. Decode speed also matched, at 24.0 and 24.1 tok/s. The official numbers come from a different GPU and model, so they cannot be compared with mine.

0.2.8 measured on my two cards (10/31)

At the maintainer's request, I measured 0.2.8 on the 3060 + 5060 Ti and reported it in #3976. In short: at default settings layer-major does not run on this pair, and with more free VRAM the wait at 20K and 32K is about half. The split was 24 / 24 layers, with UD-Q4_K_XL. “Off” is STRATA_PREFILL_LAYER_MAJOR=0. “Tuned” is --vram-reserve-mib 1700 --vram-reserve-later-mib 1700 --prefill 4096 with STRATA_PREFILL_LM_MARGIN_MIB=96.

Machine, arm8K20K32K32K readsDecode (median)
i7-7800X, 62 GB RAM, off24.3 s42.7 s51.3 s87 GB26.85 tok/s
i7-7800X, tuned24.2 s23.4 s29.8 s39 GB21.8 tok/s
2 × Xeon E5-2630 v2, 251 GB RAM, off24.8 s49.1 s62.7 s39 GB28.05 tok/s
Xeon, tuned22.7 s24.2 s30.9 s0.1 GB26.9 tok/s

Times are to the first word, on the second run of each size (i7 two runs each, Xeon three). Decode is the median of 40 answers. Correctness matched between off and tuned on both machines: needle 9/9, HumanEval 39 / 38, Japanese 11/12, tool calls 8/8. The Xeon is an AVX-only CPU with both cards at PCIe Gen3 x8. The AVX build hit no illegal instruction.

How to read this

On the i7, the tuning cost 19% of decode speed. On the Xeon the drop was 4%, but that is one run per arm with off first, so it may be noise. I have not isolated why more RAM shrinks it. The Xeon's near-zero reads when tuned include data already in the page cache. On the i7, layer-major did not run at default settings or with --layer-split auto in either card order, and 32K took 50–65 s. The Xeon did not run those arms. In #3976, others report it running slower on 3090s at PCIe Gen3 and on two V100s, so the effect depends on the cards and the link. To turn it off, set STRATA_PREFILL_LAYER_MAJOR=0.


Summary

Summary and contents

Summary

  • A 15-second ad made only with local AI, reviewed and fixed by Strata (Qwen) itself (11/17). Without a checklist it found the big problems in the finished video, aligned people and products with shared reference pictures, and finally assembled the best take of each cut (video generation).
  • Flying a simulated drone with an LLM, Strata was also faster and steadier (11/15). Three runs each with the same instruction: 305 s on average with all 3 finished, against 428 s and 2 finished for llama.cpp. Calls with an image went from 31.8 s to 10.6 s once Strata also read images on the GPU, faster than llama.cpp's 22.4 s. Flown through ROS 2 instead, the drone also completed the task. An embodied VLM (Cosmos-Reason2-8B) did not complete it (physical AI).
  • Against the latest llama.cpp (b12387), with the same model and the same GGUF, Strata is still faster (11/15). On a single RTX 5060 Ti, compared with llama.cpp's fastest setup here, Strata read a 31K prompt about 4× faster and decoded about 1.7× faster (details).
  • NVFP4 experts: long prompts about 14% faster, decode about 30% faster (11/05, evening). On an older 3060 paired with a Blackwell 5060 Ti, both cards run the same 8-bit math. The smaller weights moved the best split from 24 layers to 18, and a 31K prompt went from 29.0 s to 24.8 s. We saw no difference in answer quality (details).
  • Two-GPU prompt reads 27–34% faster (11/05). The cause was the x8 5060 Ti receiving the experts again for every chunk. With 24 layers on the 3060 and each GPU loading only its own layers' weights, a 31K prompt went from 40–44 s to 29 s (details).
  • On the Xeon E5-2630 v2 machine, two GPUs did not make long prompts faster on 0.2.9 and 0.2.10. This is from an RTX 3060 + RTX 5060 Ti with every expert in RAM (the default config here) only; on 11/05 the i7-7800X gave the same result (details). Under these conditions, 0.2.9's default layer-major never actually ran, and forcing it on left the wait unchanged; PCIe is the limit. 0.2.10 no longer uses layer-major by default. We also stopped binding to a NUMA node (details).
  • Layer-major, a faster way to read long prompts, became the default in Strata 0.2.7 (since 0.2.10 it is no longer used by default). Eight commits from this two-GPU work were merged into Strata. On my two cards, with more free VRAM set aside, the wait at 32K was about half (in configs that read part of the experts from the drive; details).
  • Strata runs correctly on an old AVX-only Xeon. From 0.1.45 to 0.2.10, the AVX builds never hit a single illegal instruction.
  • On the Xeon, the GPU's PCIe slot mattered most. Moving from PCIe Gen2 x4 to Gen3 x8 made prompt processing about 3× faster. 0.2.0 also made decode 6–12% faster.
  • 2-bit and 4-bit scored about the same on correctness, and 4-bit runs at about half the speed. On 0.2.0 with one GPU, 2-bit suited this machine. (Compared on 0.2.0 with one GPU. On 11/04, with 0.2.9 / 0.2.10, two GPUs and setup.py's default config, 4-bit decodes short prompts at 27–28.5 tok/s and reads 32K in about 45 s, about as fast as 2-bit on 0.2.0. 2-bit has not been re-measured on the newer versions.)
  • RAM capacity sets 4-bit decode speed. On an i7-7800X with 64 GB, the experts do not fit in RAM, and decode ran at about half to 60% of the Xeon's, which holds every expert in one 128 GB node. A faster SSD path makes prompt processing about 1.27× faster but leaves decode unchanged. On 11/05, with the i7 raised to 125 GiB so every expert fits, it decoded 12% faster than the Xeon (details).
  • With two GPUs, choosing the layer split by hand makes them faster than one (0.2.2, 3060 + 5060 Ti: with 24 / 24 layers and each card loading only its own layer weights, chat decode was about 1.4–2× one card and the agent-style wait shorter. On the Xeon, which holds every expert in RAM, 32K prompts were also about 1.5× one card. The automatic split helps little. The earlier “2.5–3.5× slower” was wrong. See the two-GPU re-test). All eight setups are ranked in the ranking.

Findings

Findings

Works

Correct on an AVX-only CPU

From 0.1.45 to 0.2.10, the AVX builds never hit an illegal instruction (SIGILL). CPU-side math runs on 128-bit AVX1 kernels, and their parity tests pass.

~3×

The GPU slot mattered most

Just moving the GPU from a PCIe Gen2 x4 slot to a Gen3 x8 slot made prompt processing about 3× faster and cut the agent-style wait from about 22 s to about 6 s.

Capacity

Even on old hardware, RAM capacity sets decode speed

The Xeon's DDR3 has half the i7's bandwidth per node. Because every expert fit in RAM, it still decoded about twice as fast as the i7. What matters is whether all experts fit, more than how fast the memory is. Prompt processing, on the other hand, is set by the GPU and its PCIe path. For a large model, get enough RAM for the experts first, then the fastest slot (confirmed on 11/05: with the i7 raised to 125 GiB so every expert fits, it decoded 1.12× as fast as the Xeon on the same version and config; details).

+6–12%

0.2.0 decodes faster

In the same slot, 0.2.0 decoded 6–12% faster than 0.1.45, close to the published figure (+10.2% on an RTX 3060).

2-bit

2-bit suits this machine

4-bit (UD-Q4_K_XL) runs at about half the speed, while its correctness in these tests was about the same as 2-bit (IQ2_XS). (Compared on 0.2.0 with one GPU. On 11/04, with 0.2.9 / 0.2.10, two GPUs and setup.py's default config, 4-bit decodes short prompts at 27–28.5 tok/s and reads 32K in about 45 s, about as fast as 2-bit on 0.2.0. 2-bit has not been re-measured on the newer versions.)

RAM

On the second PC, RAM and the SSD path matter more than the instruction set

On an i7-7800X with AVX-512 (64 GB RAM), the AVX build and the regular build ran at the same speed. 4-bit does not fit in RAM, and most of the decode time goes to waiting for experts from the SSD. A direct x4 SSD did not speed up decode; RAM capacity is what matters (factor analysis).

Verdict: read speed follows the GPU, write speed follows RAM capacity

What sets speed, across two PCs and eight setups

Read (prompt processing)Set by the GPU and the paths that feed it (PCIe, SSD)

On the Xeon, just moving the GPU from a Gen2 x4 to a Gen3 x8 slot made it about 3× faster. On the i7, the RTX 5060 Ti gave ×1.28 and a direct x4 SSD ×1.27, reaching about 90% of the Xeon.

Write (decode)Set by RAM capacity, enough to hold every expert

The Xeon, with every expert in RAM, decoded at 14.3 tok/s; the i7, where they do not fit, at 7–9 tok/s (as of 10/22, 4-bit, one GPU; on 11/04 the Xeon with two GPUs and the default config decodes at 27–28.5 tok/s). On 11/05 the i7, raised to 125 GiB, decoded at 30.2 tok/s under the same conditions, ahead of the Xeon. A new GPU, a faster SSD and AVX-512 barely narrowed the gap.


Speed

Speed: slot change and 0.2.0

Model IQ2_XS (2-bit), context 40,960. Medians of three runs of tools/strata_bench.py --long, each started after evicting the model files from memory (cold).

Metric0.1.45, old slot0.1.45, new slot0.2.0, new slot
Chat decode17.1 tok/s23.6 tok/s25.1 tok/s
4K prompt processing178 tok/s632 tok/s636 tok/s
16K prompt processing342 tok/s951 tok/s987 tok/s
32K prompt processing352 tok/s973 tok/s992 tok/s
Agent time to first token, per turn21.9 s6.17 s6.22 s
Agent decode20.5 tok/s27.5 tok/s30.9 tok/s
Old slot: 2026-10-16; new slot: 2026-10-20. Host-to-GPU bandwidth measured by the engine went from 1.6 GB/s to 6.0 GB/s. Each version was measured once, so differences of a few percent may be noise.
If you try this on an old motherboard

Strata moves experts that do not fit on the GPU over PCIe. On dual-Xeon boards, a slot shaped like x16 may be wired as Gen2 x4. Check the actual speed and width in the LnkSta line of lspci -vv and use the fastest slot; that mattered more than any software setting.


Correctness

Correctness

A needle-in-a-haystack test in long contexts, HumanEval problems 0–39 (scored with EvalPlus 0.3.1), 12 Japanese questions and 8 tool-calling tasks.

Test0.1.45 (Oct 16)0.2.0 (Oct 20)
Needle (4K/16K/32K × depth 10/50/90%)9/99/9
HumanEval 0–39 base / plus37/40 · 35/4039/40 · 37/40
Japanese, 12 questions11/1211/12
Tool calling, 8 tasks8/88/8
Model IQ2_XS. The slot change also slightly changes the engine's compute path, so the +2 on HumanEval cannot be credited to 0.2.0 alone.

2-bit vs 4-bit

2-bit (IQ2_XS) vs 4-bit (UD-Q4_K_XL)

Both on 0.2.0, the new slot, context 40,960. The 4-bit experts (77 GB) also fit entirely in RAM, but each expert is about twice as large, so the number held on the GPU drops from 3,813 to 1,168.

MetricIQ2_XS (2-bit)UD-Q4_K_XL (4-bit)
Chat decode25.1 tok/s14.3 tok/s
32K prompt processing992 tok/s (32 s)445 tok/s (72 s)
Agent time to first token, per turn6.2 s14.0 s
Agent decode30.9 tok/s14.0 tok/s
Needle9/99/9
HumanEval base / plus39/40 · 37/4037/40 · 36/40
Japanese / tools11/12 · 8/811/12 · 8/8
The two HumanEval problems 4-bit missed had correct function bodies but no from typing import List line, so they failed to run under the grader.

On this machine, 4-bit halves the speed, while these tests showed no difference in correctness. At the time (10/21) we chose 2-bit for daily use. As of 11/04, with two GPUs and setup.py's default config, 4-bit decodes at 27–28.5 tok/s, so we now use 4-bit (UD-Q4_K_XL). 2-bit has not been re-measured on the newer versions. The test sets are small, though, so finer quality differences were not measured.


Another PC

Second PC: Core i7-7800X (AVX-512)

From 2026-10-21 we also measured on a PC with AVX2 and AVX-512. Same RTX 3060 (PCIe Gen3 x8), same Strata 0.2.0-beta (engine 0.1.45.3), with the 4-bit UD-Q4_K_XL model.

CPU
Core i7-7800X (6 cores, AVX2, AVX-512)
RAM
64 GB DDR4-2666 (4 channels)
GPU
RTX 3060 12 GB
OS
Fedora 44

AVX build vs regular (AVX-512) build

We built the same source for old CPUs (STRATA_ISA_FLOOR=avx) and with the regular setting (-march=native) and compared them.

MetricAVX buildRegular build (AVX-512)
Chat decode6.6 tok/s7.4 tok/s
4K prompt processing139 tok/s134 tok/s
32K prompt processing274 tok/s272 tok/s
Agent time to first token, per turn27.6 s28.7 s
Cold, median of three runs. The decode difference is within the run-to-run spread (4–11 tok/s). Strata's own AVX-512 kernels need VNNI and VBMI, so on this CPU both builds use the AVX2 kernels.

Compared with the Xeon (4-bit)

MetricXeon (252 GB RAM; used one node, about 128 GB, as set up until 10/31)i7-7800X (64 GB RAM)
Chat decode14.3 tok/s7.4 tok/s
4K prompt processing293 tok/s134 tok/s
32K prompt processing445 tok/s (72 s)272 tok/s (118 s)
Agent time to first token, per turn14.0 s28.7 s
The i7-7800X uses the regular build. Its CPU (AVX2) and memory bandwidth (60 GB/s, versus about 30 GB/s for one Xeon node) are the better of the two.

The 4-bit experts (77 GB) do not fit in 64 GB of RAM, so only 38 GiB stay in RAM and the remaining ~30 GiB are read from the SSD for every token (--resident-budget-gib 38). Those reads set the speed; the instruction-set difference did not show. Comparing the CPUs themselves would need 2-bit, which fits in RAM on both machines.

RTX 5060 Ti 16 GB alone

On the same PC we swapped only the GPU for an RTX 5060 Ti (Blackwell, sm_120, PCIe Gen3 x8). Everything else stayed the same. The extra VRAM raises the experts held on the GPU from 1,374 to 2,731.

MetricRTX 3060RTX 5060 Ti
Chat decode7.4 tok/s8.9 tok/s
4K prompt processing134 tok/s159 tok/s
32K prompt processing272 tok/s (118 s)354 tok/s (91 s)
Agent time to first token, per turn28.7 s24.3 s
Agent decode11.3 tok/s13.9 tok/s
i7-7800X, regular build, 4-bit, cold medians. On the 5060 Ti, decode improved as the cache warmed up, reaching 14–16 tok/s by the third run.

The new GPU made things 20–30% faster, but still short of the Xeon with every expert in RAM.

The SSD path was PCIe Gen3 x1

The NVMe SSD holding the model (Solidigm P41 Plus) links at x4 itself, but it sits behind an M.2 expansion card (a PLX PEX 8747 switch) that connects to an x1 chipset port. Uncached reads ran at 782 MB/s, under a quarter of the SSD's own speed. Experts that miss RAM are read over this path, so at first we took it to be the main limit on 4-bit speed; the direct x4 test below showed it only affects prompt processing. You can check by walking /sys/bus/pci/devices/*/current_link_width from the SSD up through each bridge. We have not yet tried fixes such as moving the SSD to the motherboard's M.2 slot.

Moving the model to a direct x4 SSD

As a fix that needs no hardware changes, we copied the model to the /home SSD (KIOXIA BG4, Gen3 x4 direct). Uncached reads ran at 1.9 GB/s with one stream and 2.4 GB/s with four, 2.4–3× the original path (about 0.8 GB/s). GPU RTX 5060 Ti, everything else unchanged.

Metricx1 pathDirect x4For reference: Xeon
Chat decode8.9 tok/s7.8 tok/s14.3 tok/s
4K prompt processing159 tok/s202 tok/s293 tok/s
32K prompt processing354 tok/s (91 s)448 tok/s (72 s)445 tok/s (72 s)
Agent time to first token, per turn24.3 s17.2 s14.0 s
Agent decode13.9 tok/s15.0 tok/s14.0 tok/s
i7-7800X with RTX 5060 Ti, 4-bit, cold medians. The Xeon reference uses an RTX 3060 with every expert in RAM. The chat decode difference is within the run-to-run spread (5–15 tok/s).

Prompt processing got about 1.27× faster at every length and matched the Xeon at 32K. The agent-style wait before a reply starts dropped from 24 s to 17 s. Chat decode, though, stayed around 8 tok/s, still well behind the Xeon. Decode reads experts of about 4 MB a few at a time. The engine log shows each read waits about 3 ms on both x1 and direct x4, so bandwidth is not what limits it. To speed up decode as well, the sure fix is more RAM, enough to hold every expert.

Which factors matter (statistics)

We compared the six setups so far (Xeon; i7 with the AVX and AVX-512 builds; RTX 5060 Ti; direct x4 SSD; two GPUs), pairing setups that differ in one factor. Chat decode changed by ×0.47 from Xeon to i7, ×0.44 when split across two GPUs (this value was wrong; adding one card and two cards with a 24 / 24 split and trim on the Xeon with 0.2.2, two cards gave ×1.33; see the re-test), and ×1.36 with the RTX 5060 Ti. The direct x4 SSD and AVX-512 barely affected decode. The Xeon had its memory bound to one socket (126 GiB), so what mattered was one node with enough RAM for every expert (the setup until 10/31; nothing is bound since 11/04, details). Charts and 95% intervals are on the 4-bit speed factors page (English / 日本語). The eight setups, including those two, ranked by read (prompt processing) and write (decode) speed are in the effective speed ranking.

Correction and update: two GPUs re-tested (Strata 0.2.2, 10/22; Xeon results added)

We wrote here earlier that splitting across two GPUs was 2.5–3.5× slower. That was wrong. In that test the 5060 Ti link had dropped to Gen2, and the last card had only about 60 MiB of VRAM left. The engine stopped with out-of-memory errors and restarted during the run. With Strata 0.2.2 we added a VRAM reserve for the later card (--vram-reserve-later-mib 1400, the value the engine log suggests) and re-tested. No run stopped. All runs use the 0.2.2 defaults (route tail skip on), and the single-card numbers are from 0.2.2 too.

With the automatic layer split, a second GPU helped little. Strata splits the layers between the cards (the first card runs the first layers, the last card the rest and the output). The automatic split ("layer_split": "auto") gave the 3060 19 layers and the 5060 Ti 29. Reversing the card order (35 layers on the 5060 Ti) only made long prompts slower.

Choosing the layer counts by hand and loading only each card's own layer weights (--trim-stage-weights) made two GPUs clearly faster than one. The VRAM freed from the weights goes to the expert cache, so the two caches hold about 1.35× as many experts. All of these runs use the 3060 → 5060 Ti order.

Median (cold / warm)5060 Ti aloneTwo GPUs, automatic (19 / 29 layers)Two GPUs, 24 / 24 layers + trimTwo GPUs, 30 / 18 layers + trim
Chat decode (tok/s)13.0 / 16.013.6 / 19.424.8 / 22.426.1 / 23.7
4K prompt (tok/s)268 / 268203 / 200236 / 239281 / 239
16K prompt (tok/s)529 / 499421 / 441454 / 471509 / 494
32K prompt (tok/s)588 / 588538 / 541639 / 653606 / 620
Agent-style wait (s)14.0 / 14.016.0 / 17.013.8 / 12.313.3 / 12.8
Agent-style decode (tok/s)20.7 / 20.123.8 / 22.527.3 / 27.026.4 / 24.3

With 24 / 24 or 30 / 18 layers, chat decode was about 1.4–2× the single card (steady at 22–26 tok/s cold and warm), agent-style decode 20–35% faster, and the agent-style wait up to 1.7 s shorter. 32K prompts were 3–11% faster, and 4K and 16K prompts about the same as one card. Giving the 5060 Ti 36 layers (12 / 36) sped up decode but left prompt processing near the automatic split. The automatic split tends to give the faster card too many layers (the same as Strata issue #2644). We have not yet separated the effect of trimming from the effect of the layer counts.

Even so, the two caches hold only part of the commonly used experts, and the rest come from RAM and the SSD (about 900 GiB read from the SSD per test run). More RAM would still help on top of the second GPU. Our recommendation for this PC (64 GB of RAM): 3060 → 5060 Ti with "layer_split": "24" (or "30" for mostly short prompts and chat), --trim-stage-weights and --vram-reserve-later-mib 1400.

On the Xeon (E5-2630 v2, 252 GB installed; the engine was bound to one node of about 128 GB, which holds every expert; the setup until 10/31) with the same two cards, the second GPU helped even more. Same 0.2.2 (the Xeon's AVX build) and settings; we compared the 5060 Ti alone with the best split on the i7, 24 / 24 layers with trim. Both cards run at Gen3 x8. The Xeon never reads the SSD, so cold and warm came out almost the same.

Median (cold / warm)i7, one cardi7, 24 / 24 + trimXeon, one cardXeon, 24 / 24 + trim
Chat decode (tok/s)13.0 / 16.024.8 / 22.421.7 / 21.328.1 / 30.0
4K prompt (tok/s)268 / 268236 / 239345 / 346348 / 348
16K prompt (tok/s)529 / 499454 / 471689 / 689848 / 845
32K prompt (tok/s)588 / 588639 / 653698 / 6981,017 / 1,017
Agent-style wait (s)14.0 / 14.013.8 / 12.311.6 / 11.611.3 / 11.3
Agent-style decode (tok/s)20.7 / 20.127.3 / 27.023.6 / 27.235.3 / 35.2

On the Xeon, two cards made chat decode 30–40% and agent-style decode 30–50% faster than one. Unlike on the i7, prompts did not get slower: 16K was 23% and 32K 46% faster (4K and the agent-style wait about the same). The longer the prompt, the more the second card helps. On the i7 the SSD held speed back; with everything in RAM, the added GPU shows up directly. Even card for card, the old AVX-only Xeon beat the i7 (decode +33–67%, prompts +19–30%): again, what matters is RAM that holds every expert.

Unverified hypothesis: PCIe generation and RAG wait times

What follows is an estimate from the engine's own log figures and one or two measurements; it has not been tested. This PC (i7-7800X, X299) tops out at PCIe 3.0, and the 5060 Ti is an x8 card, so Gen3 x8 (6.9 GB/s measured) is its limit. Scaled the same way, Gen4 x8 would give about 14 GB/s and Gen5 x8 (32 GT/s × 8 lanes) about 28 GB/s.

Prompt processing sends the experts that are not in the GPU (about 68 GB) over PCIe once per roughly 8K-token chunk: about 10 s per chunk on Gen3, 2.5 s on Gen5. With 64 GB of RAM, though, reading the experts that miss RAM from the SSD takes longer (about 70 s for 32K) and hides the PCIe time. So we expect a faster PCIe link to change almost nothing at 64 GB. Once more RAM removes the SSD wait, PCIe starts to matter, but not 4×: compute and other work that does not overlap (about 6 s per chunk, estimated from the Xeon) still adds on.

RAG prompt64 GB RAM (now)128 GB, Gen3 x8128 GB, Gen5 x8
4–8K (one chunk)about 20 s (4K measured)about 15 sabout 8 s
32K (four chunks)about 72 s (measured)45–65 s20–35 s
Answer decodeabout 8 tok/sabout 14 tok/sabout 14 tok/s
Estimated time to first token (RTX 5060 Ti, 4-bit). The 128 GB decode figure is the Xeon's, with every expert in RAM.

RAG feeds different documents of thousands to tens of thousands of tokens each time, so earlier prompt work cannot be reused and this wait is the time until the answer starts. The hypothesis: RAM first (about 2× decode, a slightly shorter wait), then PCIe 4.0/5.0 (about half the wait). On 10/22 the Xeon, with every expert in RAM (5060 Ti alone, Gen3 x8), waited about 12 s at 4K and about 46 s at 32K and decoded about 22 tok/s. The CPU differs, so this is not a direct check, but the waits match or slightly beat the “128 GB, Gen3 x8” estimate in the table above, and decode is well above the estimate (the earlier Xeon + 3060 figure). To test it: with 128 GB of RAM, compare the 3060 at x16 (12.3 GB/s, about Gen4 x8 bandwidth) with the 5060 Ti at x8, using prompts of RAG length (8K, 16K and so on).

Untested hypothesis: RAM capacity and speed on a DDR5 PC (10/22)

What follows is an estimate from our measurements (the i7 with 64 GB and the Xeon) and one official Strata benchmark (Ryzen 9 9900X3D, 96 GB DDR5-6000, RTX 5080, the same UD-Q4_K_XL); we have not tested it. It assumes Strata 0.2.2 with the 5060 Ti alone or the 3060 → 5060 Ti pair (24 / 24 layers + trim). The Xeon has 252 GB installed, but the engine was bound to one node, so it used about 128 GB (the setup until 10/31; since 11/04 it is unbound and uses both nodes). The DDR5 PC is assumed to have dual-channel DDR5-6000, a Gen4–5 NVMe, the 5060 Ti at Gen5 x8 and the 3060 at Gen4 x8.

Estimate (except measured columns)i7, 64 GB (measured)i7, 128 GBXeon, one 128 GB node (measured)DDR5, 64 GBDDR5, 128 GB
All 4-bit experts in RAMnoyesyesnoyes
Chat decode, one card (tok/s)13.0 / 16.024–3021.7 / 21.316–2230–40
Chat decode, two cards (tok/s)24.8 / 22.430–3828.1 / 30.025–3238–50
32K prompt, one card (tok/s)588 / 588700–750698 / 698600–850800–1,100
32K prompt, two cards (tok/s)639 / 6531,000–1,1501,017 / 1,017700–1,0001,000–1,300
Total time, one card (s)about 9060–68about 7061–8642–58
Total time, two cards (s)about 7242–50about 4948–6735–46
Total time is one request that reads a 32K-token document and writes a 500-token answer (32,768 ÷ 32K prompt speed + 500 ÷ chat decode). Measured values are cold / warm medians; the estimates are good to about ±30%.

We expect RAM capacity to matter most. Whether all 4-bit experts (71.7 GiB) fit in RAM moves the total time from about 72 s to 42–50 s on the i7 (two cards), and from 48–67 s to 35–46 s on a DDR5 PC. At the same 128 GB, the Xeon (DDR3, AVX only), the i7 (DDR4) and DDR5 differ by only 10–30%, set by the CPU and memory bandwidth. A DDR5 PC with 64 GB still waits on the SSD for experts, so we expect it to be about as fast as, or a little slower than, the i7 with 128 GB.

Caveats: 128 GB can slow the memory down (four DIMMs on dual-channel DDR5 often run at DDR5-4800–5600; eight on the i7 at DDR4-2400–2133). The DDR5 figures scale down one result that differs in GPU (5080), version (0.1.39) and OS. Adding RAM to the i7 and running the same two tests would check this hypothesis.


Upstream

Reports to Strata

We submitted the results in Strata's official benchmark format. Bugs found during testing were reported with reproduction steps and proposed fixes.


Log

Timeline

Built 0.1.45 from source for AVX and ran the full test suite (speed, NUMA, needle, HumanEval, a 65-minute soak and more).

Submitted PR #2793 as an official benchmark.

Built and re-measured 0.2.0-beta. Noticed the GPU had moved to a Gen3 x8 slot, so re-measured 0.1.45 in the same slot to separate the slot's effect from the version's. Reported three bugs, updated the PR and reopened it.

Downloaded the 4-bit UD-Q4_K_XL and compared its speed and correctness with 2-bit. Then moved to a second PC (i7-7800X, AVX-512) and compared the AVX and regular builds on 4-bit.

Measured an RTX 5060 Ti alone on the same PC. Found that the model SSD's path was limited to PCIe Gen3 x1, moved the model to a direct x4 SSD and re-measured (prompt processing about 1.27× faster). Splitting across two GPUs (3060 + 5060 Ti) was 2.5–3.5× slower (later found to be wrong: out-of-VRAM restarts and a degraded link; re-tested on 0.2.2, the automatic split made prompts 8–31% slower, and a hand-chosen split made chat decode about 1.4–2× one card. See the re-test). Compared the factors that set speed across the six setups with statistics (factor analysis). Then moved the same two cards to the Xeon and measured one card and 24 / 24 layers with trim on 0.2.2, 4-bit (two cards: decode 30–50% faster, 32K prompts 46% faster; see the re-test). Reported the two-GPU results and the bug we found to Strata (#3703, #3702, a comment on #3470).

Extended the layer-major prefill of #1670 to a two-GPU layer split, compared speed and correctness on 0.2.4.2, and reported it in #3976.

Strata 0.2.7 made layer-major the default for a two-card layer split (details).

PR #2793 was closed. Its results are planned for a community benchmark page on the Strata website.

Measured 0.2.8 on the i7 and the Xeon and reported it in #3976 (details).

Measured 0.2.9 and 0.2.10 on the Xeon. Found that 0.2.10's DeepSeek runs out of memory when bound to one NUMA node.

Stopped NUMA binding (faster). Measured long prompts on two cards with 0.2.9 and 0.2.10: layer-major does not make them faster on this machine (details). Measured table-reading RAG again on 0.2.10 and compared it with the models used before Strata (RAG).

Raised the i7-7800X to 125 GiB of RAM, moved both GPUs over and re-measured. It decodes 12% faster than the Xeon, confirming that RAM capacity sets decode speed. Two-GPU layer-major is not faster on the i7 either; on a 3060 alone, 0.2.10's default walk was 11–15% faster (details). Found with nsys why two-GPU prompt reads were slow (the x8 5060 Ti receiving experts); 24 layers on the 3060 with each GPU loading only its own weights took a 31K prompt from 40–44 s to 29 s. Opened issue #5265 and PR #5255 upstream (details). In the evening, NVFP4 experts took a 31K prompt to 24.8 s and decode about 30% faster, with no visible quality difference; the everyday setup was switched, and benchmark #5279 went upstream (details).

On a single RTX 5060 Ti, compared Strata 0.2.13 with llama.cpp b12387 on the same UD-Q4_K_XL GGUF. Strata read prompts about 2–4× faster and decoded about 1.7× faster (details). Then flew a simulated drone on another PC on the LAN with each engine, 3 runs each: Strata averaged 305 s with all 3 finished, llama.cpp 428 s with 2. With Strata also reading images on the GPU, calls with an image went from 31.8 s to 10.6 s. With the drone flown through ROS 2 instead, it also completed the search and landing. Moved the drone tests to a “physical AI” page and compared one run each with an embodied VLM, Cosmos-Reason2-8B, which did not find the ball. One more run with Cosmos thinking in its official format also did not find the ball and hit the step limit (details).

Built Strata 0.2.15 and compared it with 0.2.13 on one RTX 5060 Ti. Long prompts were read about 7% faster, with the same decode speed and RAG answers, so the daily setup moved to 0.2.15 (details).

Made a 15-second ad for a fictional ketchup using only the two PCs on the LAN (Strata plans; Qwen-Image, MiniMax-H3 and VOICEVOX make the images, video and voice; about 25 min). Had Strata review the finished video, write fix plans from its findings and remake the cuts, and finally pick the best of all takes (details).


Limits

Conditions and limits