Strata research · RAG · 2026-10 to 11

Thinking longer does not read tables

With a railway company's passenger tariff (a 98-page PDF full of fare and distance tables) as the source, we tested table reading and question answering in a RAG pipeline, using a model served by Strata. We looked at how accuracy changes with the model's "thinking strength", and at the real reason a fare question was answered wrongly.

CPU
Core i7-7800X
RAM
64 GB (125 GiB for the 5 November recheck)
GPU
RTX 3060 + RTX 5060 Ti (layer split)
Strata
0.2.4.2 (official)
Summary

Summary and contents

Summary

  • Thinking time did not correlate with accuracy. When transcribing tables, the setting that thinks longest gave both the worst result (32 errors, about 33 minutes for two tables) and the best one. In one run it repeated the same passage 16 times while thinking.
  • Thinking raised confidence, not correctness. On a question that needs a rule applied, light thinking stated its conclusion firmly all three times, and all three times it misread the rule. No thinking and the strongest thinking did not decide. This question depends on interpretation and the document alone does not settle it, so not deciding is the good answer (recheck on 4 November).
  • Instructions meant for any document did not improve accuracy. On 4 November we confirmed the same tendencies on Strata 0.2.10 and also tried another document (a home-appliance manual). All runs, including those before Strata, are in one list.
  • The wrong fare answer was not the model's fault. "Which column is which station" was lost when the table was turned into search text. With that fixed, the model answers correctly without thinking in 18–45 s (before: 9–21 minutes with thinking, and one wrong answer in three).
  • Both models read the tables about equally well. The differences came from one model declaring its own good readings "unreadable" (so the pipeline dropped them) and from an output limit that was too small.
  • A much larger answer model did not answer more questions right. Of 14 questions, the current model answered 13 right and decisively, the larger one 10. The larger model never stated a wrong answer, but left the answer open when the evidence lacked the question's own words, and took about 3.6 times as long on this PC.
  • Changing only the PC changed the answer to the interpretive question. On 5 November we ran the same tests again on the i7 PC, raised to 125 GiB of RAM. Table reading and the 13 factual questions came out almost the same, and answers were 10–30% faster. On the correspondence question, though, Qwen and llm-jp (the answer model before Strata) said "cannot buy" firmly all 3 times, with the same weights at temperature 0. DeepSeek never said it firmly in 16 runs on two PCs.

Verdict

Overall assessment

Taking all the tests on this page together (25 October to 5 November 2026): this RAG is practical for looking up facts, and the best answer model is Qwen (Qwen3.8-Flash-Next) on Strata without thinking. It cannot yet be relied on for questions that need a rule applied or interpreted.

AreaRatingBasis
Looking up facts (amounts, numbers, whether something is allowed as written)◎Qwen at strength 0 answered all 13 questions on this source right and decisively on every version, and all 14 on a home-appliance manual that was not looked at while building the system. No answer stated a wrong conclusion.
Table reading (building the index)○Ordinary tables are about equal for every model (match 0.85–0.90). On staircase fare tables Strata's Qwen was best with 6–8 errors; before Strata there were 214 errors or an empty reading.
Applying and interpreting rules△On the correspondence-school question the answer varied between runs with the same settings, and thinking made a firm "cannot buy" more likely (5 of 6 runs). Adding instructions meant for any document did not fix it. On another PC on 5 November, Qwen said it firmly all 3 times even at strength 0.
Speed○26 s a question (Qwen, 0.2.10; 19 s on the i7 PC on 5 November), most of it the search on the CPU (24–28 s).
Independence from the document○The prompts for answering, planning the search and reading tables contain no words specific to this source. The staircase rewrite and prompt A's example still carry the idea of a fare table.
◎ usable as is ○ usable with caveats △ not yet reliable.

Answer models

Model13 questions: right and decisiveLeft open or no answerMean timeCharacter
Qwen3.8-Flash-Next (Strata)13026 sFast, accurate and decisive; wavers only on the interpretive question.
DeepSeek-V4.1-Flash (Strata)10352 sNever states anything false, but hedges when the evidence does not name the thing word for word.
llm-jp-4.1-8b-thinking (before Strata)11239 sClose in accuracy; its reasoning can run into the answer length limit and get cut off.
Same search and prompt, the 13 questions besides the correspondence one, strength 0. Times are the mean of 11 questions, search included (0.2.10 for A and C; llm-jp on one GPU).

What Strata changed

Weaknesses and open issues

Next steps


Findings

Findings

No correlation

Longer thinking, no better accuracy

Raising the thinking strength made runs up to more than ten times longer, but errors did not go down. Repeating the same setting gave widely different results.

First 10 %

The answer is set early

In long thinking runs the answer was fixed within roughly the first tenth; the rest repeated the same checks. A misreading at the start was not corrected by thinking more.

3 of 3 wrong

Confident is not correct

On a question that needs a rule applied, light thinking gave the same wrong answer with confidence all three times. One run without thinking pointed out the exclusion clause correctly. On plain look-up questions, thinking made no difference.

18 s

The cause was the table's text form

With staircase fare tables stored as "origin → destination: fare" lines, the model answered correctly without thinking. Before, it took 9–21 minutes with thinking.

Conclusion: the shape of the source matters more than thinking time

What decided RAG accuracy

ThinkingLength does not track accuracy; it mainly narrows the spread

Light thinking made repeated runs collapse less often, but it could also turn a misread rule into a confident answer. Strong thinking took far longer, fell into repeating itself, or would not commit to a conclusion.

SourceWhether the table's meaning survives in the text

If the search text no longer says what each column means, the model can only guess, and whether it guesses right is down to luck.


Setup

Source and setup


Table reading

Table reading: two models

Ten approved tables (forms, distance, fare and dangerous-goods tables) were read by both models, without thinking.

TableQwengemma
Distance and fare tables (3)0.99–1.000.99–1.00
Fares to a connecting JR line1.000.97
Limited-express fares (upper)0.970.98
Limited-express fares (lower)0.970.89 *
Time per table (mean)106 s52 s
Number F1. * gemma flagged this table and a dangerous-goods table "unreadable" itself, so the pipeline dropped them; scored here ignoring the flag.

Excerpt

See it: reading the fare tables

For the two limited-express fare tables where the gap was largest: the printed table, and each model's reading exactly as written. A cell differs from the fares copied by hand from the image. A fare left out cannot be marked in a cell, so it is counted as "missing" next to the error count. Rows left entirely empty are not shown.

Limited-express fares (upper): from Shinjuku

Limited-express fares (upper): from Shinjuku
The printed table (PDF cut out at 144 dpi; tap for full size)
gemma, strength 0Fare errors 0 · 69 s
小田原線 江ノ島線
新宿4504504506006007007009501,150600700700
5005005006506507507501,0001,200650750750
新百合ヶ丘450450450450450600700900 450450600
500500500500500650750950 500500650
町田450450450450450700900
500500500500500750950
相模大野450450450450700900 450450450
500500500500750950 500500500
海老名450450450600800
500500500650850
本厚木450450600800
500500650850
伊勢原450450650
500500700
秦野450650
500700
小田原
200
箱根
湯本
大和 450450
500500
藤沢 450
500
片瀬
江ノ島
上段チケットレス特別急行料金 大人
下段特別急行料金 大人
Qwen, strength 0Fare errors 2 · 71 s (2 missing)
新宿4504504506006007007009501,150600700700
5005006506507507501,0001,200650750750
新百合ヶ丘450450450450450600700900450450600
500500500500500650750950500500650
町田450450450450450700900———
500500500500500750950———
相模大野450450450450700900450450450
500500500750950500500500
海老名450450450600800———
500500500650850———
本厚木450450600800———
500500650850———
伊勢原450450650———
500500700———
秦野450650———
500700———
小田原-———
200———
箱根湯本———
大和450450
500500
藤沢450
500
片瀬江ノ島
Qwen, strength 3Fare errors 12 · 927 s (3 missing)
新宿4504504506006007007009501,150600700700
新宿5005006506507507501,0001,200650750750
新百合ヶ丘450450450450600700700900450450600
新百合ヶ丘500500500500650750950500500650
町田450450450450600700900450450600
町田500500500500650750950500500650
相模大野450450450450700900450450450
相模大野500500500500750950500500500
海老名450450450600800−−−
海老名500500500650850−−−
本厚木450450600800−−−
本厚木500500650850−−−
伊勢原450450650−−−
伊勢原500500700−−−
秦野450650−−−
秦野500700−−−
小田原−−−−
小田原200−−−
箱根湯本−−−
箱根湯本−−−
大和450450
大和500500
藤沢450
藤沢500
片瀬江ノ島
Qwen, strength 3, prompt AFare errors 2 · 1191 s (1 missing)
新宿4504504506006007007009501,150600700700
新宿5005006506507507501,0001,200650750750
新百
合ヶ丘
450450450450450600700900450450600
新百
合ヶ丘
500500500500500650750950500500650
町田450450450450450700900———
町田500500500500500500750950———
相模大野450450450450700900450450450
相模大野500500500500750950500500500
海老名450450450600800———
海老名500500500650850———
本厚木450450600800———
本厚木500500650850———
伊勢原450450650———
伊勢原500500700———
秦野450650———
秦野500700———
小田原-———
小田原200———
箱根
湯本
———
箱根
湯本
———
大和450450
大和500500
藤沢450
藤沢500
片瀬
江ノ島
上段チケットレス特別急行料金 大人
下段特別急行料金 大人
DeepSeek, strength 0Fare errors 14 · 75 s
新宿4504504506006007007009501,150600700700
新宿5005005006506507507501,0001,200650750750
新百合ヶ丘450450450450450600700900450450600
新百合ヶ丘500500500500500650750950500500650
町田450450450450450700900———
町田500500500500500750950———
相模大野450450450450450700900450450450
相模大野500500500500500750950500500500
海老名450450450450600600800———
海老名500500500500650650850———
本厚木450450450600600800———
本厚木500500500650650850———
伊勢原450450450650———
伊勢原500500500700———
秦野450450650———
秦野500500700———
小田原————
小田原200———
箱根湯本———
大和450450
大和500500
藤沢450
藤沢500
片瀬江ノ島
Earlier vision model Qwen2.5-VL-7B (before Strata), whole pageFare errors 118 · 73 s (for the whole page) (9 missing)
新宿4504504506006007007009501,150600700700
5005005006506507507501,0001,200650750750
新百合ヶ丘450450450450450450600700900450450600
500500500500500500650750950500500650
町田450450450450450450700900900450450450
500500500500500500750950950500500500
相模大野450450450450450450600800800450450450
500500500500500500750950950500500500
海老名450450450450450450600800800450450450
500500500500500500650850850500500500
本厚木450450450600800800800800800450450450
500500500650850850850850850500500500
伊勢原500500500700700700700700700500500500
秦野450450450600700700700700700500500500
小田原200200200200200200200200200200200200
箱根湯本450450450450450450450450450450450450
大和500500500500500500500500500500500500
藤沢500500500500500500500500500500500500
片瀬江ノ島

Limited-express fares (lower): from Kita-Senju

Limited-express fares (lower): from Kita-Senju
The printed table (PDF cut out at 144 dpi; tap for full size)
gemma, strength 0Fare errors 56 · 70 s (6 missing)flagged "unreadable" by the model itself
北千住〜小田原・箱根湯本北千住〜片瀬江ノ島
北千住660
710
660
710
660
710
660
710
810
860
810
860
810
860
1,160
1,210
1,360
1,410
910
960
910
960
大手町660
710
660
710
660
710
660
710
810
860
810
860
810
860
1,160
1,210
1,360
1,410
910
960
910
960
甕ケ間660
710
660
710
660
710
660
710
810
860
810
860
810
860
1,160
1,210
1,360
1,410
910
960
910
960
表参道660
710
660
710
660
710
660
710
810
860
810
860
810
860
1,160
1,210
1,360
1,410
910
960
910
960
450
500
450
500
450
500
450
500
600
650
700
750
900
950
600
650
600
650
成城学園前
450
500
450
500
450
500
450
500
600
650
700
750
900
950
新百合ヶ丘
450
500
450
500
450
500
450
500
600
650
700
750
900
950
町田
450
500
450
500
450
500
450
500
600
650
700
750
900
950
相模大野
450
500
450
500
450
500
450
500
600
650
700
750
900
950
450
500
450
500
海老名
450
500
450
500
450
500
450
500
600
650
700
750
800
850
本厚木
450
500
450
500
450
500
450
500
600
650
700
750
800
850
伊勢原
450
500
450
500
450
500
450
500
650
700
小田原
ー
200
箱根湯本
ー
藤沢
450
500
片瀬江ノ島
Qwen, strength 0Fare errors 4 · 76 s (4 missing)
北千住660
710
660
710
660
710
660
710
810
860
810
860
810
860
1,160
1,210
1,360
1,410
910
960
910
960
大手町660
710
660
710
660
710
660
710
810
860
810
860
810
860
1,160
1,210
1,360
1,410
910
960
910
960
霞ヶ関660
710
660
710
660
710
660
710
810
860
810
860
810
860
1,160
1,210
1,360
1,410
910
960
910
960
表参道660
710
660
710
660
710
660
710
810
860
810
860
810
860
1,160
1,210
1,360
1,410
910
960
910
960
成城
学園前
450
500
450
500
450
500
450
500
450
500
600
650
700
750
900
950
600
650
600
650
新百
合ヶ丘
450
500
450
500
450
500
450
500
450
500
700
750
900
950
——
町田450
500
450
500
450
500
450
500
700
750
900
950
——
相模
大野
450
500
450
500
450
500
700
750
900
950
450
500
450
500
海老名450
500
600
650
800
850
——
本厚木450
650
600
850
——
伊勢原450
500
650
700
——
小田原—
200
——
箱根湯本——
藤沢450
500
片瀬
江ノ島
Qwen, strength 3Fare errors 20 · 1060 s
北千住660
710
660
710
660
710
660
710
810
860
810
860
810
860
1,160
1,210
1,360
1,410
910
960
910
960
大手町660
710
660
710
660
710
660
710
810
860
810
860
810
860
1,160
1,210
1,360
1,410
910
960
910
960
霞ヶ関660
710
660
710
660
710
660
710
810
860
810
860
810
860
1,160
1,210
1,360
1,410
910
960
910
960
表参道660
710
660
710
660
710
660
710
810
860
810
860
810
860
1,160
1,210
1,360
1,410
910
960
910
960
成城学園前成城学園前450
500
450
500
450
500
450
500
450
500
600
650
700
750
900
950
600
650
600
650
新百合ヶ丘新百合ヶ丘450
500
450
500
450
500
450
500
450
500
700
750
900
950
——
町田町田町田450
500
450
500
450
500
450
500
700
750
900
950
——
相模大野相模大野相模大野相模大野450
500
450
500
450
500
450
500
700
750
900
950
450
450
450
450
海老名海老名海老名海老名海老名450
500
450
500
450
500
450
500
600
650
800
850
—
本厚木本厚木本厚木本厚木本厚木本厚木450
500
450
500
450
500
600
650
800
850
—
伊勢原伊勢原伊勢原伊勢原伊勢原伊勢原伊勢原450
500
450
500
450
500
650
700
—
小田原小田原小田原小田原小田原小田原小田原小田原450
500
450
500
—
200
—
箱根湯本箱根湯本箱根湯本箱根湯本箱根湯本箱根湯本箱根湯本箱根湯本箱根湯本——
藤沢藤沢藤沢藤沢藤沢藤沢藤沢藤沢藤沢藤沢450
500
片瀬江/島片瀬江/島片瀬江/島片瀬江/島片瀬江/島片瀬江/島片瀬江/島片瀬江/島片瀬江/島片瀬江/島片瀬江/島片瀬江/島
Qwen, strength 3, prompt AFare errors 1 · 1192 s
北千住660
710
660
710
660
710
660
710
810
860
810
860
810
860
1,160
1,210
1,360
1,410
910
960
910
960
大手町660
710
660
710
660
710
660
710
810
860
810
860
810
860
1,160
1,210
1,360
1,410
910
960
910
960
霞ヶ関660
710
660
710
660
710
660
710
810
860
810
860
810
860
1,160
1,210
1,360
1,410
910
960
910
960
表参道660
710
660
710
660
710
660
710
810
860
810
860
810
860
1,160
1,210
1,360
1,410
910
960
910
960
成城
学園前
450
500
450
500
450
500
450
500
450
500
600
650
700
750
900
950
600
650
600
650
新百
合ヶ丘
450
500
450
500
450
500
450
500
450
500
700
900
900
950
--
町田450
500
450
500
450
500
450
500
700
750
900
950
--
相模
大野
450
500
450
500
450
500
700
750
900
950
450
500
450
500
海老名450
500
450
500
600
650
800
850
--
本厚木450
500
600
650
800
850
--
伊勢原450
500
650
700
--
小田原-
200
--
箱根湯本--
藤沢450
500
片瀬
江ノ島
DeepSeek, strength 0Fare errors 22 · 91 s (1 missing)
北千住6606606606608108108101,1601,360910910
7107107107108608608601,2101,410960960
大手町6606606606608108108101,1601,360910910
7107107107108608608601,2101,410960960
霞ヶ関6606606606608108108101,1601,360910910
7107107107108608608601,2101,410960960
表参道6606606606608108108101,1601,360910910
7107107107108608608601,2101,410960960
成城学園前450450450450450450600700900600600
500500500500500500650750950650650
新百合ヶ丘450450450450450450700900——
500500500500500500750950——
町田450450450450450450700900——
500500500500500500750950——
相模大野450450450450450450700900450450
500500500500500500750950500500
海老名450450450600800——
500500500650850——
本厚木450450600800——
500500650850——
伊勢原450450650——
500500700——
小田原—200——
箱根湯本——
藤沢450
500
片瀬江ノ島
Earlier vision model Qwen2.5-VL-7B (before Strata), whole pageFare errors 96 · 73 s (for the whole page) (12 missing)
北千住6606606606608108101,1601,360910910
7107107107108608601,2101,410960960
大手町6606606606608108101,1601,360910910
7107107107108608601,2101,410960960
霞ヶ関6606606606608108101,1601,360910910
7107107107108608601,2101,410960960
表参道6606606606608108101,1601,360910910
7107107107108608601,2101,410960960
成城学園前450450450450450600700900600600
500500500500500650750950650650
新百合ヶ丘450450450450450700900900450450
500500500500500750950950500500
町田450450450450450700900900450450
500500500500500750950950500500
相模大野450450450450450700900900450450
500500500500500750950950500500
海老名450450450450450700900900450450
500500500500500750950950500500
本厚木450450450600800800800800800450450
500500500650850850850850850500500
伊勢原500500500700700700700700700500500
小田原200200200200200200200200200200200
箱根湯本450450450450450450450450450450450
藤沢450450450450450450450450450450450
片瀬江ノ島
92

Thinking strength

Thinking strength and accuracy

The two hardest limited-express fare tables (48 fare rows) were read with different thinking strengths, prompts and repeats. Then the strength was varied for the RAG answers too.

Transcribing a table

SettingErrors (2 tables)Time (2 tables)
gemma, strength 056139 s
Qwen, strength 06147 s
Strength 0, prompt B4161 s
Strength 0, prompt A4125 s
Strength 14 → 17 (same setting, twice)about 310 s
Strength 1, prompt B7623 s
Strength 1, prompt A7341 s
Strength 25517 s
Strength 3321,987 s
Strength 3, prompt A32,383 s
DeepSeek, strength 036166 s
DeepSeek, strength 1572,286 s
Rows not marked gemma or C use Qwen. DeepSeek was measured on 4 November 2026, with a newer Strata version on a different PC. Sampling temperature 0. Measured again on the upper table alone, strength 3 made 1 error (1,066 s).

With a little randomness (temperature 0.6) and three runs of each setting, the spread becomes visible.

Setting (temperature 0.6)Run 1Run 2Run 3
Strength 074073
Strength 14416
Errors (2 tables).

Inside a long thinking run

The same passage, 16 times

We saved the thinking of a strength-3 run that took about 18 minutes. A passage of about 1,400 characters was repeated word for word 16 times, until the server closed the thinking. Every draft written inside the loop was identical to the final answer.

It kept asking "which row is this table's header?" and "which side is the origin and which the destination?". This fare table has no header row, so the question had no answer and the model circled the same check. The answer had been settled in roughly the first tenth of the thinking.

Answering questions

On the same source, Qwen answered three questions of different kinds.

QuestionStrength 0Strength 1 (3 runs)Strength 3
"I attend a correspondence high school. Can I buy a student commuter pass?" (apply a rule)63–88 s. Did not decide; one run correctly showed the exclusion clause (good answer)102–133 s. "No" all three times, stated firmly (wrong)440 s. Did not decide
"What is the operating distance from Shinjuku to Minami-Shinjuku?" (look up)43 s. Right47 s. Right—
"What is the ticketless limited-express fee from Kita-Senju to Hon-Atsugi?" (read a fare table)47 s. Declined: "the table's columns can't be identified"549–1,247 s. Right 2 of 3; once the value of another column1,159 s. Right number, wrong explanation
First question: the commuter-pass article excludes only students of the correspondence university, not correspondence high-school students. By the text the commuter pass is not ruled out, and there are also discounted tickets for classroom sessions and exams. Whether one can actually be bought depends on the school and the railway, so this document alone does not decide it; not deciding is the good answer (recheck on 4 November).

Root cause

The real cause of the wrong fare answer

The wrong and the right runs received the same evidence (the same eight passages in the same order). They differed only in how the model guessed the table's columns.


Fix

The fix and its result

At ingestion, staircase tables are now detected and stored one pair per line, such as "Kita-Senju → Hon-Atsugi: 810". Detection uses only the table's shape, never station names, units or other content of this source.

Kita-Senju to Hon-Atsugi feeBeforeAfter
Strength 047 s. Declined18–45 s. Right (3 runs)
Strength 1549–1,247 s. Right 2 of 399 s. Right
The distance question (a square table, already stored as pairs) gave the same answer before and after.

Answer models

Comparing answer models: Qwen and DeepSeek

We swapped the answer model for another, much larger model that also runs on Strata, DeepSeek-V4.1-Flash and asked 14 questions on the same source (the version with the tables fixed). The right answers were taken from the rule text before asking. Both models ran at strength 0.

QuestionRight answer (rule text)QwenDeepSeek
Ticketless limited-express fee, Kita-Senju to Hon-Atsugi810 yenright · 45 sright · 83 s
Operating kilometres, Shinjuku to Minami-Shinjuku0.8 kmright · 34 sright · 50 s
Can a correspondence high-school student buy a student commuter pass?the document alone does not decide (not ruled out by the text; the discounted tickets can be bought)did not decide, showed that only university students are excluded (good answer) · 63 sdid not decide (without the exclusion clause) · 87 s
Student pass for a 10-year-old, 3 months2,280 yenright · 57 sgave the number, then left it open · 169 s
One adult with three 5-year-oldstwo free, one pays the child fareright · 51 sright · 136 s
Platform ticketadult 140 yen, child 70 yenright · 27 sright · 108 s
Refund fee for an unused ordinary ticket100 yen (through ticket 220 yen)right · 34 sright · 122 s
Limited-express ticket refund: deadline and feeuntil the train leaves · 100 yenright · 37 sright · 135 s
Fractions of a kilometrerounded up to 1 kmright · 24 sright · 114 s
Barrier-free surcharge on a commuter pass600 yen a monthright · 24 sright · 115 s
Ticket expires during the journeyvalid to the destination if you do not break the journeyright · 46 sright · 139 s
Can I take my cat in a carrier?yes (free, sides up to 120 cm in total, up to 10 kg)right · 38 sconditions right, then left it open · 170 s
How many suitcases can I bring free?two (up to 30 kg, up to 2 m long)right · 24 sconditions right, then left it open · 134 s
Refund fee for a commuter pass before its start date220 yenright · 24 sright · 111 s
Of the 13 questions other than the correspondence one, Qwen answered 13 right and decisively, DeepSeek 10. On the correspondence question neither decided, which is the good answer there (recheck on 4 November). Neither stated a wrong answer. DeepSeek took about 3.6 times as long on average.

Recheck (Nov 4)

Recheck on 4 November: a newer version, and instructions for any document

We asked the same questions again with Strata 0.2.10 on a PC with two CPUs. We also added two instructions meant for any document to the answer prompt and tested whether they help. Instructions tuned to one document were not allowed.

The correspondence-school question depends on interpretation

Reading the articles again: the commuter-pass article excludes only students of the correspondence university, not correspondence high-school students. Whether such a student can actually buy a commuter pass, however, depends on the school (whether it is a designated school and issues a commuting certificate) and on the railway's practice. In practice, the right answer is that this document alone does not decide it. So we scored every answer to this question again. A good answer shows that the discounted tickets can be bought and that the commuter pass is not ruled out by the text (it depends on the school), without deciding. An answer that states "no" firmly is wrong.

ConditionStrataRunsGood answerUndecided but incompleteStated "no" firmly
Qwen, strength 00.2.4.22110
Qwen, strength 10.2.4.23003
Qwen, strength 30.2.4.21010
DeepSeek, strengths 0 and 10.2.4.24040
Qwen, strength 00.2.94004
Qwen, strength 00.2.103210
Qwen, strength 10.2.103102
DeepSeek, strength 00.2.103030
DeepSeek, strength 10.2.103210
llm-jp (the answer model before Strata)before3030
A "good answer" shows the discounted tickets and that the commuter pass is not ruled out by the text (only university students are excluded) or depends on the school, and does not decide. 0.2.4.2 ran on the earlier PC, 0.2.9 and 0.2.10 on the two-CPU PC. The 9 further runs with the instructions below all stated a wrong conclusion firmly. llm-jp answered all 3 times "yes, if you meet the condition (the school's commuting certificate)" without the discounted tickets, so it is counted as incomplete. The results from another PC on 5 November are in the next section.

Other questions and speed

Adding instructions meant for any document

We added two instructions to the answer prompt. Instruction 1: do not conclude "cannot" unless the evidence clearly states a prohibition or exclusion, and separate what is written from what is not. Instruction 2: when a general rule and a rule for a specific case appear together, do not treat the specific rule as overriding the general one beyond what it says. Besides this source, we used 14 questions on a home-appliance manual (204 pages) that was not looked at while writing the instructions. Right answers were fixed from the text before asking.

Setting (Qwen, strength 0, 0.2.9)Manual, 14 questionsThis source, 13 questionsCorrespondence question (3 runs)
Current prompt1413"no" all three times
Instruction 11413"you buy the discounted tickets instead" all three times
Instruction 21412"you buy the discounted tickets instead" all three times
Instructions 1 and 21413"you buy the discounted tickets instead" all three times
Number answered right. The manual's 14 questions include 7 that it clearly says cannot be done, 2 that can be done with conditions, 3 it does not mention, and 2 plain facts.

Recheck (Nov 5)

Recheck on 5 November: on another PC, the interpretive question got different answers

We raised the earlier PC (Core i7-7800X, "i7" below) to 125 GiB of RAM and moved the two GPUs from the two-CPU PC into it. Then we ran the same Strata 0.2.10, settings, index and questions as on 4 November. Only the PC was different.

Setting13 questions besides the correspondence one: right and decisiveMean time, 11 questionsCorrespondence: good answersCorrespondence: firm "cannot buy"
Qwen, strength 013 (13)19 s (26 s)0 (2)3 (0)
Qwen, strength 1——0 (1)3 (2)
DeepSeek, strength 011 (10)46 s (52 s)2 (0)0 (0)
DeepSeek, strength 1——2 (2)0 (0)
llm-jp (the answer model before Strata)11 (11)28 s (39 s)0 (0)3 (0)
Values on the i7. The values in brackets are from the two-CPU PC on 4 November. The correspondence question was asked 3 times in each setting. One Qwen strength-0 run answered "not a commuter pass; you can buy the discounted tickets". That is the same rewording as with instruction 1 above, so it is counted as firm. Times are per question, search included.
Recheck (Nov 5, evening)

Evening of 5 November: switching Qwen's experts to 4-bit NVFP4 did not change answer quality

We swapped Qwen3.8-Flash-Next's experts from UD-Q4_K_XL to NVFP4 (NVIDIA's 4-bit format, in the GPTQ pack made for Strata). The experts get 12% smaller, which makes long prompts and decoding faster (research page). To see whether the smaller weights cost answer quality, we compared the two on the same i7 PC with the same index and questions, in the same session. Both used the faster layer split found earlier that day.

CheckUD-Q4_K_XLNVFP4
13 questions besides the correspondence one: right and decisive (strength 0)1313
Mean time, 11 questions (strength 0)16 s15 s
Correspondence: good answers / runs (strength 0)0 / 30 / 3
Correspondence: good answers / runs (strength 1)0 / 50 / 5 (one run hedged: "the material does not settle it")
Table reading: F1 over 10 crops0.870.87
Table reading: mean per crop95 s87 s
Staircase fare table errors (strength 0, upper + lower)1 + 23 + 0
Staircase fare table errors (strength 1, upper + lower)1 + 71 + 0
One run each (the correspondence question 3 or 5 times). Times are per question, search included. Scoring as in the sections above.

All runs

All runs in one list (including before Strata)

Table reading on the same 10 crops with the same scoring, and answers to the same questions, side by side, with the version and the PC. The earlier vision model is Qwen2.5-VL-7B-Instruct.

Table reading

ConditionStrataMatch on 10 crops (F1)Mean per cropStaircase fare errors (upper + lower)Time for those 2
Earlier vision model (Qwen2.5-VL-7B), whole pagebefore——118 + 9673 s (one page)
gemma, strength 0 (output cap 256, as deployed)before——both cut off, marked "unreadable"9 + 9 s
gemma, strength 0 (output cap 8192)before0.72 (0.90 ignoring its flag)52 s0 + 5669 + 70 s
Qwen, strength 00.2.4.20.86106 s2 + 471 + 76 s
Qwen, strength 10.2.4.2——1 + 3 (1 + 16 on a rerun)195 + 125 s
Qwen, strength 30.2.4.2——12 + 20927 + 1,060 s
Qwen, strength 00.2.100.8586 s2 + 669 + 71 s
Qwen, strength 10.2.10——2 + 13130 + 105 s
DeepSeek, strength 00.2.100.8896 s14 + 2275 + 91 s
DeepSeek, strength 10.2.10——19 + 381,160 + 1,126 s
Qwen, strength 0 (i7)0.2.100.85100 s3 + 486 + 81 s
Qwen, strength 1 (i7)0.2.10——5 + 169 (a format slip; 5 + 3 once fixed)80 + 108 s
DeepSeek, strength 0 (i7)0.2.100.8696 s20 + 1893 + 86 s
DeepSeek, strength 1 (i7)0.2.10——22 + 381,224 + 1,148 s
Qwen, strength 0 (i7, faster split, UD)0.2.100.8795 s1 + 284 + 80 s
Qwen, strength 1 (i7, faster split, UD)0.2.10——1 + 796 + 55 s
Qwen, strength 0 (i7, NVFP4)0.2.100.8787 s3 + 075 + 70 s
Qwen, strength 1 (i7, NVFP4)0.2.10——1 + 049 + 57 s
Match is the all-token F1. gemma marked 2 of the 10 crops "unreadable" itself, and the pipeline dropped them (0.72); scored ignoring the flag it is 0.90. gemma and 0.2.4.2 ran on the earlier PC (GPU), 0.2.10 on the two-CPU PC, and "i7" is the earlier PC raised to 125 GiB of RAM on 5 November. The 169 for Qwen at strength 1 on the i7 is a format slip: it joined the two fares of a cell with a space instead of a line break. Scored with the spaces turned into line breaks, it is 3. Errors count wrong values among the 22 + 26 fare rows.

Answers (questions on this source)

ConditionStrata13 questions besides the correspondence one: right and decisiveLeft openMean time, 11 questionsCorrespondence question (good / runs)
llm-jp (the answer model before Strata)before111 (and 1 with no answer)39 s0 / 3
Qwen, strength 00.2.4.213035 s1 / 2
DeepSeek, strength 00.2.4.2103132 s0 / 2
Qwen, strength 00.2.913024 s0 / 4
Qwen, strength 00.2.1013026 s2 / 3
DeepSeek, strength 00.2.1010352 s0 / 3
llm-jp (i7)before111 (and 1 wrong)28 s0 / 3
Qwen, strength 0 (i7)0.2.1013019 s0 / 3
DeepSeek, strength 0 (i7)0.2.1011246 s2 / 3
Qwen, strength 0 (i7, faster split, UD)0.2.1013016 s0 / 3
Qwen, strength 0 (i7, NVFP4)0.2.1013015 s0 / 3
Apart from the correspondence question, the only answer that stated a wrong conclusion was one from llm-jp on the i7 (the three infants). 0.2.4.2 ran on the earlier PC, 0.2.9 and 0.2.10 on the two-CPU PC, and "i7" is the earlier PC raised to 125 GiB of RAM on 5 November. llm-jp is the answer model this search system used before Strata; it ran on one GPU and was scored on 4 November with the same search and prompt. Times are per question, search included.

Upstream

Reports to Strata

Reports submitted to the official Strata project during this research: measurements in the official benchmark format, and bugs with reproduction steps and a proposed fix.


Limits

Conditions and limits