Garbled text: nothing is lost
pilot, 15 Nov 2026 · exploratory · 6 cross-script layouts, 50 items each, one call per item
In every setting plotted, decoding has a cost: with reasoning on, reasoning tokens and, in part, accuracy; with it off, accuracy.
7 of 8
settings with reasoning off score within ten points of the floor.
21 to 67 times
the reasoning tokens of native text: what the 8 settings that recover spend on the same items typed on the wrong layout.
4 to 9 points
below native text: where the best 4 settings still end, with reasoning on.
| setting | as written | wrong layout | change, points | no valid answer | reasoning tokens per call, wrong layout (mean) |
|---|---|---|---|---|---|
gpt-6-astra effort low | 96.7% [94.0, 98.2] | 92.3% [88.8, 94.8] | −4.3 [−7.7, −1.7] | 0 of 300 | 105 |
gpt-6.1-sol effort high | 96.0% [93.1, 97.7] | 92.0% [88.4, 94.6] | −4.0 [−6.3, −2.0] | 0 of 300 | 632 |
gpt-6.1-sol effort low | 95.3% [92.3, 97.2] | 90.7% [86.8, 93.5] | −4.7 [−7.3, −2.0] | 0 of 300 | 163 |
claude-opus-5-5 effort low | 97.0% [94.4, 98.4] | 88.0% [83.8, 91.2] | −9.0 [−13.7, −5.0] | 0 of 300 | 420 |
kimi-k3-1-pro-261024 thinking on † | 95.0% [91.9, 96.9] | 78.0% [73.0, 82.3] | −17.0 [−20.3, −14.0] | 45 of 300 | 10,894 |
gpt-6.1-terra effort high | 94.7% [91.5, 96.7] | 76.0% [70.9, 80.5] | −18.7 [−25.0, −12.3] | 0 of 300 | 972 |
claude-sonnet-5-5 effort high | 96.0% [93.1, 97.7] | 70.7% [65.3, 75.5] | −25.3 [−31.0, −19.7] | 7 of 300 | 1,174 |
gpt-6-luna effort high | 93.7% [90.3, 95.9] | 65.3% [59.8, 70.5] | −28.3 [−34.3, −22.7] | 0 of 300 | 5,156 |
gpt-6.1-terra effort none | 94.7% [91.5, 96.7] | 57.0% [51.3, 62.5] | −37.7 [−44.7, −30.7] | 0 of 300 | 0 |
doubao-seed-2-1-lite-260915 thinking on | 94.7% [91.5, 96.7] | 48.7% [43.1, 54.3] | −46.0 [−54.7, −37.0] | 0 of 300 | 9,587 |
claude-sonnet-5-5 thinking off | 95.0% [91.9, 96.9] | 44.3% [38.8, 50.0] | −50.7 [−58.3, −43.0] | 6 of 300 | 0 |
kimi-k3-1-pro-261024 thinking off | 94.3% [91.1, 96.4] | 42.0% [36.6, 47.7] | −52.3 [−60.3, −44.7] | 0 of 300 | 0 |
doubao-seed-2-1-lite-260915 thinking off | 93.3% [89.9, 95.6] | 40.0% [34.6, 45.6] | −53.3 [−61.3, −45.3] | 0 of 300 | 0 |
doubao-seed-2-1-pro-260915 thinking off | 93.7% [90.3, 95.9] | 40.0% [34.6, 45.6] | −53.7 [−62.7, −44.3] | 0 of 300 | 0 |
deepseek-v4-1-flash-261017 thinking off | 93.0% [89.5, 95.4] | 37.7% [32.4, 43.3] | −55.3 [−61.7, −48.3] | 0 of 300 | 0 |
gpt-6-luna effort none | 92.3% [88.8, 94.8] | 36.7% [31.4, 42.3] | −55.7 [−63.0, −48.0] | 0 of 300 | 0 |
deepseek-v4-1-flash-261017 thinking on † | 94.7% [91.5, 96.7] | 36.0% [30.8, 41.6] | −58.7 [−63.7, −53.3] | 164 of 300 | 23,229 |
deepseek-v4-pro-ga-260813 thinking off | 94.0% [90.7, 96.2] | 33.7% [28.6, 39.2] | −60.3 [−68.7, −52.3] | 0 of 300 | 0 |
glm-5-3-flash-260828 thinking off requested † | 90.3% [86.5, 93.2] | 32.0% [27.0, 37.5] | −58.3 [−65.0, −51.7] | 43 of 300 | 1,582 |
Floor without reading: expected accuracy of always picking the option with the largest share of its letter triples found in the passage, ties sharing credit, computed on the wrong-layout text: 34.8% on average, 29% to 41% by language. Chance: 25%.
Layouts: Russian, Korean, Hebrew, Thai, Arabic and Egyptian Arabic (the last two share one keyboard). Every reply counts; one cut off at the token limit counts as wrong. Reasoning tokens as each API reports them.
† Many wrong-layout replies reached the output-token limit and count as wrong: 164 of 300 (deepseek-v4-1-flash-261017, thinking on), 45 of 300 (kimi-k3-1-pro-261024, thinking on), 43 of 300 (glm-5-3-flash-260828, thinking off requested).
Listed under reasoning on: glm-5-3-flash-260828, which reports reasoning tokens although thinking off was requested.
Not shown: claude-haiku-5-5, whose wrong-layout cells were not run in full.
Round trip: undoing the pilot’s wrong-layout converter restores 280 of 300 Russian texts (passages, questions, options) exactly; Latin letters already inside Russian text become ambiguous.
Hypothesis: reading wrong-layout text takes up to four steps. Models fail at different steps (a description, not a tested hypothesis).
-
57.0%86.7%
no sentencelayout named
Recognise that a key map applies, and which. One sentence naming the layout lifts
gpt-6.1-terra, reasoning off, from 57.0% to 86.7%; 6 other reasoning-off settings show no clear gain (one wording, no control sentence). -
94.0%40.7%88.0%
real Russian layoutrandom maps, mean of 3first map + note + letter table
Recall the map, or be given it. Same setting, Russian: 94.0% on the real layout, as on native text; 40.7% on equally reversible random maps; 88.0% on the first of them, given with a note and its letter table.
-
6 points
below native text, with the table
Apply it. Even given the table, the same setting stays 6 points below native text on the same 50 items.
-
76% to 87%
random maps, with reasoning
Decipher it, where no map is known. With reasoning, 4 settings read Russian typed through random maps at 76% to 87%, for about 10 to 24 times their real-layout reasoning tokens.
Decipher step: gpt-6.1-terra (effort high), gpt-6.1-sol (effort low and high) and gpt-6-astra (effort low).