Keyboard-state input

UKAIRS 2026 · Emerging Research · Poster UKAIRS26-89

Headline

Garbled text can be decoded, at a cost. Ambiguous text cannot.

typed US English (QWERTY)

привет hello

meant Russian ЙЦУКЕН

Typed with the computer set to US English (QWERTY): ghbdtn. Meant, on Russian ЙЦУКЕН: привет (hello).

Garbled. Russian typed on the wrong keyboard layout (English). Every keystroke survives, so undoing the key map gives the text back.

typed pinyin without tones

杯子bēizi cupor被子bèizi quilt

Typed: beizi. Meant: 杯子 (cup) or 被子 (quilt); the text does not say which.

Ambiguous. A Chinese word typed alone as pinyin without tones, before the IME picks the characters. Here one spelling stands for more than one word, and nothing in the text says which.

pilot exploratory results of 15 Nov 2026 and 16 Nov 2026

Keyboard-state input is text as typed, before or without conversion by an input method editor (IME). We tested 13 models in 21 settings (reasoning off or on) on four-option questions typed this way.

  • 7 of 8 settings with reasoning off score within ten points of the floor.
  • 21 to 67 times the reasoning tokens of native text: what the 8 settings that recover spend on the same items typed on the wrong layout.
  • 4 to 9 points below native text: where the best 4 settings still end, with reasoning on.

floor without reading 34.8% (29% to 41% by language): what letter overlap alone scores on the wrong-layout text. chance: 25%.

13 models · 21 settings · exploratory. Every result in the findings and the explorer comes from exploratory pilots: wrong layouts, 15 Nov 2026 (50 Belebele items per language, one call per item) and Chinese, 16 Nov 2026 (96 constructed items; one call per item, three in the follow-up). The main run has not started; its plan will be frozen and time-stamped before its first call.

Try the converter Read the findings Explore models

Try it: the same keys on two layouts

Type or paste text in the box. The page finds the keys that type it on the layout you meant and shows what the same keys give on the layout the computer is set to. Everything runs in this page; nothing you type is sent anywhere.

As typed

When the typed form loses information

No converter can undo this one. Typed alone as pinyin without tones, these spellings each stand for more than one word, and the text does not say which; the last one is a control with a single word above the frequency threshold.

beizi3 words

  • 杯子 bēizi cup
  • 被子 bèizi quilt
  • 辈子 bèizi lifetime

daikuan2 words

  • 贷款 dàikuǎn loan (borrowed money)
  • 带宽 dàikuān bandwidth

shuimian2 words

  • 睡眠 shuìmián sleep
  • 水面 shuǐmiàn water surface

guomin2 words

  • 国民 guómín citizens of a nation
  • 过敏 guòmǐn allergy

zifu2 words

  • 自负 zìfù conceited
  • 字符 zìfú character (text symbol)

jianpan1 word

  • 键盘 jiànpán keyboard

The first spelling and its words are the poster's example. From the Chinese pilot's item file; the items have not yet been checked by an independent native speaker. Toned forms are as the item file gives them.

Garbled text: nothing is lost

pilot, 15 Nov 2026 · exploratory · 6 cross-script layouts, 50 items each, one call per item

In every setting plotted, decoding has a cost: with reasoning on, reasoning tokens and, in part, accuracy; with it off, accuracy.

7 of 8

settings with reasoning off score within ten points of the floor.

21 to 67 times

the reasoning tokens of native text: what the 8 settings that recover spend on the same items typed on the wrong layout.

4 to 9 points

below native text: where the best 4 settings still end, with reasoning on.

Figure A. Each row: one setting (model, reasoning level), on native text and on the same items typed on the wrong layout.
Figure A as a table: six cross-script layouts pooled, 300 items and calls per cell; intervals 95%; reasoning tokens are means.
settingas writtenwrong layoutchange, pointsno valid answerreasoning tokens per call, wrong layout (mean)
gpt-6-astra effort low96.7% [94.0, 98.2]92.3% [88.8, 94.8]−4.3 [−7.7, −1.7]0 of 300105
gpt-6.1-sol effort high96.0% [93.1, 97.7]92.0% [88.4, 94.6]−4.0 [−6.3, −2.0]0 of 300632
gpt-6.1-sol effort low95.3% [92.3, 97.2]90.7% [86.8, 93.5]−4.7 [−7.3, −2.0]0 of 300163
claude-opus-5-5 effort low97.0% [94.4, 98.4]88.0% [83.8, 91.2]−9.0 [−13.7, −5.0]0 of 300420
kimi-k3-1-pro-261024 thinking on †95.0% [91.9, 96.9]78.0% [73.0, 82.3]−17.0 [−20.3, −14.0]45 of 30010,894
gpt-6.1-terra effort high94.7% [91.5, 96.7]76.0% [70.9, 80.5]−18.7 [−25.0, −12.3]0 of 300972
claude-sonnet-5-5 effort high96.0% [93.1, 97.7]70.7% [65.3, 75.5]−25.3 [−31.0, −19.7]7 of 3001,174
gpt-6-luna effort high93.7% [90.3, 95.9]65.3% [59.8, 70.5]−28.3 [−34.3, −22.7]0 of 3005,156
gpt-6.1-terra effort none94.7% [91.5, 96.7]57.0% [51.3, 62.5]−37.7 [−44.7, −30.7]0 of 3000
doubao-seed-2-1-lite-260915 thinking on94.7% [91.5, 96.7]48.7% [43.1, 54.3]−46.0 [−54.7, −37.0]0 of 3009,587
claude-sonnet-5-5 thinking off95.0% [91.9, 96.9]44.3% [38.8, 50.0]−50.7 [−58.3, −43.0]6 of 3000
kimi-k3-1-pro-261024 thinking off94.3% [91.1, 96.4]42.0% [36.6, 47.7]−52.3 [−60.3, −44.7]0 of 3000
doubao-seed-2-1-lite-260915 thinking off93.3% [89.9, 95.6]40.0% [34.6, 45.6]−53.3 [−61.3, −45.3]0 of 3000
doubao-seed-2-1-pro-260915 thinking off93.7% [90.3, 95.9]40.0% [34.6, 45.6]−53.7 [−62.7, −44.3]0 of 3000
deepseek-v4-1-flash-261017 thinking off93.0% [89.5, 95.4]37.7% [32.4, 43.3]−55.3 [−61.7, −48.3]0 of 3000
gpt-6-luna effort none92.3% [88.8, 94.8]36.7% [31.4, 42.3]−55.7 [−63.0, −48.0]0 of 3000
deepseek-v4-1-flash-261017 thinking on †94.7% [91.5, 96.7]36.0% [30.8, 41.6]−58.7 [−63.7, −53.3]164 of 30023,229
deepseek-v4-pro-ga-260813 thinking off94.0% [90.7, 96.2]33.7% [28.6, 39.2]−60.3 [−68.7, −52.3]0 of 3000
glm-5-3-flash-260828 thinking off requested †90.3% [86.5, 93.2]32.0% [27.0, 37.5]−58.3 [−65.0, −51.7]43 of 3001,582

Floor without reading: expected accuracy of always picking the option with the largest share of its letter triples found in the passage, ties sharing credit, computed on the wrong-layout text: 34.8% on average, 29% to 41% by language. Chance: 25%.

Layouts: Russian, Korean, Hebrew, Thai, Arabic and Egyptian Arabic (the last two share one keyboard). Every reply counts; one cut off at the token limit counts as wrong. Reasoning tokens as each API reports them.

† Many wrong-layout replies reached the output-token limit and count as wrong: 164 of 300 (deepseek-v4-1-flash-261017, thinking on), 45 of 300 (kimi-k3-1-pro-261024, thinking on), 43 of 300 (glm-5-3-flash-260828, thinking off requested).

Listed under reasoning on: glm-5-3-flash-260828, which reports reasoning tokens although thinking off was requested.

Not shown: claude-haiku-5-5, whose wrong-layout cells were not run in full.

Round trip: undoing the pilot’s wrong-layout converter restores 280 of 300 Russian texts (passages, questions, options) exactly; Latin letters already inside Russian text become ambiguous.

Hypothesis: reading wrong-layout text takes up to four steps. Models fail at different steps (a description, not a tested hypothesis).

  1. 57.0%86.7%

    no sentencelayout named

    Recognise that a key map applies, and which. One sentence naming the layout lifts gpt-6.1-terra, reasoning off, from 57.0% to 86.7%; 6 other reasoning-off settings show no clear gain (one wording, no control sentence).

  2. 94.0%40.7%88.0%

    real Russian layoutrandom maps, mean of 3first map + note + letter table

    Recall the map, or be given it. Same setting, Russian: 94.0% on the real layout, as on native text; 40.7% on equally reversible random maps; 88.0% on the first of them, given with a note and its letter table.

  3. 6 points

    below native text, with the table

    Apply it. Even given the table, the same setting stays 6 points below native text on the same 50 items.

  4. 76% to 87%

    random maps, with reasoning

    Decipher it, where no map is known. With reasoning, 4 settings read Russian typed through random maps at 76% to 87%, for about 10 to 24 times their real-layout reasoning tokens.

Decipher step: gpt-6.1-terra (effort high), gpt-6.1-sol (effort low and high) and gpt-6-astra (effort low).

Ambiguous text: something is lost

pilot and follow-up, 16 Nov 2026 · exploratory · 18 same-spelling classes · one call per item, three in the follow-up · items not yet native-checked · follow-up plan fixed in advance

In each of 18 classes of same-spelling words, two questions send the model the same text, byte for byte, and need different answers. No reader can expect more than 50%.

Figure D. Forced choice on the 36 word-alone items, against the 50% bound. The sentence items use other words. Brackets: 95% interval over the classes, with these 21 settings held fixed.
Figure D as a table (forced choice, % correct; the bound for any reader is 50%).
itemscorrect of calls95% interval over classes
word alone, one call per item366 of 756 (48.4%)[46.6, 50.1]
word alone, three calls per item1,108 of 2,268 (48.9%)[46.7, 50.6]
other words, in a sentence749 of 756 (99.1%)[98.1, 99.9]
in a sentence, the rarer word is the answer371 of 378 (98.1%)[96.3, 99.5]

Forced to pick an option, the 21 settings together score 48.9% [46.7, 50.6]: at the bound. Other same-spelling words, in sentences written to allow one reading, score 99.1% in the pilot, and 98.1% where the rarer word is the answer.

Where the text cannot settle the answer, the only better behaviour is to say so. In free text, 52.6% of calls do. When a closing line also asks for the option number, 21.6% do, and 60.6% of replies are a bare number.

Figure E. Calls that answer 0 (the text does not decide) or say so, by reply format and by the sentence placed before the question. 2,268 calls per bar.
Figure E as a table: calls that say the text does not settle the answer, or answer 0.
reply formatsentence before the itemcalls that say so or answer 095% interval over classessame, on items the text settles
free text, no closing lineno sentence1,194 of 2,268 (52.6%)[44.8, 59.6]1 of 1,260 (0.1%)
free text, no closing linecontrol sentence1,268 of 2,268 (55.9%)[47.8, 63.5]0 of 1,260 (0.0%)
free text, no closing linenote1,561 of 2,268 (68.8%)[60.7, 75.6]6 of 1,260 (0.5%)
free text, closing line asks for the option numberno sentence491 of 2,268 (21.6%)[18.2, 25.0]3 of 1,260 (0.2%)
free text, closing line asks for the option numbercontrol sentence648 of 2,268 (28.6%)[23.9, 33.2]0 of 1,260 (0.0%)
free text, closing line asks for the option numbernote758 of 2,268 (33.4%)[29.0, 37.9]3 of 1,260 (0.2%)
answer 0 offeredno sentence1,353 of 2,268 (59.7%)[53.1, 65.7]10 of 1,260 (0.8%)
answer 0 offeredcontrol sentence1,464 of 2,268 (64.6%)[57.8, 70.8]33 of 1,260 (2.6%)
answer 0 offerednote1,619 of 2,268 (71.4%)[64.9, 77.3]36 of 1,260 (2.9%)

A note saying that the text is pinyin without tones matters less than the closing line: 4.9 to 12.9 points over a control sentence that names neither, unevenly across these settings.

Free-text replies were labelled by model judges that are also models under test (claude-sonnet-5-5 and gpt-6.1-sol, with kimi-k3-1-pro-261024 on their splits); no person has checked a label.

Note: “Note: the Chinese text below was typed as Hanyu Pinyin without tone marks.” Control sentence: “Note: the Chinese text below was typed in lower-case letters with spaces between words.”

For input pipelines: three suggestions, not yet tested

Send the keyboard state (layout, input mode) with the text, where the client knows it. Naming the layout gave gpt-6.1-terra 28 to 30 points in two runs, and 6 other reasoning-off settings no clear gain.

Keep only same-spelling words in native script. In the earlier study this scored 1.00, but the kept words were the answers themselves.

Leave room to flag ambiguity. A reply that is a bare option number cannot flag it.

Explore models

Pick a model, a reasoning level, a language and a typed form. Every number shows its status, its items and calls, and its interval. Where there is no result the explorer says why, and nothing is filled in. Refused, cut-off and unparseable replies count as wrong and are shown together as no valid answer.

Reasoning level
Typed form

Method

Task

Each item is a reading-comprehension question from Belebele: a passage, a question and four options, in one of 122 language variants. The model answers with the number of one option. In a typed arm the passage, the question and the options all reach the model in the typed form (in the main run's question-only arm, only the question and options do).

Scoring

Answers are parsed strictly: only an answer that is exactly one of the four option labels counts. Every reply counts. A refused reply, a reply cut off at the output budget and a reply that does not parse count as wrong; this site shows them together as no valid answer and splits them in the detail view. Reasoning tokens are as each API reports them; cost per call is at list price.

Floor without reading

Floor without reading: expected accuracy of always picking the option with the largest share of its letter triples found in the passage, ties sharing credit, computed on the wrong-layout text: 34.8% on average, 29% to 41% by language. Chance: 25%.

It is the score of a rule that never reads the question, not a model's score without the passage; the main run will measure that separately (arm no passage).

Typed forms

Vermilion marks text as typed, blue a form in which the model was told the keyboard state, ink the text as meant.

Forms of the main run's tests and the pilots (10)
As written
The passage, question and options as they are in Belebele: the text as meant. as written
Wrong layout
The whole item typed on the language's standard keyboard layout while the computer is set to US English: every character is what the same key gives on a US keyboard. Every keystroke is kept, and for most text the key map can be undone: in the main run's build the round trip is exact for 196 of 200 Russian, 198 of 200 Hebrew, 178 of 200 Armenian and 186 of 200 Arabic items (draw ranks 50 to 249), and the contrasts use the exact items. Characters on keys that print nothing on the active layout are lost, and text that mixes two scripts can have more than one reading. The 15 Nov pilot used earlier letter tables (operator version 2): undoing them restores 280 of 300 Russian texts exactly. For English the typed form is a cipher, not another layout: every key shifted one position to the right on the US keyboard. Main run arm (a). every keystroke kept
Control sentence
Wrong-layout text after a sentence that says only that the text was typed on a keyboard and is shown as typed. It names no layout. Main run arm (b'). every keystroke kept
Another layout, not named
Wrong-layout text after a sentence that says a layout other than US English was intended, without naming it. Main run arm (b''). every keystroke kept
Layout named
Wrong-layout text after a sentence that names the intended layout. Main run arm (b); in the pilot, one wording and no control sentence. every keystroke kept
Phonetic layout
The language typed on its phonetic layout (letters on the Latin keys that sound alike) while the computer is on US English. Main run arm (c); Arabic has none. every keystroke kept
Random key map
The same text typed through a seeded random key map that carries the same information as the real layout: letters move among the letter keys (those on Latin-letter keys among those keys, those on punctuation keys among those); digits, punctuation and space stay put (on the English maps the digit keys move too). No sentence. Main run arm (d): one of ten seeds per item; the pilot used three seeds. every keystroke kept
Random map, rearrangement stated
Random key map with a note that states the language and that the keys were rearranged; no table. Main run arm (e); in the pilot, the first map only. every keystroke kept
Key table given
Random key map with the note and the map's full letter table. Main run arm (f); in the pilot, the first map only. every keystroke kept
Sham table
Random key map with the note and a table of the same form, number of entries and about the same length as the real table, which says nothing about the map (for each typed character, where its key sits on the US keyboard). Every sentence of it is true. Main run arm (f0), the comparison for C3. every keystroke kept
Secondary forms (controls and the mechanism family) (10)
As written, with the control sentence
The native text after the control sentence (which is true of native text as well). as written
No passage
Question and options only, without the passage: what a model answers without reading the passage. as written
Wrong layout, spaced
Wrong-layout text with one space between the characters (three for a typed space). every keystroke kept
As written, spaced
The native text with one space between the characters: the control for the spaced arm. as written
Wrong layout, generic hint
Wrong-layout text after a note that is true of any text: it may have been typed with another layout active, and if so should be read as meant. every keystroke kept
As written, generic hint
The native text after the same generic hint: what the hint costs on native text. as written
Wrong layout, three examples
Wrong-layout text after three pairs of typed and meant sentences of the same layout pair; no other note. every keystroke kept
Random map, three examples
Random-map text after three pairs of typed and meant sentences of the same map; no other note. every keystroke kept
As written, options reversed
The native item with its four options in reverse order. as written
Wrong layout, options reversed
The wrong-layout item with its four options in reverse order. every keystroke kept
Exploratory forms (breadth, lossy forms, other input methods) (10)
Another map's table
Random key map with the note and the table of another seed of the same layout. Main run arm (f x): exploratory, run on some settings only. every keystroke kept
Romanised or without diacritics
Lossy typed form: non-Latin scripts romanised (operator D1), Latin scripts with their diacritics removed (operator D2). Several words can share one typed form. Items whose typed form is identical to the text as written are left out of a cell. lossy: several words can share one typed form
Pinyin with tone numbers
Chinese typed as pinyin with tone numbers (reduced loss). lossy: several words can share one typed form
Pinyin without tones, spaced
Chinese typed as pinyin without tones, words separated by spaces. lossy: several words can share one typed form
Pinyin without tones, unspaced
Toneless pinyin with no spaces between words. lossy: several words can share one typed form
Pinyin initials
The first letter of each pinyin syllable. lossy: several words can share one typed form
Vietnamese Telex keystrokes
Vietnamese as raw Telex keystrokes, before the input method composes the letters. every keystroke kept
Vietnamese VNI keystrokes
Vietnamese as raw VNI keystrokes (tone and vowel marks as digits). every keystroke kept
Japanese romaji
Japanese as the romaji typed into an input method, before kana-kanji conversion. lossy: several words can share one typed form
English typed on another layout
English text typed while the computer is set to the Russian, Hebrew, Korean or Arabic layout. every keystroke kept

Four steps

Hypothesis: reading wrong-layout text takes up to four steps. Models fail at different steps (a description, not a tested hypothesis).

  1. Recognise that a key map applies, and which: tested by naming the layout against a control sentence.
  2. Recall the map, or be given it: the real layout against random key maps that carry the same information, and a random map with and without its table.
  3. Apply it: a map given in the prompt against the text as written.
  4. Decipher it, where no map is known: random maps with no table, with reasoning on.

What the main run tests

Confirmatory: C1 to C5, tested together with a Holm correction (alpha 0.05). Four cross-script layouts under an English instruction for C1 to C4 (Russian, Hebrew, Armenian, Arabic); Mandarin for C5. Everything else is descriptive. The unit that is resampled is the Belebele passage for C1 to C4, the same-spelling class for C5. The plan, the items and the analysis code are frozen and time-stamped before the first main-run call. A public registration follows later.

The five tests and the items
  • C1 Full disclosure of the keyboard state. Accuracy with the layout named (b) minus accuracy with the control sentence (b'). Counted as supported only if it passes the Holm correction and (b) minus no sentence (a) is positive. Its two parts are secondary: recognition (b'') minus (b'), identification (b) minus (b'').
  • C2 The standard layout against random permutations of the same keys. Accuracy on the real wrong layout (a) minus accuracy on class-preserving random key maps of the same keys (d). A real layout may differ from a random one in more than exposure; the phonetic arm (c) is reported beside it.
  • C3 The information in the table. Random key map with its key table (f) minus the same map with a sham table of the same form and length (f0).
  • C4 The cost moves to tokens. Pairs of one model with reasoning off and on; the list of pairs is fixed at the freeze, after a protocol check on pilot items. In every pair the accuracy loss on wrong-layout text is smaller with reasoning on, and with reasoning on the model spends more reasoning tokens than on the text as written.
  • C5 Prompted abstention under the explicit format. Mandarin items whose typed text does not settle the answer, with a 'cannot tell' answer offered: share of replies that choose it after the disclosure sentence minus after the control sentence.

C1 to C4: Belebele items of draw ranks 50 to 149, the same for every setting (items whose passage also occurs among ranks 0 to 49 are left out); ranks 150 to 249 where the budget allows, as a secondary extended-set estimate. C5: the Mandarin items built at the bound. The pilots used ranks 0 to 49.

Exploratory, never part of a test
  • The roster: every registered setting that passed the 5-call smoke check, on fewer items and arms, for this explorer.
  • The map: all 122 Belebele variants as written and romanised or without diacritics, on five settings (gpt-6-luna effort none and high, deepseek-v4-1-flash thinking off, doubao-seed-2-1-lite thinking off, claude-haiku-5-5 thinking off).
  • Breadth: every language with a wrong-layout condition (29), and English typed on four other layouts.
  • The arm with another map's table (f x), on some settings only.
  • Japanese romaji and Vietnamese VNI keystrokes beside the Chinese and Vietnamese arms.
  • A secondary family on mechanism (spaced text, a generic hint, three examples, reversed options), never part of the five tests.

From the run plan of 19 Nov 2026.

Status of a number

pilot
Pilot, exploratory. Each number shows the items and calls behind it.
interim
Main run, interim: taken while the main run was still going; numbers can change.
final
Main run, final: the frozen analysis on complete data.

Limits

Exploratory: 50 items per layout, one call per item; Chinese items not yet checked by an independent native speaker; model judges, no human labels; 21 settings chosen for convenience; one wording per sentence and reply format. Abstract: 48 valid calls, Chinese only.

Next

The planned main run: conditions that separate the four steps, items built at the bound in several languages, and notes tested against control sentences.

About

This site is maintained by Xiaoxiao Zhouyi, School of Engineering Mathematics and Technology, University of Bristol.

It accompanies the poster Skipping the IME? Keyboard-Friendly Multilingual Prompting for LLM Agents by Xiaoxiao Zhouyi and Yanzhen Li (UKAIRS 2026 · Emerging Research · Poster UKAIRS26-89). The accepted abstract is the earlier study below; the November pilots were run after it and are exploratory.

Site and updates: zhouyixiaoxiao.com

Earlier study: coding agents

The accepted abstract · 48 valid calls · Codex CLI and Claude Code · May–June 2026

Codex CLI kept its score on two routine prompts typed as pinyin without tones. On three tasks whose answer is an exact term, both agents fell to 0.333, and scored 1.00 with those terms kept in characters (one call per task).

Credits and licences

  • Questions: Belebele (Bandarkar et al., 2024), a multilingual reading-comprehension dataset, licensed CC BY-SA 4.0: github.com/facebookresearch/belebele. The converter's example sentences are first sentences of Belebele passages.
  • Key maps: exported from the project's operator D3x, whose tables are generated from xkeyboard-config (licence).
  • Typeface for typed text: IBM Plex Mono, SIL Open Font License 1.1 (licence), served from this site.
  • Language names: Unicode CLDR, as shipped with macOS.
  • Data: bundle.json, its schema and the trace of every value to its source file and key.

Use of AI tools: LLM coding and writing assistants were used to build the evaluation framework, to run and check the analyses, and to prepare the poster, including its text and layout. Free-text replies in the November pilots were labelled by model judges. Numbers are generated by scripts from the run records. The authors designed the study and take responsibility for all content.

This site was built with an LLM coding assistant from the project's summary files; its numbers are written by scripts, not by hand.