The Setup

When OpenAI released o1-preview in September 2024, “how many r’s are in strawberry?” was already a familiar demonstration of language-model failure. The coincidence was irresistible. It was also reported too neatly: OpenAI’s own release page shows o1-preview decoding a cipher whose answer is “THERE ARE THREE R’S IN STRAWBERRY.” That is evidence of one successful, scaffolded task, not a benchmark of character counting. It does rule out the stronger joke that the newly released model simply could not count the letters in its supposed codename.

Before the opinions: the answer is three. s-t-r-a-w-b-e-r-r-y. One in the str- cluster, two in the -rry ending. Some model versions and prompt formulations have answered this incorrectly. Without a recorded model snapshot, interface, prompt, settings, date, and repeated trials, “most models return two” is not a result I can defend.

The question worth asking is not “why is the model stupid.” It is not stupid, and “stupid” is not a useful category here. The question is: what does this specific error reveal about the structure of the system?

The answer involves tokenisation, and it is actually interesting.

How You Count Letters (and How the Model Doesn’t)

When you count the r’s in “strawberry,” you do something like this: scan the string left to right, maintain a running count, increment it each time you see the target character. This is a sequential operation over a character array. It requires no semantic knowledge about the word — it does not matter whether “strawberry” is a fruit, a colour, or a nonsense string. The characters are the input; the count is the output.

A GPT-style language model does not ordinarily receive a character array. It receives token IDs produced by a model-specific tokeniser. Byte Pair Encoding (BPE) and related subword methods often group several characters or bytes into one token. The split is not universal: OpenAI’s documentation explicitly lists different encodings for different model families and does not identify the o1-preview encoding. A cl100k_base demonstration therefore cannot establish the token sequence seen by o1-preview.

For a known encoding and tokeniser version, the split can be measured rather than guessed. The resulting integer IDs are looked up in an embedding table. The model is then operating on one vector per token, not on a conveniently indexed array with one element per letter.

That mismatch matters. It is not yet a causal account of one answer.

What BPE Does (and Doesn’t) Preserve

BPE is a greedy compression algorithm. Starting from individual bytes, it iteratively merges the most frequent pair of adjacent symbols into a single new token:

$$\text{merge}(a, b) \;:\; \underbrace{a \;\; b}_{\text{separate}} \;\longrightarrow\; \underbrace{ab}_{\text{single token}}$$

Applied to a large text corpus until a fixed vocabulary size is reached, this produces a vocabulary of common subwords. Frequent words and common word-parts become single tokens; rare sequences stay as multi-token fragments.

What BPE optimises for is a compact, useful vocabulary, not character-level transparency. A multi-character token still preserves its bytes in the tokeniser: OpenAI’s tiktoken documentation demonstrates both decoding a token sequence back to text and recovering the bytes represented by each token. Tokenisation is therefore not lossy in the simple information- theoretic sense.

The harder distinction is architectural. After embedding lookup, the transformer does not receive the tokeniser’s byte table as an indexed scratch pad. It may learn spelling regularities from training, and tools or generated intermediate text can make character operations easier, but exact access is not guaranteed merely because the original string is reversibly tokenised.

What an Error Would Establish

An incorrect answer would establish that this particular run failed an exact string operation. Repeated, versioned tests across controlled prompt variants could establish a pattern. Tokenisation supplies a plausible pressure: the operation requested by the prompt and the units presented to the transformer do not line up.

It does not follow that a model answering “two” must have counted the two Rs in berry and missed the one in str. That story depends on a token split that may not apply, and it attributes a hidden computation from the final answer. Different models may spell, retrieve a memorised answer, call a tool, or use other intermediate strategies. The output alone does not distinguish them.

Scaffolding Can Help

Prompting a model to spell the word before counting can expose intermediate mistakes and may improve a given model’s answer. It does not necessarily put ten one-character tokens into the context: generated text is tokenised by the same model-specific encoding, and adjacent characters can still share tokens. No cross-model error-rate estimate is available here.

For a result that must be exact, the robust intervention is simpler: use a deterministic string operation. A model can write or call that operation and report the result. Visible spelling remains useful as a check, not a proof of the model’s internal representation.

The Right Frame

The “how many r’s” test is sometimes cited as evidence that language models don’t “really” understand text, or that they are sophisticated autocomplete engines with no genuine knowledge. These framing choices produce more heat than light.

The more precise statement is this: next-token prediction does not directly train an indexable character array or a guaranteed exact-count procedure. Character counting can therefore expose a mismatch between representation and operation. How often it does so is an empirical question about a specified model and evaluation, not a timeless property of all language models.

“Strawberry” sits squarely in that class. A wrong answer is neither proof of general stupidity nor evidence that tokenisation deleted a letter. It is a small test of whether the system can bridge token-level processing and an exact character-level request.

Literature and product documentation checked through 2026-07-11. The article does not claim an error rate for current models or an undocumented tokenizer mapping for o1-preview.

A follow-up post covers a structurally different failure mode: Should I Drive to the Car Wash? — where an incompletely documented prompt family illustrates a different boundary: missing situation-specific world state.

References

Changelog

  • 2025-12-01: Corrected the tokenisation of “strawberry” from two tokens (straw|berry) to three tokens (str|aw|berry), matching the actual cl100k_base tokeniser used by GPT-4. The directional argument (token boundaries obscure character-level information) is unchanged; the specific analysis was updated accordingly.

  • 2026-07-11: Removed the unsupported o1 tokenizer attribution and cross-model failure-rate claims; corrected the claim that tokenisation loses character information; and separated a plausible representational pressure from an unobserved causal account of any model answer.