Follow-up to Three Rs in Strawberry, which covered a different LLM failure: the mismatch between token-level input and character-level operations such as counting letters. This one is about something structurally different.


The Video

Someone asked their car’s built-in AI assistant: “Should I drive to the car wash today?” It was raining. The assistant said yes, enthusiastically, with reasons: regular washing extends the life of the paintwork, removes road salt, and so on. Technically correct statements, all of them. Completely beside the point.

The clip spread. The reactions were the usual split: one camp said this proves AI is useless, the other said it proves people expect too much from AI. Both camps are arguing about the wrong thing.

This article does not archive the clip, and it does not identify the assistant, its model version, its system prompt, or whether it could have called a weather service. What follows relies only on what the answer shows: the weather did not enter it.

The interesting question is what kind of failure this is, and whether it is the same kind as the strawberry problem.

It is not. The failures look similar from the outside — confident wrong answer, context apparently ignored — but they differ in where the needed information is, and the difference matters if you want to understand what these systems can and cannot do.


Strawberry: The Letters Are in the Input

In the strawberry case the information is in the input, in units that do not match the operation asked for. The question asks for a count of characters (the r’s), and the model receives token IDs. On its own, “strawberry” is str|aw|berry in cl100k_base and st|raw|berry in o200k_base; after a space, as in “How many r’s are in strawberry?”, it is a single token in both [[1]]. Either way the model works on one vector per token, not on an indexable array of letters.

That is a mismatch between representation and operation. Byte-pair encoding [[2]] is reversible: the letters have not literally been destroyed. But the model is asked to recover and count character-level structure through a representation organised around subword units, a task for which next-token prediction is an awkward mechanism. How often that produces a wrong answer is an empirical question about a specified model.

I used to say here that the model failed because of that gap, citing the bare-word split. The strawberry post withdrew that account in July 2026, and in the typed question neither encoding splits the word at all.

The car wash case is different. The model received a well-formed question that asks for nothing below the level of its words. “Should I drive to the car wash today?” encodes as nine tokens in both cl100k_base and o200k_base, one per word and one for the question mark [[1]]; the assistant’s own tokeniser is unknown, but nothing in the request depends on how its words are cut. The wording poses no unusual linguistic ambiguity. The failure is that the answer depends on information that was not in the question and, as far as the answer shows, never reached the model.


The Missing Context

What would you need to answer “should I drive to the car wash today?” correctly?

  • The current weather (is it raining now?)
  • The weather forecast for the rest of the day (will it rain later?)
  • The current state of the car (how dirty is it?)
  • Possibly: how recently was it last washed, what kind of dirt (road salt after winter, tree pollen in spring), whether there is a time constraint

None of this is in the question. A human asking the question has access to some of it through direct perception (look out the window) and some through memory (I just drove through mud). A language model given only the question has access to none of it.

Let $X$ denote the question and $C$ denote this context — the current state of the world that the question is implicitly about. The correct answer $A$ is a function of both:

$$A = f(X, C) \tag{1}$$

The model has $X$. It does not have $C$. One useful idealisation is that it produces something like an expectation over the contexts that are possible given the question, marginalising out the unknown $C$:

$$\hat{A} = \mathbb{E}_{C \mid X}\!\left[\, f(X, C) \,\right]$$

This is a schematic decision model, not a claim that the assistant literally computes this expectation or that its training distribution supplies a known probability over weather. It captures the structural problem: a generic answer based on $X$ need not match the answer for the actual, unobserved $C$. For a yes-or-no answer, code “go” as 1 and “stay” as 0: $\hat{A}$ is then the probability that “go” is right, and an unqualified “yes” reports it as probability one.

One quantity that measures how strongly the right answer depends on the missing context is the mutual information between the answer and the context, given the question:

$$I(A;\, C \mid X) \;=\; H(A \mid X) - H(A \mid X, C) \tag{2}$$

Here $H(A \mid X)$ is the residual uncertainty in the answer given only the question, and $H(A \mid X, C)$ is the residual uncertainty once the context is also known, both in bits. Eq. (1) makes the answer a function of question and context, so the second term is zero: the mutual information is the uncertainty about the right answer that is left after reading the question. For a stable factual question such as “what is the capital of France?”, the situational context contributes nothing in this idealisation: $f$ does not depend on $C$, and Eq. (2) is zero. For the car-wash decision it can reach one bit, the most a yes-or-no answer can carry.

The Weather Is Worth at Most One Bit

Suppose the only state that matters is binary, rain today or not, and that given everything the question reveals, rain has probability $p$. Suppose further that the right answer flips with the weather: stay if it rains, go if it does not. For this question, answer and weather then determine each other, and Eq. (2) becomes

$$ \begin{aligned} I(A;\, C \mid X) &= H(C \mid X) = H_b(p) \\ &= -p \log_2 p - (1-p) \log_2 (1-p), \end{aligned} \tag{3} $$

where $H_b$ is the binary entropy, in bits. If the right answer were the same in both states, the mutual information would be zero instead. Those are the two extremes: mutual information is never negative, and $I(A;\, C \mid X) = H(C \mid X) - H(C \mid A, X) \le H(C \mid X)$, because the answer cannot carry more information about the weather than the weather has. A binary state with prior $p$ is therefore worth at most $H_b(p)$ bits to the answer, the full bit only at $p = 1/2$ (Fig. 1). Even at $p = 0.1$ or $p = 0.9$ it is still worth 0.47 bits.

Figure 1. A binary weather state is worth at most one bit to the answer, and the full bit only when rain and no rain are equally likely. The curve is Eq. (3) for a question whose right answer flips with the weather, against the prior probability of rain $p$.

Mutual information also leaves out what a wrong answer costs: a wasted wash and a needlessly postponed one are weighed by the utilities of a decision, which Eq. (2) does not contain.


Confident Anyway

This is the part that produces the most indignation in the viral clips: not just that the model was wrong, but that it was confident about being wrong. It did not say “I don’t know what the current weather is.” It said, in effect, “yes, here are five reasons you should go.”

Two distinctions help describe the response, although the clip alone cannot establish the system’s training data or internal process.

A generic-answer pattern. Questions of the form “should I do X?” often invite answers of the form “here are some considerations.” The response fits that pattern instead of first checking whether the decision depends on missing local state. That observation does not tell us which training examples or instructions produced the behaviour.

No explicit uncertainty signal. The response did not say “I cannot answer this because I lack context C.” A better response must recognise that the answer is sensitive to unsupplied information, then ask for it, retrieve it, or state the limitation. The clip shows that this did not happen; without the system prompt, model version, or trace, it does not show why.

This resembles a calibration problem: for a probabilistic classifier, an 80%-confidence group is calibrated when roughly 80% of its predictions are correct. Guo et al. [[3]] study that formal property for classifiers. Fluent, unqualified language from an assistant is not itself a numerical confidence estimate, so the video does not measure calibration in that technical sense. Words can carry a calibrated confidence when they state one: GPT-3, fine-tuned to add a confidence level to its answers to arithmetic questions, stated well-calibrated levels, and moderately calibrated ones under distribution shift [[4]]. The assistant in the clip stated none. What the clip demonstrates is an uncertainty-communication failure alongside the missing-context failure.


What Grounding Means

The classic statement of grounding is Harnad’s. He named the symbol grounding problem in 1987 and set it out in a 1990 paper [[5]]: the problem of connecting symbol systems to the things they refer to — how does the word “apple” connect to actual apples, rather than just to other symbols? A symbol system that only connects symbols to other symbols (dictionary definitions, synonym relations) has the form of meaning without the substance. Harnad’s image is learning Chinese from a Chinese/Chinese dictionary alone.

Applied to a language model trained on text: the model has rich internal representations of concepts like “rain,” “car wash,” “dirty car,” and their relationships. But those representations are grounded in text about those things, not in the things themselves. The model knows what rain is. It does not know whether it is raining right now, because “right now” is not a location in the training data.

That makes two gaps: Harnad’s, between words and the things they name, and a narrower one, between a model and the present state of the world. The car wash turns on the second, which is what grounding means in the rest of this post. A person whose word “rain” is grounded in Harnad’s sense would not know either, sitting in a windowless room.

Making a model bigger or training it on more historical text cannot, by itself, reveal whether it is raining at the user’s location now. A deployed system may of course receive sensors, retrieved data, or tool results at inference time. The structural limitation belongs to the information supplied to this particular interaction, not to every possible language-model system.


What Tool Use Gets You (and What It Doesn’t)

The standard engineering response to grounding problems is tool use: give the model access to a weather API, a calendar, a search engine. Now when asked “should I drive to the car wash today?” the model can query the weather service, get the current conditions, and factor that into the answer.

This is genuinely useful: with a reading in hand, the weather part of the answer can come from data rather than from a generic pattern. But tool use solves the problem only if two conditions hold:

  1. The model knows it needs the tool. It must recognise that this question has $I(A; C \mid X) > 0$ for context $C$ that a weather tool can provide, and that it is missing that context. This requires the meta-level awareness described above, and training a model to use tools does not guarantee that it calls them when they are needed. Toolformer, a 6.7-billion-parameter model trained to decide for itself when to call APIs, among them a calendar that returns today’s date, called the calendar on 54.8% of the questions in a test set that cannot be answered without the current date [[6]]. That is a 2023 model; I have no measurement for the clip’s assistant.

  2. The right tool exists and returns clean data. Weather APIs exist. “How dirty is my car?” does not have an API. “Am I the kind of person who cares about car cleanliness enough that this matters?” has no API. Some missing context can be retrieved; some is inherently private to the person asking.

Fig. 2 sorts the missing context along that line.

Figure 2. Part of the missing context can be fetched; the rest mostly has to come from the person asking, and either route opens only if the assistant notices the gap.

The deeper issue is not tool availability but knowing what you don’t know. A model that does not recognise its own information gaps cannot reliably decide when to use a tool, ask a clarifying question, or express uncertainty. This is a hard problem — arguably harder than making the model more capable at the tasks it already handles. For questions with checkable answers it is partly learnable. In a 2022 Anthropic preprint, models trained to predict whether they know the answers to trivia questions could also pick out arithmetic and code questions they could answer, but, in the authors’ words, they “struggle with calibration” on new tasks [[7]].


The Contrast, Stated Plainly

The strawberry failure and the car wash failure look alike from the outside — confident wrong answer — but they are different enough that conflating them produces confused diagnosis and confused solutions.

Strawberry: the input contains the information, but a subword representation and a learned procedure do not guarantee exact character counting. For a count that must be exact, the dependable fix is a deterministic string operation, which the model can write or call, rather than generated letter-by-letter reasoning.

Car wash: unless something supplies it, the model does not have the information (current weather, car state). No fix to the model’s architecture or prompt engineering gives it information it was never given. The information has to come from outside: provide the context explicitly, or give the model a tool that can retrieve it, or design the system so that context-dependent questions are routed to systems that have access to the relevant state.

A model that confidently answers the car wash question without access to current conditions is not, on the evidence of its answer, failing at language understanding: its reasons were about car washing. Without weather access, it answered the question it was given with the information it was given, and did not say that the information was not enough. Knowing which kind of failure you are looking at is most of the work in figuring out what to do about it.


The grounding problem connects to the broader question of what it means for a language model to “know” something — which comes up in a different form in the context window post, where the issue is not missing context but irrelevant context drowning out the relevant signal.

A few weeks later, variants of a different car-wash prompt showed a third kind of failure: Car Wash, Part Three: The AI Said Walk. There the prompt itself said that the car needed washing and that the car wash was close, and some answers still recommended walking. A fourth post, Pelican on a Bicycle is the New Carwash, follows that prompt into the benchmarks that absorbed it.


References

Literature, calculations and token counts checked through 2026-10-25.

[1] OpenAI. (2026). tiktoken (Version 0.14.0, released 17 August 2026) [Computer software]. Encodings cl100k_base and o200k_base; splits measured on 25 October 2026. https://github.com/openai/tiktoken

[2] Sennrich, R., Haddow, B., & Birch, A. (2016). Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1715–1725. DOI: 10.18653/v1/P16-1162

[3] Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, PMLR 70, pp. 1321–1330. https://proceedings.mlr.press/v70/guo17a.html

[4] Lin, S., Hilton, J., & Evans, O. (2022). Teaching models to express their uncertainty in words. Transactions on Machine Learning Research. OpenReview: 8s8K2UZGTZ; arXiv:2205.14334

[5] Harnad, S. (1990). The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(1–3), 335–346. DOI: 10.1016/0167-2789(90)90087-6. Author’s copy: arXiv:cs/9906002

[6] Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Hambro, E., Zettlemoyer, L., Cancedda, N., & Scialom, T. (2023). Toolformer: Language models can teach themselves to use tools. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023). Paper; arXiv:2302.04761

[7] Kadavath, S., Conerly, T., Askell, A., et al. (2022). Language models (mostly) know what they know. Preprint, Anthropic. arXiv:2207.05221


Changelog

  • 2026-07-11: Corrected the Harnad DOI; clarified that BPE does not literally discard characters, that the information-theory equations are an idealisation rather than a measured model of the assistant, and that fluent certainty is not a numerical calibration estimate.

  • 2026-10-03: Corrected the BPE split of “strawberry” from “straw” + “berry” to “str” + “aw” + “berry” (cl100k_base, verified with tiktoken), consistent with the strawberry article’s own 2025-12-01 correction.

  • 2026-10-25: Withdrew the claim that the strawberry model failed because of its token split, as the strawberry post did on 2026-07-11, and measured the splits (tiktoken 0.14.0): after a space “strawberry” is one token, and the car-wash question is nine. Rewrote the closing, which credited the answer to training that the body says the clip cannot show, and qualified the missing-information claims, since the assistant’s weather access is unknown. Renamed two headings that asserted causes. Said what grounding means here (access to the current state of the world, not Harnad’s sense); Harnad named the problem in 1987, not 1990. Added the binary bound $I(A;C\mid X) \le H_b(p) \le 1$ bit (Fig. 1); “the quantity” became “one quantity”, and “contributes little” became zero for a fixed fact. Replaced unsourced tool claims with Toolformer; added Lin et al., Kadavath et al., Fig. 2 and the pelican link. Summary: “go” → “drive”. Citations are now numbered.