Follow-up to Three Rs in Strawberry, which covered a different LLM failure: tokenisation and why models cannot count letters. This one is about something structurally different.

The Video

Someone asked their car’s built-in AI assistant: “Should I drive to the car wash today?” It was raining. The assistant said yes, enthusiastically, with reasons: regular washing extends the life of the paintwork, removes road salt, and so on. Technically correct statements, all of them. Completely beside the point.

The clip spread. The reactions were the usual split: one camp said this proves AI is useless, the other said it proves people expect too much from AI. Both camps are arguing about the wrong thing.

The interesting question is: why did the model fail here, and is this the same kind of failure as the strawberry problem?

It is not. The failures look similar from the outside — confident wrong answer, context apparently ignored — but the underlying causes are different, and the difference matters if you want to understand what these systems can and cannot do.

The Strawberry Problem Was About Representation

In the strawberry case, the model failed because of the gap between its input representation (BPE tokens: “straw” + “berry”) and the task (count the character “r”). The character information was not accessible in the model’s representational units. The model understood the task correctly — “count the r’s” is unambiguous — but the input structure did not support executing it.

That is a representation-and-computation failure. Byte-pair encoding is reversible: the letters have not literally been destroyed. But the model is asked to recover and count character-level structure through a representation organised around subword units, a task for which next-token prediction is an awkward mechanism.

The car wash case is different. The model received a perfectly well-formed question and had no representation problem at all. “Should I drive to the car wash today?” is tokenised without any information loss. The wording poses no unusual linguistic ambiguity. The failure is that the answer depends on information that was never in the input in the first place.

The Missing Context

What would you need to answer “should I drive to the car wash today?” correctly?

  • The current weather (is it raining now?)
  • The weather forecast for the rest of the day (will it rain later?)
  • The current state of the car (how dirty is it?)
  • Possibly: how recently was it last washed, what kind of dirt (road salt after winter, tree pollen in spring), whether there is a time constraint

None of this is in the question. A human asking the question has access to some of it through direct perception (look out the window) and some through memory (I just drove through mud). A language model has access to none of it.

Let $X$ denote the question and $C$ denote this context — the current state of the world that the question is implicitly about. The correct answer $A$ is a function of both:

$$A = f(X, C)$$

The model has $X$. It does not have $C$. One useful idealisation is that it produces something like an expectation over possible contexts, marginalising out the unknown $C$:

$$\hat{A} = \mathbb{E}_C\!\left[\, f(X, C) \,\right]$$

This is a schematic decision model, not a claim that the assistant literally computes this expectation or that its training distribution supplies a known probability over weather. It captures the structural problem: a generic answer based on $X$ need not match the answer for the actual, unobserved $C$.

The quantity that measures how much the missing context matters is the mutual information between the answer and the context, given the question:

$$I(A;\, C \mid X) \;=\; H(A \mid X) - H(A \mid X, C)$$

Here $H(A \mid X)$ is the residual uncertainty in the answer given only the question, and $H(A \mid X, C)$ is the residual uncertainty once the context is also known. For a stable factual question such as “what is the capital of France?”, the relevant situational context contributes little once the question is fixed. For the car-wash decision it contributes much more. The equation names that difference; this article does not estimate the mutual information empirically.

Why the Model Was Confident Anyway

This is the part that produces the most indignation in the viral clips: not just that the model was wrong, but that it was confident about being wrong. It did not say “I don’t know what the current weather is.” It said “yes, here are five reasons you should go.”

Two distinctions help describe the response, although the clip alone cannot establish the system’s training data or internal process.

A generic-answer pattern. Questions of the form “should I do X?” often invite answers of the form “here are some considerations.” The response fits that pattern instead of first checking whether the decision depends on missing local state. That observation does not tell us which training examples or instructions produced the behaviour.

No explicit uncertainty signal. The response did not say “I cannot answer this because I lack context C.” A better response must recognise that the answer is sensitive to unsupplied information, then ask for it, retrieve it, or state the limitation. The clip shows that this did not happen; without the system prompt, model version, or trace, it does not show why.

This resembles a calibration problem: for a probabilistic classifier, an 80%-confidence group is calibrated when roughly 80% of its predictions are correct. Guo et al. (2017) study that formal property for classifiers. Fluent, unqualified language from an assistant is not itself a numerical confidence estimate, so the video does not measure calibration in that technical sense. What it demonstrates is an uncertainty-communication failure alongside the missing-context failure.

What Grounding Means

The term grounding in AI has a precise origin. Harnad (1990) used it to describe the problem of connecting symbol systems to the things they refer to — how does the word “apple” connect to actual apples, rather than just to other symbols? A symbol system that only connects symbols to other symbols (dictionary definitions, synonym relations) has the form of meaning without the substance.

Applied to language models: the model has rich internal representations of concepts like “rain,” “car wash,” “dirty car,” and their relationships. But those representations are grounded in text about those things, not in the things themselves. The model knows what rain is. It does not know whether it is raining right now, because “right now” is not a location in the training data.

Making a model bigger or training it on more historical text cannot, by itself, reveal whether it is raining at the user’s location now. A deployed system may of course receive sensors, retrieved data, or tool results at inference time. The structural limitation belongs to the information supplied to this particular interaction, not to every possible language-model system.

What Tool Use Gets You (and What It Doesn’t)

The standard engineering response to grounding problems is tool use: give the model access to a weather API, a calendar, a search engine. Now when asked “should I go to the car wash today?” the model can query the weather service, get the current conditions, and factor that into the answer.

This is genuinely useful. The model with a weather tool call will answer this question correctly in most circumstances. But tool use solves the problem only if two conditions hold:

  1. The model knows it needs the tool. It must recognise that this question has $I(A; C \mid X) > 0$ for context $C$ that a weather tool can provide, and that it is missing that context. This requires the meta-level awareness described above. Models trained on tool use learn to invoke tools for recognised categories of question; for novel question types, or questions that superficially resemble answerable ones, the tool call may not be triggered.

  2. The right tool exists and returns clean data. Weather APIs exist. “How dirty is my car?” does not have an API. “Am I the kind of person who cares about car cleanliness enough that this matters?” has no API. Some missing context can be retrieved; some is inherently private to the person asking.

The deeper issue is not tool availability but knowing what you don’t know. A model that does not recognise its own information gaps cannot reliably decide when to use a tool, ask a clarifying question, or express uncertainty. This is a hard problem — arguably harder than making the model more capable at the tasks it already handles.

The Contrast, Stated Plainly

The strawberry failure and the car wash failure look alike from the outside — confident wrong answer — but they are different enough that conflating them produces confused diagnosis and confused solutions.

Strawberry: the input contains the information, but the model’s subword representation and learned procedure do not make exact character counting reliable. A dependable fix is procedural or architectural: expose character-level units or use a deterministic string operation, rather than trusting generated letter-by-letter reasoning.

Car wash: the model does not have the information (current weather, car state). No fix to the model’s architecture or prompt engineering gives it information it was never given. The fix is exogenous: provide the context explicitly, or give the model a tool that can retrieve it, or design the system so that context-dependent questions are routed to systems that have access to the relevant state.

A model that confidently answers the car wash question without access to current conditions is not failing at language understanding. It is behaving exactly as its training shaped it to behave, given its lack of situational grounding. Knowing which kind of failure you are looking at is most of the work in figuring out what to do about it.

The grounding problem connects to the broader question of what it means for a language model to “know” something — which comes up in a different form in the context window post, where the issue is not missing context but irrelevant context drowning out the relevant signal.

A second car wash video a few weeks later produced a third, different failure: Car Wash, Part Three: The AI Said Walk — the model had the right world state but chose the wrong interpretation of the question.

References

Literature checked through 2026-07-11.

Changelog

  • 2026-07-11: Corrected the Harnad DOI; clarified that BPE does not literally discard characters, that the information-theory equations are an idealisation rather than a measured model of the assistant, and that fluent certainty is not a numerical calibration estimate.