The Video

Same question as last month’s: “Should I drive to the car wash?” New video, new AI, new wrong answer. This time the assistant replied that walking was the better option — better for health, better for the environment, and the car wash was only fifteen minutes away on foot.

Accurate, probably. Correct, arguably. Useful? No.

The model did not fail because of tokenisation. It did not fail because it lacked access to the current weather. It failed because it read the wrong question. The user was asking “is now a good time to have my car washed?” The model answered “what is the most sustainable way for a human to travel to the location of a car wash?”

These are different questions. The model chose the second one. This is a pragmatic inference failure, and it is the most instructive of the three failure modes in this series — because the model was not, by any obvious measure, working incorrectly. It was working exactly as designed, on the wrong problem.

What the Question Actually Meant

“Should I drive to the car wash?” is not about how to travel. The word “drive” here is not a transportation verb; it is part of the idiomatic compound “drive to the car wash,” which means “take my car to get washed.” The presupposition of the question is that the speaker owns a car, the car needs or might benefit from washing, and the speaker is deciding whether the current moment is a good one to go. Nobody asking this question wants to know whether cycling is a viable alternative.

Linguists distinguish between what a sentence says — its literal semantic content — and what it implicates — the meaning a speaker intends and a listener is expected to infer. Paul Grice formalised this in 1975 with a set of conversational maxims describing how speakers cooperate to communicate:

• Quantity: say as much as is needed, no more
• Quality: say only what you believe to be true
• Relation: be relevant
• Manner: be clear and orderly

The maxims are not rules; they are defaults. When a speaker says “should I drive to the car wash?”, a cooperative listener applies the maxim of Relation to infer that the question is about car maintenance and current conditions, not about personal transport choices. The “drive” is incidental to the real question, the way “I ran to the store” does not invite commentary on jogging technique.

The model violated Relation — in the pragmatic sense. Its answer was technically relevant to one reading of the sentence, and irrelevant to the only reading a cooperative human would produce.

A Taxonomy of the Three Failures

It is worth being precise now that we have three examples:

Strawberry (tokenisation failure): The information needed to answer was present in the input string but lost in the model’s representation. “Strawberry” →

— the character “r” in “straw” is not directly accessible. The model understood the task correctly; the representation could not support it.

Car wash, rainy day (grounding failure): The model understood the question. The information needed to answer correctly — current weather — was never in the input. The model answered by averaging over all plausible contexts, producing a sensible-on-average response that was wrong for this specific context.

Car wash, walk (pragmatic inference failure): The model had all the relevant words. It may have had access to the weather, the location, the car state. It chose the wrong interpretation of what was being asked. The sentence was read at the level of semantic content rather than communicative intent.

Formally: let $\mathcal{I}$ be the set of plausible interpretations of an utterance $u$. The intended interpretation $i^*$ is the one a cooperative, contextually informed listener would assign. A well-functioning pragmatic reasoner computes:

The model appears to have assigned high probability to the transportation-choice interpretation $i_{\text{walk}}$, apparently on the surface pattern: “should I

to

?” generates responses about modes of transport. It is a natural pattern-match. It is the wrong one.

Why This Failure Mode Is More Elusive

The tokenisation failure has a clean diagnosis: look at the BPE splits, find where the character information was lost. The grounding failure has a clean diagnosis: identify the context variable $C$ the answer depends on, check whether the model has access to it.

The pragmatic failure is harder to pin down because the model’s answer was not, in isolation, wrong. Walking is healthy. Walking to a car wash that is fifteen minutes away is plausible. If you strip the question of its conversational context — a person standing next to their dirty car, wondering whether to bother — the model’s response is coherent.

The error lives in the gap between what the sentence says and what the speaker meant, and that gap is only visible if you know what the speaker meant. In a training corpus, this kind of error is largely invisible: there is no ground truth annotation that marks a technically-responsive answer as pragmatically wrong.

This is a version of a known problem in computational linguistics: models trained on text predict text, and text does not contain speaker intent. A model can learn that “should I drive to X?” correlates with responses about travel options, because that correlation is present in the data. What it cannot easily learn from text alone is the meta-level principle: this question is about the destination’s purpose, not the journey.

The Gricean Model Did Not Solve This

It is tempting to think that if you could build in Grice’s maxims explicitly — as constraints on response generation — you would prevent this class of failure. Generate only responses that are relevant to the speaker’s probable intent, not just to the sentence’s semantic content.

This does not obviously work, for a simple reason: the maxims require a model of the speaker’s intent, which is exactly what is missing. You need to know what the speaker intends to know which response is relevant; you need to know which response is relevant to determine the speaker’s intent. The inference has to bootstrap from somewhere.

Human pragmatic inference works because we come to a conversation with an enormous amount of background knowledge about what people typically want when they ask particular kinds of questions, combined with contextual cues (tone, setting, previous exchanges) that narrow the interpretation space. A person asking “should I drive to the car wash?” while standing next to a mud-spattered car in a conversation about weekend plans is not asking for a health lecture. The context is sufficient to fix the interpretation.

Language models receive text. The contextual cues that would fix the interpretation for a human — the mud on the car, the tone of the question, the setting — are not available unless someone has typed them out. The model is not in the conversation; it is receiving a transcript of it, from which the speaker’s intent has to be inferred indirectly.

Where This Leaves the Series

Three videos, three failure modes, three diagnoses. None of them are about the model being unintelligent in any useful sense of the word. Each of them is a precise consequence of how these systems work:

1. Models process tokens, not characters. Character-level structure can be lost at the representation layer.
2. Models are trained on static corpora and have no real-time connection to the world. Context-dependent questions are answered by marginalising over all plausible contexts, which is wrong when the actual context matters.
3. Models learn correlations between sentence surface forms and response types. The correlation between “should I \[travel verb\]to \[place\]?” and transport-related responses is real in the training data. It is the wrong correlation for this question.

The useful frame, in all three cases, is not “the model failed” but “what, precisely, does the model lack that would be required to succeed?” The answers point in different directions: better tokenisation; real-time world access and calibrated uncertainty; richer models of speaker intent and conversational context. The first is an engineering problem. The second is partially solvable with tools and still hard. The third is unsolved.

References

• Grice, P. H. (1975). Logic and conversation. In P. Cole & J. Morgan (Eds.), Syntax and Semantics, Vol. 3: Speech Acts (pp. 41–58). Academic Press.

• Levinson, S. C. (1983). Pragmatics. Cambridge University Press.

The Video

Someone asked their car’s built-in AI assistant: “Should I drive to the car wash today?” It was raining. The assistant said yes, enthusiastically, with reasons: regular washing extends the life of the paintwork, removes road salt, and so on. Technically correct statements, all of them. Completely beside the point.

The clip spread. The reactions were the usual split: one camp said this proves AI is useless, the other said it proves people expect too much from AI. Both camps are arguing about the wrong thing.

The interesting question is: why did the model fail here, and is this the same kind of failure as the strawberry problem?

It is not. The failures look similar from the outside — confident wrong answer, context apparently ignored — but the underlying causes are different, and the difference matters if you want to understand what these systems can and cannot do.

The Strawberry Problem Was About Representation

In the strawberry case, the model failed because of the gap between its input representation (BPE tokens: “straw” + “berry”) and the task (count the character “r”). The character information was not accessible in the model’s representational units. The model understood the task correctly — “count the r’s” is unambiguous — but the input structure did not support executing it.

That is a representation failure. The information needed to answer correctly was present in the original string but was lost in the tokenisation step.

The car wash case is different. The model received a perfectly well-formed question and had no representation problem at all. “Should I drive to the car wash today?” is tokenised without any information loss. The model understood it. The failure is that the correct answer depends on information that was never in the input in the first place.

The Missing Context

What would you need to answer “should I drive to the car wash today?” correctly?

• The current weather (is it raining now?)
• The weather forecast for the rest of the day (will it rain later?)
• The current state of the car (how dirty is it?)
• Possibly: how recently was it last washed, what kind of dirt (road salt after winter, tree pollen in spring), whether there is a time constraint

None of this is in the question. A human asking the question has access to some of it through direct perception (look out the window) and some through memory (I just drove through mud). A language model has access to none of it.

Let $X$ denote the question and $C$ denote this context — the current state of the world that the question is implicitly about. The correct answer $A$ is a function of both:

The model has $X$. It does not have $C$. What it produces is something like an expectation over possible contexts, marginalising out the unknown $C$:

Averaged over all plausible contexts in which someone might ask this question, “going to the car wash” is probably a fine idea — most of the time when people ask, it is not raining and the car is dirty. $\hat{A}$ is therefore approximately “yes.” The model returns “yes.” In this particular instance, where $C$ happens to include “it is currently raining,” $\hat{A} \neq f(X, C)$.

The quantity that measures how much the missing context matters is the mutual information between the answer and the context, given the question:

Here $H(A \mid X)$ is the residual uncertainty in the answer given only the question, and $H(A \mid X, C)$ is the residual uncertainty once the context is also known. For most questions in a language model’s training distribution — “what is the capital of France?”, “how do I sort a list in Python?” — this mutual information is near zero: the context does not change the answer. For situationally grounded questions like the car wash question, it is large: the answer is almost entirely determined by the context, not the question.

Why the Model Was Confident Anyway

This is the part that produces the most indignation in the viral clips: not just that the model was wrong, but that it was confident about being wrong. It did not say “I don’t know what the current weather is.” It said “yes, here are five reasons you should go.”

Two things are happening here.

Training distribution bias. Most questions in the training data that resemble “should I do X?” have answers that can be derived from general knowledge, not from real-time world state. “Should I use a VPN on public WiFi?” “Should I stretch before running?” “Should I buy a house or rent?” All of these have defensible answers that do not depend on the current weather. The model learned that this question form typically maps to answers of the form “here are some considerations.” It applies that pattern here.

No explicit uncertainty signal. The model was not trained to say “I cannot answer this because I lack context C.” It was trained to produce helpful-sounding responses. A response that acknowledges missing information requires the model to have a model of its own knowledge state — to know what it does not know. This is harder than it sounds. The model has to recognise that $I(A; C \mid X)$ is high for this question class, which requires meta-level reasoning about information structure that is not automatically present.

This is sometimes called calibration: the alignment between expressed confidence and actual accuracy. A well-calibrated model that is 80% confident in an answer is right about 80% of the time. A model that is confident about answers it cannot possibly know from its training data is miscalibrated. The car wash video is a calibration failure as much as a grounding failure.

What Grounding Means

The term grounding in AI has a precise origin. Harnad (1990) used it to describe the problem of connecting symbol systems to the things they refer to — how does the word “apple” connect to actual apples, rather than just to other symbols? A symbol system that only connects symbols to other symbols (dictionary definitions, synonym relations) has the form of meaning without the substance.

Applied to language models: the model has rich internal representations of concepts like “rain,” “car wash,” “dirty car,” and their relationships. But those representations are grounded in text about those things, not in the things themselves. The model knows what rain is. It does not know whether it is raining right now, because “right now” is not a location in the training data.

This is not a solvable problem by making the model bigger or training it on more text. More text does not give the model access to the current state of the world. It is a structural feature of how these systems work: they are trained on a static corpus and queried at inference time, with no automatic connection to the world state at the moment of the query.

What Tool Use Gets You (and What It Doesn’t)

The standard engineering response to grounding problems is tool use: give the model access to a weather API, a calendar, a search engine. Now when asked “should I go to the car wash today?” the model can query the weather service, get the current conditions, and factor that into the answer.

This is genuinely useful. The model with a weather tool call will answer this question correctly in most circumstances. But tool use solves the problem only if two conditions hold:

1. The model knows it needs the tool. It must recognise that this question has $I(A; C \mid X) > 0$ for context $C$ that a weather tool can provide, and that it is missing that context. This requires the meta-level awareness described above. Models trained on tool use learn to invoke tools for recognised categories of question; for novel question types, or questions that superficially resemble answerable ones, the tool call may not be triggered.

2. The right tool exists and returns clean data. Weather APIs exist. “How dirty is my car?” does not have an API. “Am I the kind of person who cares about car cleanliness enough that this matters?” has no API. Some missing context can be retrieved; some is inherently private to the person asking.

The deeper issue is not tool availability but knowing what you don’t know. A model that does not recognise its own information gaps cannot reliably decide when to use a tool, ask a clarifying question, or express uncertainty. This is a hard problem — arguably harder than making the model more capable at the tasks it already handles.

The Contrast, Stated Plainly

The strawberry failure and the car wash failure look alike from the outside — confident wrong answer — but they are different enough that conflating them produces confused diagnosis and confused solutions.

Strawberry: the model has the information (the string “strawberry”), but the representation (BPE tokens) does not preserve character-level structure. The fix is architectural or procedural: character-level tokenisation, chain-of-thought letter spelling.

Car wash: the model does not have the information (current weather, car state). No fix to the model’s architecture or prompt engineering gives it information it was never given. The fix is exogenous: provide the context explicitly, or give the model a tool that can retrieve it, or design the system so that context-dependent questions are routed to systems that have access to the relevant state.

A model that confidently answers the car wash question without access to current conditions is not failing at language understanding. It is behaving exactly as its training shaped it to behave, given its lack of situational grounding. Knowing which kind of failure you are looking at is most of the work in figuring out what to do about it.

The grounding problem connects to the broader question of what it means for a language model to “know” something — which comes up in a different form in the context window post, where the issue is not missing context but irrelevant context drowning out the relevant signal.

A second car wash video a few weeks later produced a third, different failure: Car Wash, Part Three: The AI Said Walk — the model had the right world state but chose the wrong interpretation of the question.

References

• Harnad, S. (1990). The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(1–3), 335–346. https://doi.org/10.1016/0167-2789(90)90087-6

• Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On calibration of modern neural networks. ICML 2017. https://arxiv.org/abs/1706.04599

I agree with the impulse. I am less sure about what the phrase actually asks of us, or what it assumes is possible. This post is my attempt to think through that question — and to document a small project that emerged from it.

A Personal Starting Point

I am on the spectrum. I was diagnosed in adulthood, which is not unusual, and the diagnosis explained a great deal about a life spent finding some things effortless and others bewildering.

Code is easy. The internal structure of a problem, the satisfaction of a clean abstraction, the deep rabbit holes that open when a concept catches my attention and refuses to let go — that is the natural medium. Hyperfocus is not a metaphor for me; it is literally how I spend a Tuesday afternoon. I have written entire systems because I could not stop.

Emotions are harder. Not absent — that is a misconception I will address in a moment — but differently structured. Reading a room is work. Social cues that seem to operate as obvious background noise for most people arrive for me as data that requires conscious decoding. The reverse appears to be true for most neurotypical people: emotional processing runs in the background, effortlessly; formal abstraction requires deliberate effort.

Neither is better. They are different cognitive architectures, and both come with costs.

I raise this not to centre myself, but because it is relevant to the question the post is actually about. I spent years navigating a social world that was not built for how I process it. That experience sits close to the experience of people with mental illness — not the same, but adjacent. And it made me think hard about what “understanding” across neurological difference actually means.

Mental Illness Is Still a Grey Zone

The progress on mental health stigma over the past decade is real. People talk about therapy more openly than they did. Burnout is acknowledged at work. The language of mental health has entered mainstream use — sometimes usefully, sometimes in ways that dilute clinical concepts into lifestyle descriptors. Anxiety is now a brand attribute. Trauma is a metaphor for mild inconvenience. This is a problem, but it is a second-order problem; the first-order problem — that serious mental illness is still heavily stigmatised, underfunded, and misunderstood — is the one that matters more.

Corrigan and Watson [1] documented what the stigma research consistently shows: people with mental illness face two compounding problems. The first is public stigma [3] — the prejudice of others, leading to discrimination in employment, housing, relationships. The second is self-stigma — the internalised application of those same prejudices to oneself. The second is often worse. It is the mechanism by which stigma becomes a barrier to seeking help, creating the feedback loop that keeps serious mental illness invisible precisely because the people experiencing it have been taught that it is shameful.

The phrase “make it visible” is a response to this dynamic. If mental illness is visible — discussed, depicted, normalised — the argument goes that stigma decreases. There is evidence for this. Contact-based interventions, where people without mental illness interact with people who have it, consistently outperform education-only approaches [2]. The visibility of real people matters more than information campaigns.

But there is a difference between visibility and understanding.

What Visibility Actually Achieves

When we say “make it visible”, we usually mean one of several different things, which are worth separating.

Normalisation means that a condition becomes part of accepted human variation rather than a mark of failure or danger. This is achievable through visibility and is genuinely important. Knowing that a colleague takes antidepressants, or that a public figure manages bipolar disorder, reduces the sense of aberration. It does not require the observer to understand the experience — only to register that it exists and is survivable.

Representation means that people with a condition see themselves reflected in culture, media, and institutions. This matters for the affected person; it is about recognition, not about inducing empathy in the non-affected.

Empathy is the hardest and most frequently over-promised goal. It is what the simulation approaches aim for: put a neurotypical person in a room with distorted audio and flickering visuals and tell them this is what psychosis sounds like. Does it work?

The honest answer from the research is: somewhat, temporarily, and with significant caveats.

The Empathy Gap

Let me be direct about something. A person who has never experienced severe depression cannot know what it is. Not in the way that a person who has experienced it knows it. This is not a failure of empathy or imagination; it is a structural fact about how knowledge of mental states works.

Philosophers call this the problem of other minds. We have no direct access to another person’s experience. We infer it, imperfectly, by analogy to our own. For experiences that have no analogue in our own history, inference breaks down. You can read every clinical description of dissociation ever written and still not know what dissociation is, because the knowledge that matters is not propositional — it is not a set of facts — but experiential.

This is the gap that simulation approaches try to bridge, and it is genuinely unbridgeable. What simulation can do is something weaker but not worthless: it can create an affective response, a discomfort, a disruption of the observer’s normal processing, that functions as a rough proxy signal. Not “now you know what it is like”, but “now you have a small, incomplete, distorted approximation of some dimension of the experience”.

The risk is misrepresentation. Schizophrenia simulations have been criticised — fairly — for reducing a complex condition to its most dramatic phenomenological features (auditory hallucinations, paranoia) while omitting the cognitive, relational, and longitudinal aspects that define how people actually live with the condition. A five-minute visual experience of “what depression feels like” that emphasises darkness and slow motion tells you almost nothing about the specific exhaustion of getting through a Tuesday morning, or the way time warps over months.

So: you cannot truly understand what you have not experienced. But you can try to approximate something, and approximation, done honestly and with appropriate epistemic humility, is better than nothing.

Metaphor as a Communication Tool

There is a long tradition of using metaphor and art to communicate internal states that resist direct description. This is not a bug; it is a feature of how language handles subjective experience.

The poet uses metaphor because “my heart is heavy” is not literally true but captures something that “I am experiencing low mood” does not. The musician uses dissonance and rhythm to structure emotional experience in the listener. The visual artist uses colour and texture to evoke states rather than depict them. None of these are representations in the scientific sense — they do not accurately model the referent — but they create a kind of resonance that purely descriptive language cannot.

Mental health communication has increasingly moved in this direction. The vocabulary of “emotional weight”, “spiralling”, “crashing”, “the fog” — these are metaphors that have become clinical shorthand precisely because they communicate something essential that clinical terms do not. When someone says “I couldn’t get out of bed”, they are not describing paralysis; they are describing a particular quality of anhedonia and executive dysfunction that no diagnostic manual entry captures as well.

This is the space where a project like inner-echo operates.

Inner Echo: The Idea

inner-echo is a browser-based audiovisual experiment. It takes a webcam feed and applies condition-specific visual and audio effects that function as metaphorical overlays on the user’s own image. The output is not a simulation of a mental health condition in any clinical sense. It is an attempt to construct a visual and auditory language for internal states, using the user’s own presence as the anchor.

The technical architecture is deliberately minimal: React, WebGL/Canvas for video processing, optional WebAudio. Everything runs in the browser, client-side, with no backend. No data leaves the device. This is not incidental — privacy is load-bearing for a project that deals with sensitive self-reflection. Safe Mode and an emergency stop function are built in.

The condition-profile system supports three modes:

• Preset mode: a single-condition metaphorical composition — one set of effects mapped to one cluster of experiences
• Multimorbid mode: weighted stacking of multiple condition profiles, acknowledging that most people with mental health conditions do not have one thing
• Symptom-first mode: dimension-level control, letting the user build from individual symptom representations rather than diagnostic labels

The last of these is, I think, the most honest design choice. Diagnostic categories are administrative conveniences as much as they are natural kinds. Two people with the same diagnosis can have radically different experiences. Structuring the system around dimensions of experience rather than labels is both clinically more accurate and communicatively more flexible.

What It Is Not

Being clear about limitations is not false modesty; it is the only way this kind of project retains its integrity.

inner-echo is not a simulation of any condition in the sense of accurately modelling its phenomenology. It does not claim to show you “what depression is like”. It offers metaphorical approximations of some dimensions of some experiences, and it does so using effects that are legible to the observer — visual distortion, audio modification, altered feedback — that bear a designed but non-literal relationship to the internal states they are meant to evoke.

It is not a diagnostic tool. It is not a therapeutic intervention. It is not a substitute for any clinical process.

What it might be is a starting point for a conversation. Something a person experiencing a condition could use to gesture toward an aspect of their experience. Something a person without that experience could encounter with enough curiosity to ask a better question than they would have otherwise.

That is a modest claim. I think modest claims are appropriate here.

Why This, Why Now

Mental health awareness has become a genre. The awareness campaigns, the celebrity disclosures, the workplace wellness programmes — these are real goods, and I do not want to be cynical about them. But the communication problem has not been solved. The words exist. The willingness to use them, in many contexts, exists. What is still missing is a language for the texture of experience that the words point to but do not reach.

I find myself better able to build something than to explain it in words. That is probably a spectrum thing. inner-echo is an attempt to build toward a language that I do not fully have — for my own internal experience, and for the experiences of people navigating conditions quite different from mine.

The gap cannot be closed. But the attempt to reach across it is worth making, and worth being honest about.

References

[1] Corrigan, P.W. & Watson, A.C. (2002). Understanding the impact of stigma on people with mental illness. World Psychiatry, 1(1), 16–20.

[2] Corrigan, P.W., Morris, S.B., Michaels, P.J., Rafacz, J.D. & Rüsch, N. (2012). Challenging the public stigma of mental illness: A meta-analysis of outcome studies. Psychiatric Services, 63(10), 963–973.

[3] Goffman, E. (1963). Stigma: Notes on the Management of Spoiled Identity. Prentice-Hall.

inner-echo repository: https://github.com/sebastianspicker/inner-echo

In September 2024, OpenAI publicly confirmed that their new reasoning model had been code-named “Strawberry” during development. This landed with a particular thud because “how many r’s are in strawberry?” had, by that point, become one of the canonical demonstrations of language model failure. The model named after strawberry could not count the letters in strawberry. The internet had opinions.

Before the opinions: the answer is three. s-t-r-a-w-b-e-r-r-y. One in the str- cluster, two in the -rry ending. Count carefully and you will find that most people get this right on the first try, and most large language models get it wrong, returning “two” with apparent confidence.

The question worth asking is not “why is the model stupid.” It is not stupid, and “stupid” is not a useful category here. The question is: what does this specific error reveal about the structure of the system?

The answer involves tokenisation, and it is actually interesting.

How You Count Letters (and How the Model Doesn’t)

When you count the r’s in “strawberry,” you do something like this: scan the string left to right, maintain a running count, increment it each time you see the target character. This is a sequential operation over a character array. It requires no semantic knowledge about the word — it does not matter whether “strawberry” is a fruit, a colour, or a nonsense string. The characters are the input; the count is the output.

A language model does not receive a character array. It receives a sequence of tokens — chunks produced by a compression algorithm called Byte Pair Encoding (BPE) that the model was trained with. In the tokeniser used by GPT-class models, “strawberry” is most likely split as:

Three tokens. The model’s input is these three integer IDs, each looked up in an embedding table to produce a vector. There is no character array. There is no letter “r” sitting at a known position. There are three dense vectors representing “str,” “aw,” and “berry.”

What BPE Does (and Doesn’t) Preserve

BPE is a greedy compression algorithm. Starting from individual bytes, it iteratively merges the most frequent pair of adjacent symbols into a single new token:

Applied to a large text corpus until a fixed vocabulary size is reached, this produces a vocabulary of common subwords. Frequent words and common word-parts become single tokens; rare sequences stay as multi-token fragments.

What BPE optimises for is compression efficiency, not character-level transparency. The token “straw” encodes the sequence s-t-r-a-w as a unit, but that character sequence is not explicitly represented anywhere inside the model once the embedding lookup has occurred. The model receives a vector for “straw,” not a list of its constituent letters.

The character composition of a token is only accessible to the model insofar as it was implicitly learned during training — through seeing “straw” appear in contexts where its internal structure was relevant. For most tokens, most of the time, that character structure was not relevant. The model learned what “straw” means, not how to spell it character by character.

Why the Error Is Informative

Most people say the model returns “two r’s,” not “one” or “four” or “none.” This is not random noise. It is a systematic error, and systematic errors are diagnostic.

“berry” contains two r’s: b-e-r-r-y. If you ask most models “how many r’s in berry?” they get it right. The model has seen that question, or questions closely enough related, that the right count is encoded somewhere in the weight structure.

“str” contains one r: s-t-r. But as a token it is a short, common prefix that appears in hundreds of words — string, strong, stream — contexts in which its internal letter structure is rarely attended to. “aw” contains no r’s. When the model answers “two,” it is almost certainly counting the r’s in “berry” correctly and failing to notice the one in “str.” The token boundaries are where the error lives.

This is not stupidity. It is a precise failure mode that follows directly from the tokenisation structure. You can predict where the error will occur by looking at the token split.

Chain of Thought Partially Fixes This (and Why)

If you prompt the model to “spell out the letters first, then count,” the error rate drops substantially. The reason is not mysterious: forcing the model to generate a character-by-character expansion — s, t, r, a, w, b, e, r, r, y — puts the individual characters into the context window as separate tokens. Now the model is not working from “straw” and “berry”; it is working from ten single-character tokens, and counting sequential characters in a flat list is a task the model handles much better.

This is, in effect, making the model do manually what a human does automatically: convert the compressed token representation back to an enumerable character sequence before counting. The cognitive work is the same; the scaffolding just has to be explicit.

The Right Frame

The “how many r’s” test is sometimes cited as evidence that language models don’t “really” understand text, or that they are sophisticated autocomplete engines with no genuine knowledge. These framing choices produce more heat than light.

The more precise statement is this: language models were trained to predict likely next tokens in large text corpora. That training objective produces a system that is very good at certain tasks (semantic inference, translation, summarisation, code generation) and systematically bad at others (character counting, exact arithmetic, precise spatial reasoning). The system is not doing what you are doing when you read a sentence. It is doing something different, which happens to produce similar outputs for a very wide range of inputs — and different outputs for a class of inputs where the character-level structure matters.

“Strawberry” sits squarely in that class. The model is not failing to read the word. It is succeeding at predicting what a plausible-sounding answer looks like, based on a compressed representation that does not preserve the information needed to get the count right. Those are not the same thing, and the distinction is worth keeping clear.

The tokenisation argument here is a simplified version. Real BPE vocabularies, positional encodings, and the specific way character information is or isn’t preserved in embedding tables are more complicated than this post suggests. But the core point — that the model’s input representation is not a character array and never was — holds.

A follow-up post covers a structurally different failure mode: Should I Drive to the Car Wash? — where the model understood the question perfectly but lacked access to the world state the question was about.

References

• Gage, P. (1994). A new algorithm for data compression. The C Users Journal, 12(2), 23–38.

• Sennrich, R., Haddow, B., & Birch, A. (2016). Neural machine translation of rare words with subword units. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016), 1715–1725. https://arxiv.org/abs/1508.07909

Changelog

• 2025-12-01: Corrected the tokenisation of “strawberry” from two tokens (straw|berry) to three tokens (str|aw|berry), matching the actual cl100k_base tokeniser used by GPT-4. The directional argument (token boundaries obscure character-level information) is unchanged; the specific analysis was updated accordingly.

Why Misconceptions Are Not Just Wrong Answers

Before the list, a clarification that matters pedagogically.

When education researchers say “misconception,” they do not mean a random error or a gap in knowledge. A misconception is a stable, self-consistent mental model that students actively use to interpret new information. It persists not because the student hasn’t heard the correct explanation but because the incorrect model handles a wide range of everyday experience reasonably well.

“Fire is a thing that makes heat and light and consumes fuel” is a perfectly adequate mental model for everything a student encounters outside a physics class. It explains candles, campfires, gas hobs, and car engines. The fact that it also leads the same student to conclude that the Sun “burns” in the chemical combustion sense is not a failure of intelligence — it is the natural extension of a model that works.

The implication, which Bransford, Brown, and Cocking put plainly in 2000: if you ignore what students already believe and simply present the correct model, “the understanding they develop can vary substantially from what the instructor intended.” The new information gets interpreted through the existing model, not in place of it. You end up with students who can repeat “the Sun fuses hydrogen” while still, in their mental model, imagining it as a very large and very hot fire.

With that said: here is the list.

The Sun Is Not a Star

This one leads because it is the most structurally interesting.

Bailey et al. (2009), in a study of students’ pre-instructional ideas about stars and star formation, document the following category of response: the Sun is a special kind of astronomical body with its own distinct properties. It is not a star. Stars are the things you see in the night sky. The Sun is different.

This is not an isolated finding. Schecker et al. (2018) document the same pattern in the German context. Students who know perfectly well that “the Sun is a star” as a stated fact will nonetheless, when asked to reason about stellar properties, implicitly exempt the Sun from those properties. Stars are far away, they are small and faint, they are cold and distant. The Sun is close, large, and bright. Ergo the Sun cannot really be a star, whatever the textbook says.

The pedagogical consequence is that teaching stellar evolution to students who hold this model requires first collapsing the Sun/star distinction — otherwise everything that follows is about something unfamiliar and distant rather than about the object eight light minutes away that we can observe in detail.

A companion misconception: all stars are smaller than the Sun. This is the inverse problem. Students who correctly classify the Sun as a star but have only seen stars as faint points of light infer that stars must be small. The Sun, which they know to be large, therefore cannot be a typical star. Betelgeuse — a red supergiant with a radius approximately 700 times the Sun’s, which if placed at the Sun’s position would extend past the asteroid belt — tends to produce strong cognitive dissonance when it is first encountered.

The Sun Is on Fire

The combustion model of stellar energy is, empirically, the most common student conception and the hardest to dislodge.

From Favia et al.’s misconception inventory, translated loosely:

• “The Sun is made of fire.”
• “Stars run on fuel: petrol or natural gas.”
• “The Sun is made of molten lava.”
• “The Sun is a heat-planet.”

Bailey et al.’s quantitative data: when asked how stars produce light, 32% of students described chemical burning. A further 28% described unspecified “chemical reactions.” Only 7% named nuclear fusion. Only 3% could both name fusion and correctly connect it to the production of light.

The combustion model is coherent and consistent. It gives you a mechanism (fuel + oxygen → heat and light), a timescale (stars eventually run out of fuel and go dark), and a product (visible light and heat). What it cannot handle is the scale: the Sun has been burning for 4.6 billion years and has approximately 5 billion years of fuel remaining. Chemical combustion at the Sun’s luminosity would exhaust any chemically plausible fuel supply in tens of thousands of years. This is the crack in the model that fusion fills — not by saying “the Sun burns differently” but by replacing the entire energy mechanism with one that operates at scales the combustion model cannot reach.

One related misconception worth noting explicitly: the Sun is hottest at its surface. This is the intuitive model — things are hot near the fire and cooler further away. The corona’s temperature of a million Kelvin, far above the photospheric 5,778 K, violates this so thoroughly that it remained an active research problem for decades (and, in some senses, still is). Students encountering coronal heating for the first time do not usually reject it, but they do find it genuinely strange in a way they cannot articulate — which is the signature of something colliding with a stable prior model.

Gravity Only Works When Things Move

The gravity misconceptions documented in the research literature are worth treating separately because they have direct consequences for understanding stellar formation — which depends entirely on gravity acting on stationary or slowly drifting gas clouds.

The relevant findings:

Gravity requires motion (Palmer, 2001). A significant proportion of students believe that gravity only acts on objects that are in motion. A stationary object is not subject to gravitational attraction. A table sitting on the floor: fine, no gravity needed. A gas cloud drifting slowly through space: also fine. A gas cloud being compressed by gravitational self-attraction: this requires gravity to act on particles that are not yet moving, which the model cannot accommodate.

Force implies movement (Gunstone & Watts, 1985). The more general version: forces produce motion, and where there is no net motion, there is no net force. The concept of force balance — two equal and opposite forces summing to zero net force, with the object not moving — is not available to students holding this model. It is hard to overstate how consequential this is for astrophysics. Almost every stable astrophysical structure — a main-sequence star, a planetary orbit, a galaxy’s rotation — is a force balance. Students without the concept cannot reason about any of them correctly.

Gravity only acts on Earth (Bar, Brosh, and Sneider, 2016). Students in the space context often reason that gravity is a property of Earth specifically. In space, things are “weightless” — and weightlessness is interpreted as the absence of gravity rather than as the experience of free fall in a gravitational field. The result: gravity cannot be the mechanism by which an interstellar gas cloud collapses, because gas clouds are in space and gravity does not work there. Asghar and Libarkin (2010) found that only one in five non-physics college students could correctly describe gravity as an attractive force between masses, using the correct vocabulary.

These are not fringe findings. They are the majority conception at the pre-instructional stage. Any unit on stellar formation that opens with “gravity compresses the gas cloud” is speaking to students who mostly do not believe that gravity can do that to a gas cloud in space.

Metals Always Existed

This misconception is my personal favourite because it requires no incorrect intuition — it requires an absence of information that most people have never had reason to acquire.

Students and adults who have not encountered stellar nucleosynthesis simply have no model for where heavy elements come from. Asked directly, a common response is that metals “always existed” — they are a feature of the universe, present from the beginning. The alternative framing: “stars create matter from nothing” — which captures the sense that something is being generated, without a mechanism.

The correct picture: the Big Bang produced primarily hydrogen and helium, with trace amounts of lithium and beryllium. Every heavier element — including all the carbon in your body, all the iron in your blood, all the oxygen in every breath — was synthesised in a stellar interior or in a supernova. The gold in a wedding ring was produced in a neutron star merger. We are, in the precise sense of the phrase, made of star stuff; but not because stars are somehow magical, because the nuclear physics of stellar interiors and violent stellar deaths is the only process in the universe that can manufacture these elements.

This has a direct implication for stellar evolution education: if students believe metals always existed, the cycle of stellar death and new star formation — in which dying stars enrich the interstellar medium with heavy elements that become part of the next generation of stars and their planets — loses most of its meaning. The cycle is interesting precisely because it explains why later-generation stars and their planets have a richer elemental composition than first-generation stars. Remove that frame and you have a sequence of events with no cumulative significance.

Some Language-Based Misconceptions (A Brief Digression)

Since I promised something about quantum leaps: the phrase “quantum leap” in everyday usage means a sudden, large, discontinuous advance. In physics, a quantum transition is the smallest possible discrete change in a system’s energy state. The electron moves from one energy level to another; the photon is emitted or absorbed; the scale of change is on the order of electron-volts. It is, emphatically, not large.

The astronomy version of this class of error:

“Light year” used as a unit of time. “That happened light years ago.” A light year is the distance light travels in one year — approximately 9.46 × 10¹² kilometres. It is a unit of distance, not time. This one is so embedded in everyday usage that correcting it usually produces mild annoyance rather than reconsideration.

“Shooting stars.” Meteors — small rocky or metallic bodies entering the atmosphere — have nothing to do with stars. They are typically the size of a grain of sand to a pebble. The visual resemblance to a moving point of light crossing the sky is where the name comes from; the resemblance to stellar physics is zero.

“Black holes suck things in.” Black holes do not have more gravity than the object that formed them at the same distance. If the Sun were replaced by a black hole of equal mass, the planets would continue on their current orbits. A black hole is only a black hole within its Schwarzschild radius; beyond that it is a gravitational field like any other. What black holes have is a point of no return — the event horizon — beyond which escape velocity exceeds the speed of light. They do not actively pull. They are very massive objects that objects can fall into.

“The dark side of the Moon.” The Moon has a far side (permanently facing away from Earth, due to tidal locking) and a near side. Both sides receive approximately equal sunlight over the lunar cycle. The far side is not permanently dark; it has a day and a night like the near side. “Dark side” persists in common usage because Pink Floyd used it as an album title and nobody wanted to call it “The Far Side of the Moon.” (Although Douglas Adams would have had something to say about that.)

Why This List Matters for Teaching

The misconceptions described above are not randomly distributed. They cluster around three areas where intuitive extrapolation from everyday experience leads systematically away from the correct physics:

1. Scale: human intuition was not built for 150 million kilometres, let alone 4.6 billion years or the 9.46 × 10¹² km in a light year. The Sun cannot be fire because fire cannot last 4.6 billion years; but “4.6 billion years” is not a number that everyday experience makes graspable.

2. Energy mechanism: combustion is the dominant frame for “things that produce heat and light.” Nuclear fusion is not part of everyday experience at any scale. The conceptual distance between them is not factual but mechanistic — it requires replacing an entire causal model.

3. Gravity: our direct experience of gravity is of a downward force, active at Earth’s surface, which keeps things from floating away. The idea of gravity as a universal mutual attraction between all masses — active in empty space, responsible for cloud collapse and galaxy formation — is a substantive generalisation that everyday experience does not motivate.

The pedagogical literature’s recommendation is not to avoid these topics but to surface the prior models explicitly before presenting the correct physics. If you ask students “where does the Sun’s energy come from?” before you teach nuclear fusion, you learn what they believe and you create the cognitive conditions for productive conceptual conflict. If you simply present the fusion model without that step, students add “fusion” to their vocabulary while retaining “fire” in their mental model.

The experiments Alexander Küpper and I have been developing through the astro-lab project — described in the stellar evolution post and the astro-lab@home post — are designed with these specific misconceptions in mind. The net-based gravity experiment addresses the “gravity doesn’t work in space” and “force requires motion” problems directly, by making gravitational attraction between all particles visible as a material structure. The pressure-temperature experiment makes the “compression heats the gas” step concrete before any mention of fusion.

These are not complete solutions to deeply held misconceptions. But they are a start at building the conceptual scaffolding that makes “and then fusion begins” something other than an assertion to be memorised and filed away without understanding.

References

Asghar, A. A., & Libarkin, J. C. (2010). Gravity, magnetism, and “down”: Non-physics college students’ conceptions of gravity. The Science Educator, 19(1), 42–55.

Bailey, J. M., Prather, E. E., Johnson, B., & Slater, T. F. (2009). College students’ preinstructional ideas about stars and star formation. Astronomy Education Review, 8(1). https://doi.org/10.3847/AER2009038

Bar, V., Brosh, Y., & Sneider, C. (2016). Weight, mass, and gravity: Threshold concepts in learning science. Science Educator, 25(1), 22–34.

Bransford, J. D., Brown, A. L., & Cocking, R. R. (Eds.) (2000). How People Learn: Brain, Mind, Experience, and School. National Academy Press.

Favia, A., Comins, N. F., & Thorpe, G. L. (2013). The elements of item response theory and its framework in analyzing introductory astronomy college student misconceptions. I. Galaxies. Astronomy Education Review.

Gunstone, R., & Watts, M. (1985). Force and motion. In R. Driver, E. Guesne, & A. Tiberghien (Eds.), Children’s Ideas in Science (pp. 85–104). Open University Press.

Palmer, D. (2001). Students’ alternative conceptions and scientifically acceptable conceptions about gravity. International Journal of Science Education, 23(7), 691–706. https://doi.org/10.1080/09500690010006527

Schecker, H., Wilhelm, T., Hopf, M., & Duit, R. (Eds.) (2018). Schülervorstellungen und Physikunterricht. Springer.

Changelog

• 2025-09-14: Updated the DOI for Bailey et al. (2009) to the correct 10.3847/AER2009038.
• 2025-09-14: Changed “would extend past Mars” to “would extend past the asteroid belt” for Betelgeuse at ~700 R☉. At ~3.26 AU, Betelgeuse’s radius exceeds Mars’s orbital distance (1.52 AU) by more than a factor of two and reaches well into the asteroid belt (2.2–3.3 AU).