There is a popular intuition in LLM engineering that context is a resource you should always spend freely: more background, more history, more examples — inevitably better answers. This intuition is wrong often enough to be dangerous. Context has a signal-to-noise structure, attention has a position-dependent bias, and the architecture that processes all of it scales quadratically. Adding irrelevant tokens does not leave performance neutral; it actively degrades it. This post argues for structured sparsity as a design principle: give a model exactly the context it needs for the decision it is making right now, and nothing else.

Background

The “more is more” assumption has an obvious origin. Transformers were designed to condition on sequences, and every new token in the context window is, in principle, available to every attention head. The release of models with 128k, 200k, and now million-token context windows reinforced the story: the constraint is gone, so pack in everything you have.

Two lines of empirical and theoretical work complicate this story.

The lost-in-the-middle problem. Liu et al. [1] showed that retrieval accuracy on multi-document question answering degrades sharply when the relevant passage appears in the middle of a long context, compared to the beginning or end. Performance on 20-document prompts dropped by more than 20 percentage points relative to the single-document baseline — not because the model lacked the information, but because it was buried. The effect is consistent across model families and persists at model scales where you would not expect it.

The complexity argument. Standard scaled dot-product attention [2] is

The \( QK^{\top} \) product is \( O(n^2) \) in sequence length \( n \). Inference-time KV-cache mitigates compute cost, but memory grows linearly and the softmax normalises over a denominator that grows with \( n \). A head attending to 128 000 tokens is averaging over a vastly noisier signal than one attending to 512.

The Idea

Context as a noisy channel

Think of the information reaching a given attention head as a noisy channel in the Shannon sense. The signal is the subset of tokens that are actually relevant to the current decoding step; the rest is noise. Signal-to-noise ratio is

where \( \mathcal{S} \subset \{1, \ldots, n\} \) is the set of relevant token positions and \( n \) is total context length. For a fixed task, \( |\mathcal{S}| \) is roughly constant. So SNR is a decreasing function of \( n \). Adding irrelevant context makes the problem strictly harder in this framing — it does not leave it unchanged.

This is a toy model, but it captures something real. The softmax in the attention head distributes a probability mass of 1.0 across \( n \) positions. If the attended sequence doubles in length and the relevant positions remain the same, each relevant position receives roughly half the probability mass it did before — unless the model’s learned attention patterns are precise enough to suppress the irrelevant positions to near-zero, which is a strong assumption.

Position bias compounds the problem

Empirically, transformers exhibit a U-shaped recall curve over context position: tokens near the start (primacy) and tokens near the end (recency) are retrieved more reliably than tokens in the middle. If you stuff a long context with background material and bury the task-relevant information in the middle, you are fighting the architecture’s learned inductive bias.

The effect is roughly consistent with what would emerge if the model’s attention weight distribution were modelled as a mixture of a flat prior and a position-biased component. Under that model, increasing \( n \) inflates the flat component’s contribution and dilutes the position-biased recovery of relevant tokens.

What structured sparsity looks like in practice

The corrective is not to artificially shrink context windows — it is to ensure that at each decision point, the context is populated with tokens that are relevant to that decision. Three practical expressions of this principle:

1. Retrieval over recall. Rather than prepending a full document corpus, retrieve the top-\( k \) passages at query time. This keeps \( n \) small and \( |\mathcal{S}| / n \) high.

2. Rolling summarisation. Compress history into a running summary and discard the raw transcript. The summary carries the signal; the raw transcript is mostly noise by the time it is several turns old.

3. Phased orchestration. Decompose a multi-step task into phases, each with its own focused context. Phase \( t \) receives only the output of phase \( t-1 \) (plus any task-specific retrieval), not the entire accumulated history of all prior phases. This keeps per-phase \( n \) bounded even as the total task length grows.

Discussion

The argument above is not novel — pieces of it appear scattered across the alignment and inference-efficiency literatures. What I think is underappreciated is that it applies to agentic systems with particular force. A single-shot prompt has a fixed, author-controlled context. An agent accumulating tool outputs, prior reasoning traces, and retrieved documents across a long task trajectory will naturally inflate its own context window over time — and degrade its own performance as a result, without any external change in task difficulty.

The naive fix is to give the agent a bigger context window. The correct fix is to never let it accumulate a bloated context in the first place.

Limitations. The SNR framing treats all irrelevant tokens as equally noisy, which is false — some irrelevant tokens are actively misleading (distractors) [3], others are benign fillers. The quadratic cost argument mostly applies to full-attention models; sparse and linear attention variants have different scaling properties. And “relevant” is itself a function of the model’s knowledge, which makes the optimisation circular in practice.

What would make this publishable. Controlled ablation: fix a task, vary context length by inserting null tokens of increasing volume, measure performance as a function of \( n \) and of the position of the relevant material. Do this across model sizes and families to separate architectural effects from scale effects. The lost-in-the-middle paper is close to this but does not isolate null-token inflation from document-count inflation.

References

[1] Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024). Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12, 157–173. https://arxiv.org/abs/2307.03172

[2] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30. https://arxiv.org/abs/1706.03762

[3] Shi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E., Schärli, N., & Zhou, D. (2023). Large language models can be easily distracted by irrelevant context. Proceedings of the 40th International Conference on Machine Learning (ICML 2023). https://arxiv.org/abs/2302.00093

The phased orchestration argument in the Discussion section is not just theoretical hand-waving — I have been building a concrete implementation of it. The current state lives at sebastianspicker/phased-agent-orchestration. It is rough, but the core idea is there: each agent phase gets a bounded, purpose-built context rather than the full accumulated history. Feedback very welcome.

It will also be wrong about what that means.

The Constitution Problem

In 2023, as universities began deploying AI detection tools at scale, educators started testing them on texts that were definitively not AI-generated. The results were instructive. The United States Constitution received high AI-probability scores from multiple commercial detectors. GPTZero returned a rating of 92% likely AI-written. The Federalist Papers fared similarly. So did sections of the King James Bible and Kant’s Critique of Pure Reason. Historical documents, written by humans, for human purposes, in an era when no AI existed — flagged as machine-generated.

This was not a marginal edge case. It was consistent across tools and across documents. And while it was widely reported as evidence that the detectors were broken, there is a more precise reading available: the detectors were working correctly, and we had misunderstood what they were measuring.

What the Detectors Actually Measure

Most commercial AI detectors — GPTZero, Turnitin’s detection layer, Copyleaks — use some combination of two statistical signals.

Perplexity. A language model assigns a probability to each token given the preceding tokens. Low perplexity means the text was, token by token, what the model expected — it sits close to the centre of the probability distribution. AI-generated text tends to have low perplexity because that is precisely what generation does: it samples from the high-probability region of the distribution [1]. Human text, on average, has higher perplexity, because humans write for specific contexts with idiosyncratic word choices, rhetorical effects that require the unexpected, and the accumulated noise of composing for a real reader.

Burstiness. A term introduced by Edward Tian, GPTZero’s creator: human writing has high burstiness — sentence lengths vary widely, vocabulary density shifts, complex constructions alternate with simple ones. AI writing is more uniform. The statistical distribution of sentence lengths in LLM output is narrower than in most human prose [2].

The underlying assumption these tools share: human writing is variable, contextually messy, idiosyncratic. AI writing is smooth and predictable.

This is accurate for a large class of human writing — casual prose, personal essays, social media, student writing in informal registers. It is wrong about a different and well-defined class of human writing. The Constitution sits in that class. So does a lot of other text.

The Systemising Brain

Simon Baron-Cohen’s empathising–systemising (E-S) theory distinguishes two cognitive orientations. Empathising involves attending to social and emotional cues, inferring mental states, navigating the pragmatic, implicit layer of communication — what is meant rather than what is said. Systemising involves attending to rules, patterns, and underlying regularities — the drive to understand how things work and to represent them in explicit, transferable, internally consistent terms [3].

Both orientations are distributed across the human population. They are not exclusive, and neither is pathological. But autism spectrum conditions are robustly associated with high systemising and relatively lower empathising — not because autistic people lack emotions or care about others, but because the cognitive mode that comes naturally to them is one of rules, structures, and explicit representation rather than social inference and pragmatic implication. The intense world theory [4] adds a complementary perspective: autistic brains may be characterised by hyper-reactivity and hyper-plasticity, with pattern-seeking and systematising serving partly as a way of making a too-intense world navigable. The systematicity is not a deficit. It is an adaptation.

This has direct consequences for writing.

High-systemising writing tends toward:

• Consistent vocabulary. The same term is used for the same concept throughout, because substituting a synonym introduces ambiguity about whether the referent is actually the same. Neurotypical writing freely uses synonyms for stylistic variety; systemising writing resists this on principle.

• Explicit logical structure. Claims are supported by stated reasons rather than left to pragmatic inference. If there are three conditions, all three are named. Nothing is “needless to say.”

• Low social hedging. Phrases like “as everyone knows” or “obviously” are avoided, because they perform social alignment rather than convey information — and they depend on shared assumptions the writer is not confident are actually shared. (This connects to a point I made in the car-wash-walk post about Gricean pragmatics: autistic communication often violates the maxim of quantity in the direction of over-informing, because nothing is assumed implicit.)

• Grammatical parallelism. Parallel logical content takes parallel grammatical form. This is not stylistic affectation; it is a natural consequence of representing structure explicitly.

• Minimal rhetorical noise. The prose does not meander, warm up, or perform relatability. It states what needs to be stated.

Now run text with these properties through an AI detector. Consistent vocabulary reads as low lexical diversity. Explicit structure reads as low burstiness. Minimal rhetorical noise reads as smooth, generated output. The detector is measuring these properties accurately. The attribution to machine generation is where it goes wrong.

Liang et al. [5] demonstrated a closely related failure empirically: AI detectors are significantly more likely to flag writing by non-native English speakers as AI-generated. Non-native writers at advanced levels of formal English tend to write more carefully, more consistently, and more in accordance with explicit grammar rules — because they learned the language as a system of explicit rules rather than acquiring it through immersive social exposure. More systematic writing: higher AI probability score. The mechanism is the same. The population is different.

The Physicist Brain

Physics writing has its own conventions, independently developed but pointing in the same direction.

Scientific prose requires defined terms used consistently: in a paper about quantum error correction, “logical qubit,” “physical qubit,” and “syndrome” each mean exactly one thing, used identically in section 2 and section 5. It requires explicit assumptions: “We assume the noise is Markovian.” “In the limit of large N.” These are not vague hedges; they are precise statements about the domain of validity of the results. It requires logical derivation over rhetorical persuasion: the connectives are “since,” “therefore,” “it follows that” — explicit logical operators, not narrative bridges. And the passive construction of “the signal was measured” rather than “I measured the signal” removes the individual from the result, because the result should be reproducible regardless of who performs the measurement.

The outcome is prose that is systematic, consistent, and structurally predictable. From the outside — and from the vantage point of an AI detector — it looks machine-generated.

Paul Dirac is the physicist who comes to mind first here. His 1928 paper deriving the relativistic wave equation for the electron contains almost no rhetorical apparatus. Motivation, equation, consequence: each stated once, clearly, with no warm-up and no elaboration beyond what the argument requires. It is not warm. It is not discursive. It is beautiful in the way that a proof is beautiful: every element earns its place. Run it through GPTZero and see what you get.

This connection between the physicist’s prose style and the autistic cognitive mode is not accidental. Baron-Cohen et al. [6] surveyed Cambridge students by academic discipline and found that physical scientists and mathematicians scored consistently higher on the Autism Quotient (AQ) than humanities students and controls, with mathematicians scoring highest of all. The systemising orientation associated with autism spectrum conditions is also associated with — and presumably selected for — in quantitative scientific disciplines. The physicist’s prose reflects this. So does the writing of a high-systemising person who has never studied physics.

The categories overlap without being identical. What they share is a cognitive preference for explicit structure, consistent vocabulary, and logical transparency over social performance and rhetorical persuasion. The writing that emerges from that preference looks, to an AI detector, like it was generated by a machine.

It was not.

The Category Error

The error AI detectors make is not a measurement error. It is a category error.

They are trained to distinguish two things: output generated by a contemporary LLM, and a specific subset of human writing — typically casual, personal, or student prose collected from online sources. When they encounter text outside either of those training categories — systematic and precise but human-generated — the classifier has no good option. The text does not match the “AI” training data exactly, and it does not match the “human” baseline either. It gets assigned to the bin it fits least badly.

What is happening when the Constitution is flagged: it is systematic, definitional, prescriptive, and internally consistent. It was written by lawyers and statesmen who understood that ambiguity in foundational documents creates legal chaos. They wrote to be unambiguous. The result is text with low perplexity and low burstiness — the statistical signature the detector associates with AI.

GPTZero’s creator Edward Tian acknowledged this problem when it was reported: the Constitution appears so frequently in LLM training data that it registers as “already known” to the model, which artificially lowers its perplexity score. That is a real and specific issue. But it is secondary. The deeper issue is that the Constitution would score low-perplexity even without the training-data contamination effect, because systematic, definitional prose is intrinsically low-perplexity. Precise language is predictable language. That is partly the point of precise language.

The baseline assumption — that human writing is variable and idiosyncratic — holds for much human writing. It does not hold for legal drafting, technical documentation, scientific papers, sacred and historical texts written to be durable and precise, writing by people with high systemising orientation, or writing by non-native speakers at formal registers. That is not a small population of edge cases. It is a substantial fraction of all written material that exists.

Right and Wrong at the Same Time

So: if you think these posts are AI-generated, you are right and wrong at the same time.

Right, in two ways. First: yes, I use AI tools. LLM assistance is part of my writing process — not an occasional aid, but a regular part of how research notes and half-formed arguments become structured posts. Second: the writing style of these posts is systematic and precise in ways that detectors register as machine-generated. That systematicity is real, and if a detector picks it up, it is measuring something.

Wrong, also in two ways. First: the ideas, judgments, and connections in these posts are mine. The decisions about what to include and what to leave out, which papers to cite and how to frame their implications, where the interesting tension lies between neurodiversity research and the assumptions baked into AI detection tools — those are not outputs of a language model working in isolation. They are the product of someone who works at the intersection of these fields and has thought about them for a while. An LLM cannot generate these posts without a human who has already decided what to say.

Second, and more important for the argument here: the systematic, precise character of this writing is not evidence of machine generation. It is a cognitive signature — one associated with physics training, with high systemising orientation, with the overlap between those two things that I have written about elsewhere in the context of neurodiversity more broadly.

The detector is measuring a real property of the text. It is misattributing the origin of that property.

The interesting question this opens is not “did AI write this?” That question is increasingly poorly posed in an era where thinking and writing are already deeply entangled with machine assistance, in ways that differ sharply from person to person and task to task. The better question is: whose judgment is in the text? Whose choices about what to include, what to connect, what to leave out?

The systematicity in this writing is mine. The recognition that AI detectors systematically disadvantage autistic writers, physicist writers, and non-native speakers is a judgment I made, not one a language model was prompted to produce. The connection to the Constitution — a document written to be maximally unambiguous, flagged as maximally AI-like — is a connection I found worth drawing.

Whether that makes this text “human” is a philosophical question I am happy to leave open. What it is not is AI hallucination.

References

[1] Mitchell, E., Lee, Y., Khazatsky, A., Manning, C. D., & Finn, C. (2023). DetectGPT: Zero-shot machine-generated text detection using probability curvature. Proceedings of the 40th International Conference on Machine Learning (ICML 2023). https://arxiv.org/abs/2301.11305

[2] Gehrmann, S., Strobelt, H., & Rush, A. M. (2019). GLTR: Statistical detection and visualization of generated text. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, 111–116. https://doi.org/10.18653/v1/P19-3019

[3] Baron-Cohen, S. (2009). Autism: The empathising–systemising (E-S) theory. Annals of the New York Academy of Sciences, 1156(1), 68–80. https://doi.org/10.1111/j.1749-6632.2009.04467.x

[4] Markram, K., & Markram, H. (2010). The intense world theory — a unifying theory of the neurobiology of autism. Frontiers in Human Neuroscience, 4, 224. https://doi.org/10.3389/fnhum.2010.00224

[5] Liang, W., Yuksekgonul, M., Mao, Y., Wu, E., & Zou, J. (2023). GPT detectors are biased against non-native English writers. Patterns, 4(7), 100779. https://doi.org/10.1016/j.patter.2023.100779

[6] Baron-Cohen, S., Wheelwright, S., Skinner, R., Martin, J., & Clubley, E. (2001). The autism-spectrum quotient (AQ): Evidence from Asperger syndrome/high-functioning autism, males and females, scientists and mathematicians. Journal of Autism and Developmental Disorders, 31(1), 5–17. https://doi.org/10.1023/A:1005653411471

The Video

Same question as last month’s: “Should I drive to the car wash?” New video, new AI, new wrong answer. This time the assistant replied that walking was the better option — better for health, better for the environment, and the car wash was only fifteen minutes away on foot.

Accurate, probably. Correct, arguably. Useful? No.

The model did not fail because of tokenisation. It did not fail because it lacked access to the current weather. It failed because it read the wrong question. The user was asking “is now a good time to have my car washed?” The model answered “what is the most sustainable way for a human to travel to the location of a car wash?”

These are different questions. The model chose the second one. This is a pragmatic inference failure, and it is the most instructive of the three failure modes in this series — because the model was not, by any obvious measure, working incorrectly. It was working exactly as designed, on the wrong problem.

What the Question Actually Meant

“Should I drive to the car wash?” is not about how to travel. The word “drive” here is not a transportation verb; it is part of the idiomatic compound “drive to the car wash,” which means “take my car to get washed.” The presupposition of the question is that the speaker owns a car, the car needs or might benefit from washing, and the speaker is deciding whether the current moment is a good one to go. Nobody asking this question wants to know whether cycling is a viable alternative.

Linguists distinguish between what a sentence says — its literal semantic content — and what it implicates — the meaning a speaker intends and a listener is expected to infer. Paul Grice formalised this in 1975 with a set of conversational maxims describing how speakers cooperate to communicate:

• Quantity: say as much as is needed, no more
• Quality: say only what you believe to be true
• Relation: be relevant
• Manner: be clear and orderly

The maxims are not rules; they are defaults. When a speaker says “should I drive to the car wash?”, a cooperative listener applies the maxim of Relation to infer that the question is about car maintenance and current conditions, not about personal transport choices. The “drive” is incidental to the real question, the way “I ran to the store” does not invite commentary on jogging technique.

The model violated Relation — in the pragmatic sense. Its answer was technically relevant to one reading of the sentence, and irrelevant to the only reading a cooperative human would produce.

A Taxonomy of the Three Failures

It is worth being precise now that we have three examples:

Strawberry (tokenisation failure): The information needed to answer was present in the input string but lost in the model’s representation. “Strawberry” →

— the character “r” in “straw” is not directly accessible. The model understood the task correctly; the representation could not support it.

Car wash, rainy day (grounding failure): The model understood the question. The information needed to answer correctly — current weather — was never in the input. The model answered by averaging over all plausible contexts, producing a sensible-on-average response that was wrong for this specific context.

Car wash, walk (pragmatic inference failure): The model had all the relevant words. It may have had access to the weather, the location, the car state. It chose the wrong interpretation of what was being asked. The sentence was read at the level of semantic content rather than communicative intent.

Formally: let $\mathcal{I}$ be the set of plausible interpretations of an utterance $u$. The intended interpretation $i^*$ is the one a cooperative, contextually informed listener would assign. A well-functioning pragmatic reasoner computes:

The model appears to have assigned high probability to the transportation-choice interpretation $i_{\text{walk}}$, apparently on the surface pattern: “should I

to

?” generates responses about modes of transport. It is a natural pattern-match. It is the wrong one.

Why This Failure Mode Is More Elusive

The tokenisation failure has a clean diagnosis: look at the BPE splits, find where the character information was lost. The grounding failure has a clean diagnosis: identify the context variable $C$ the answer depends on, check whether the model has access to it.

The pragmatic failure is harder to pin down because the model’s answer was not, in isolation, wrong. Walking is healthy. Walking to a car wash that is fifteen minutes away is plausible. If you strip the question of its conversational context — a person standing next to their dirty car, wondering whether to bother — the model’s response is coherent.

The error lives in the gap between what the sentence says and what the speaker meant, and that gap is only visible if you know what the speaker meant. In a training corpus, this kind of error is largely invisible: there is no ground truth annotation that marks a technically-responsive answer as pragmatically wrong.

This is a version of a known problem in computational linguistics: models trained on text predict text, and text does not contain speaker intent. A model can learn that “should I drive to X?” correlates with responses about travel options, because that correlation is present in the data. What it cannot easily learn from text alone is the meta-level principle: this question is about the destination’s purpose, not the journey.

The Gricean Model Did Not Solve This

It is tempting to think that if you could build in Grice’s maxims explicitly — as constraints on response generation — you would prevent this class of failure. Generate only responses that are relevant to the speaker’s probable intent, not just to the sentence’s semantic content.

This does not obviously work, for a simple reason: the maxims require a model of the speaker’s intent, which is exactly what is missing. You need to know what the speaker intends to know which response is relevant; you need to know which response is relevant to determine the speaker’s intent. The inference has to bootstrap from somewhere.

Human pragmatic inference works because we come to a conversation with an enormous amount of background knowledge about what people typically want when they ask particular kinds of questions, combined with contextual cues (tone, setting, previous exchanges) that narrow the interpretation space. A person asking “should I drive to the car wash?” while standing next to a mud-spattered car in a conversation about weekend plans is not asking for a health lecture. The context is sufficient to fix the interpretation.

Language models receive text. The contextual cues that would fix the interpretation for a human — the mud on the car, the tone of the question, the setting — are not available unless someone has typed them out. The model is not in the conversation; it is receiving a transcript of it, from which the speaker’s intent has to be inferred indirectly.

Where This Leaves the Series

Three videos, three failure modes, three diagnoses. None of them are about the model being unintelligent in any useful sense of the word. Each of them is a precise consequence of how these systems work:

1. Models process tokens, not characters. Character-level structure can be lost at the representation layer.
2. Models are trained on static corpora and have no real-time connection to the world. Context-dependent questions are answered by marginalising over all plausible contexts, which is wrong when the actual context matters.
3. Models learn correlations between sentence surface forms and response types. The correlation between “should I \[travel verb\]to \[place\]?” and transport-related responses is real in the training data. It is the wrong correlation for this question.

The useful frame, in all three cases, is not “the model failed” but “what, precisely, does the model lack that would be required to succeed?” The answers point in different directions: better tokenisation; real-time world access and calibrated uncertainty; richer models of speaker intent and conversational context. The first is an engineering problem. The second is partially solvable with tools and still hard. The third is unsolved.

References

• Grice, P. H. (1975). Logic and conversation. In P. Cole & J. Morgan (Eds.), Syntax and Semantics, Vol. 3: Speech Acts (pp. 41–58). Academic Press.

• Levinson, S. C. (1983). Pragmatics. Cambridge University Press.

The Video

Someone asked their car’s built-in AI assistant: “Should I drive to the car wash today?” It was raining. The assistant said yes, enthusiastically, with reasons: regular washing extends the life of the paintwork, removes road salt, and so on. Technically correct statements, all of them. Completely beside the point.

The clip spread. The reactions were the usual split: one camp said this proves AI is useless, the other said it proves people expect too much from AI. Both camps are arguing about the wrong thing.

The interesting question is: why did the model fail here, and is this the same kind of failure as the strawberry problem?

It is not. The failures look similar from the outside — confident wrong answer, context apparently ignored — but the underlying causes are different, and the difference matters if you want to understand what these systems can and cannot do.

The Strawberry Problem Was About Representation

In the strawberry case, the model failed because of the gap between its input representation (BPE tokens: “straw” + “berry”) and the task (count the character “r”). The character information was not accessible in the model’s representational units. The model understood the task correctly — “count the r’s” is unambiguous — but the input structure did not support executing it.

That is a representation failure. The information needed to answer correctly was present in the original string but was lost in the tokenisation step.

The car wash case is different. The model received a perfectly well-formed question and had no representation problem at all. “Should I drive to the car wash today?” is tokenised without any information loss. The model understood it. The failure is that the correct answer depends on information that was never in the input in the first place.

The Missing Context

What would you need to answer “should I drive to the car wash today?” correctly?

• The current weather (is it raining now?)
• The weather forecast for the rest of the day (will it rain later?)
• The current state of the car (how dirty is it?)
• Possibly: how recently was it last washed, what kind of dirt (road salt after winter, tree pollen in spring), whether there is a time constraint

None of this is in the question. A human asking the question has access to some of it through direct perception (look out the window) and some through memory (I just drove through mud). A language model has access to none of it.

Let $X$ denote the question and $C$ denote this context — the current state of the world that the question is implicitly about. The correct answer $A$ is a function of both:

The model has $X$. It does not have $C$. What it produces is something like an expectation over possible contexts, marginalising out the unknown $C$:

Averaged over all plausible contexts in which someone might ask this question, “going to the car wash” is probably a fine idea — most of the time when people ask, it is not raining and the car is dirty. $\hat{A}$ is therefore approximately “yes.” The model returns “yes.” In this particular instance, where $C$ happens to include “it is currently raining,” $\hat{A} \neq f(X, C)$.

The quantity that measures how much the missing context matters is the mutual information between the answer and the context, given the question:

Here $H(A \mid X)$ is the residual uncertainty in the answer given only the question, and $H(A \mid X, C)$ is the residual uncertainty once the context is also known. For most questions in a language model’s training distribution — “what is the capital of France?”, “how do I sort a list in Python?” — this mutual information is near zero: the context does not change the answer. For situationally grounded questions like the car wash question, it is large: the answer is almost entirely determined by the context, not the question.

Why the Model Was Confident Anyway

This is the part that produces the most indignation in the viral clips: not just that the model was wrong, but that it was confident about being wrong. It did not say “I don’t know what the current weather is.” It said “yes, here are five reasons you should go.”

Two things are happening here.

Training distribution bias. Most questions in the training data that resemble “should I do X?” have answers that can be derived from general knowledge, not from real-time world state. “Should I use a VPN on public WiFi?” “Should I stretch before running?” “Should I buy a house or rent?” All of these have defensible answers that do not depend on the current weather. The model learned that this question form typically maps to answers of the form “here are some considerations.” It applies that pattern here.

No explicit uncertainty signal. The model was not trained to say “I cannot answer this because I lack context C.” It was trained to produce helpful-sounding responses. A response that acknowledges missing information requires the model to have a model of its own knowledge state — to know what it does not know. This is harder than it sounds. The model has to recognise that $I(A; C \mid X)$ is high for this question class, which requires meta-level reasoning about information structure that is not automatically present.

This is sometimes called calibration: the alignment between expressed confidence and actual accuracy. A well-calibrated model that is 80% confident in an answer is right about 80% of the time. A model that is confident about answers it cannot possibly know from its training data is miscalibrated. The car wash video is a calibration failure as much as a grounding failure.

What Grounding Means

The term grounding in AI has a precise origin. Harnad (1990) used it to describe the problem of connecting symbol systems to the things they refer to — how does the word “apple” connect to actual apples, rather than just to other symbols? A symbol system that only connects symbols to other symbols (dictionary definitions, synonym relations) has the form of meaning without the substance.

Applied to language models: the model has rich internal representations of concepts like “rain,” “car wash,” “dirty car,” and their relationships. But those representations are grounded in text about those things, not in the things themselves. The model knows what rain is. It does not know whether it is raining right now, because “right now” is not a location in the training data.

This is not a solvable problem by making the model bigger or training it on more text. More text does not give the model access to the current state of the world. It is a structural feature of how these systems work: they are trained on a static corpus and queried at inference time, with no automatic connection to the world state at the moment of the query.

What Tool Use Gets You (and What It Doesn’t)

The standard engineering response to grounding problems is tool use: give the model access to a weather API, a calendar, a search engine. Now when asked “should I go to the car wash today?” the model can query the weather service, get the current conditions, and factor that into the answer.

This is genuinely useful. The model with a weather tool call will answer this question correctly in most circumstances. But tool use solves the problem only if two conditions hold:

1. The model knows it needs the tool. It must recognise that this question has $I(A; C \mid X) > 0$ for context $C$ that a weather tool can provide, and that it is missing that context. This requires the meta-level awareness described above. Models trained on tool use learn to invoke tools for recognised categories of question; for novel question types, or questions that superficially resemble answerable ones, the tool call may not be triggered.

2. The right tool exists and returns clean data. Weather APIs exist. “How dirty is my car?” does not have an API. “Am I the kind of person who cares about car cleanliness enough that this matters?” has no API. Some missing context can be retrieved; some is inherently private to the person asking.

The deeper issue is not tool availability but knowing what you don’t know. A model that does not recognise its own information gaps cannot reliably decide when to use a tool, ask a clarifying question, or express uncertainty. This is a hard problem — arguably harder than making the model more capable at the tasks it already handles.

The Contrast, Stated Plainly

The strawberry failure and the car wash failure look alike from the outside — confident wrong answer — but they are different enough that conflating them produces confused diagnosis and confused solutions.

Strawberry: the model has the information (the string “strawberry”), but the representation (BPE tokens) does not preserve character-level structure. The fix is architectural or procedural: character-level tokenisation, chain-of-thought letter spelling.

Car wash: the model does not have the information (current weather, car state). No fix to the model’s architecture or prompt engineering gives it information it was never given. The fix is exogenous: provide the context explicitly, or give the model a tool that can retrieve it, or design the system so that context-dependent questions are routed to systems that have access to the relevant state.

A model that confidently answers the car wash question without access to current conditions is not failing at language understanding. It is behaving exactly as its training shaped it to behave, given its lack of situational grounding. Knowing which kind of failure you are looking at is most of the work in figuring out what to do about it.

The grounding problem connects to the broader question of what it means for a language model to “know” something — which comes up in a different form in the context window post, where the issue is not missing context but irrelevant context drowning out the relevant signal.

A second car wash video a few weeks later produced a third, different failure: Car Wash, Part Three: The AI Said Walk — the model had the right world state but chose the wrong interpretation of the question.

References

• Harnad, S. (1990). The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(1–3), 335–346. https://doi.org/10.1016/0167-2789(90)90087-6

• Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On calibration of modern neural networks. ICML 2017. https://arxiv.org/abs/1706.04599

December 2025

It happened fast. In the twelve months before I am writing this, agentic coding went from a niche research topic to the default mode for several categories of software engineering task. Codex runs code in a sandboxed container and submits pull requests. Claude Code works through a task list in your terminal while you make coffee. Cursor’s agent mode rewrites a file, runs the tests, reads the failures, and tries again — automatically, without waiting for you to press a button.

The demos are impressive. The production reality is messier.

The problem is not that these systems do not work. They work well enough, often enough, to be genuinely useful. The problem is that “works” means something different when an agent is executing than when a human is. A human who makes a mistake can tell you what they were thinking. An agent that produces a subtly wrong result leaves you with a diff and no explanation. And an agent run that worked last Tuesday might not work today, because the model changed, or the context window filled differently, or the prompt-to-output mapping is, at bottom, a stochastic function.

This is the problem the Ralph Loop is designed to address: not “make agents more capable” but “make agent runs reproducible.”

The Reproducibility Problem, Formally

An LLM tool call is a stochastic function. Given a prompt $p$, the model samples from a distribution over possible outputs:

where $\mathcal{P}$ is the space of prompts, $\mathcal{O}$ is the space of outputs, and $\Delta(\mathcal{O})$ denotes the probability simplex over $\mathcal{O}$.

At temperature zero — the most deterministic setting most systems support — this collapses toward a point mass:

where $o^*$ is the argmax token sequence. “Approximately” because hardware non-determinism, batching effects, and floating-point accumulation mean that even $T_0$ is not strictly reproducible across runs, environments, or model versions.

A naive agentic loop composes these calls. If an agent takes $k$ sequential tool calls to complete a task, the result is a $k$-fold composition:

The variance does not merely add — it propagates through the dependencies. Early outputs condition later prompts; a small deviation at step 2 can shift the trajectory of step 5 substantially. This is not a theoretical concern. It is the practical experience of anyone who has tried to reproduce a multi-step agent run.

The Ralph Loop does not solve the stochasticity of $T$. What it does is prevent the composition.

The Ralph Loop as a State Machine

The system’s state at any point in a run is a triple:

where:

• $Q = (s_1, s_2, \ldots, s_n)$ is the ordered story queue — the PRD (product requirements document) — with stories sorted by priority, then by ID
• $S \in \lbrace \texttt{open}, \texttt{passing}, \texttt{skipped} \rbrace^n$ is the status vector, one entry per story
• $L \in \lbrace \texttt{free}, \texttt{held} \rbrace$ is the file-lock state protecting $S$ from concurrent writes

The transition function $\delta$ at each step is:

1. Select: $i^* = \min\lbrace i : S[i] = \texttt{open} \rbrace$ — deterministic by construction, since $Q$ has a fixed ordering
2. Build: $p = \pi(s_{i^*},\; \text{CODEX.md})$ — a pure function of the story definition and the static policy document; no dependency on previous tool outputs
3. Execute: $o \sim T(p)$ — exactly one tool call, output captured
4. Accept: $\alpha(o) \in \lbrace \top, \bot \rbrace$ — parse the acceptance criterion (was the expected report file created at the expected path?)
5. Commit: if $\alpha(o) = \top$, set $S[i^*] \leftarrow \texttt{passing}$; otherwise increment the attempt counter; write atomically under lock $L$

The next state is $\sigma' = (Q, S', L)$ where $S'$ differs from $S$ in exactly one position. The loop continues until no open stories remain or a story limit $N$ is reached.

Termination. Since $|Q| = n$ is finite, $S$ has at most $n$ open entries, and each step either closes one entry or increments an attempt counter bounded by $A_{\max}$, the loop terminates in at most $n \cdot A_{\max}$ steps. Under the assumption that $T$ eventually satisfies any reachable acceptance criterion — which is what CODEX.md’s constraints are designed to encourage — the loop converges in exactly $n$ successful transitions.

Replay. The entire trajectory $\sigma_0 \to \sigma_1 \to \cdots \to \sigma_k$ is determined by $Q$ and the sequence of tool outputs $o_1, o_2, \ldots, o_k$. The .runtime/events.log records these outputs. If tool outputs are deterministic, the run is fully deterministic. If they are not — as in practice they will not be — the stochasticity is at least isolated to individual steps rather than allowed to compound across the chain.

The One-Tool-Call Invariant

The most important constraint in the Ralph Loop is also the simplest: exactly one tool call per story attempt.

This is not the natural design. A natural agentic loop would let the model plan, execute, observe, reflect, and re-execute within a single story. Some frameworks call this “inner monologue” or “chain-of-thought with tool use.” The model emits reasoning tokens, calls a tool, reads the result, emits more reasoning, calls another tool, and eventually produces the final output.

This is more capable for complex tasks. It is also what makes reproducibility hard. Each additional tool call in the chain is a fresh draw from $T$, conditioned on the previous outputs. After five tool calls, the prompt for the fifth includes four previous outputs — each of which varied slightly from the last run. The fifth output is now conditioned on a different input.

Formally: let the multi-call policy use $k$ sequential calls per story. Each call $c_j$ produces output $o_j \sim T(p_j)$, where $p_j = f(o_1, \ldots, o_{j-1}, s_{i^*})$ for some conditioning function $f$. The variance of the final output $o_k$ depends on the accumulated conditioning:

$$\text{Var}(o_k) ;=; \text{Var}_{o_1}!\left[, \mathbb{E}[o_k \mid o_1] ,\right]

• \mathbb{E}_{o_1}!\left[, \text{Var}(o_k \mid o_1) ,\right]$$

By the law of total variance, applied recursively, the total variance decomposes into explained and residual components — conditioning redistributes variance but does not eliminate the residual term. In a well-designed, low-variance chain the residual may stay small; in practice, LLM outputs have non-trivial variance at each step, and that variance propagates through the conditioning chain.

The one-call constraint collapses $k$ to 1:

The output depends only on the story definition and the static policy document. Not on previous tool outputs. The stories are designed to be atomic enough that one call is sufficient. If a story requires more, it should be split into two stories in the PRD. This is a forcing function toward better task decomposition, which I consider a feature rather than a limitation.

Scope as a Topological Constraint

In fixing mode, each story carries a scope[] field listing the files or directories the agent is permitted to modify. The runner captures a snapshot of the repository state before execution:

where $h(f)$ is a hash of the file contents. After the tool call:

The diff $\Delta = F_{\text{after}} \setminus F_{\text{before}}$ must satisfy:

This is a locality constraint on the filesystem graph: the agent’s writes are confined to the neighbourhood $\mathcal{N}(s_{i^*})$ defined by the story’s scope declaration. Writes that escape this neighbourhood are a story failure, regardless of whether they look correct.

The motivation is containment. When a fixing agent makes a “small repair” to one file but also helpfully tidies up three adjacent files it noticed while reading, you have three undocumented changes outside the story’s intent. In a system with many stories running sequentially, out-of-scope changes accumulate silently. The scope constraint prevents this. Crucially, prompt instructions alone are not sufficient — an agent told “only modify files in scope” can still modify out-of-scope files if the instructions are interpreted loosely or the context is long. The runner enforces scope at the file system level, after the fact, and that enforcement cannot be argued with.

Acceptance Criteria: Grounding Evaluation in Filesystem Events

Each story’s acceptance criterion is a single line of the form Created <path> — the path where the report or output file should appear.

This is intentionally minimal. The alternative — semantic acceptance criteria (“did the agent identify all relevant security issues?”) — would require another model call to evaluate, reintroducing stochasticity at the evaluation layer and creating the infinite regress of “who checks the checker.” A created file at the right path is a necessary condition for a valid run. It is not a sufficient condition for correctness, but necessary conditions that can be checked deterministically are already more than most agentic pipelines provide.

The quality of the outputs — whether the audit findings are accurate, whether the fix is correct — depends on the model and the prompt quality. The Ralph Loop gives you a framework for running agents safely and repeatably. Verifying that the agent was right is a different problem and, arguably, a harder one.

Why Bash

A question I have fielded: why Bash and jq, not Python or Node.js?

The practical reason: the target environment is an agent sandbox that has reliable POSIX tooling but variable package availability. Python dependency management inside a constrained container is itself a source of variance. Bash with jq has no dependencies beyond what any standard Unix environment provides.

The philosophical reason: the framework’s job is orchestration, not computation. It selects stories, builds prompts from templates, calls one external tool, parses one file path, and updates one JSON field. None of this requires a type system or a rich standard library. Bash is the right tool for glue that does not need to be impressive.

The one place Bash becomes awkward is the schema validation layer, which is implemented with a separate jq script against a JSON Schema. This works but is not elegant. If the PRD schema grows substantially, that component would be worth replacing with something that has native schema validation support.

What This Is Not

The Ralph Loop is not an agent. It is a harness for agents. It does not decide what tasks to run, does not reason about a codebase, and does not write code. It sequences discrete, pre-specified stories, enforces the constraints on each execution, and records the outcomes. The intelligence is in the model and in the story design; the framework contributes only discipline.

This distinction matters because the current wave of agentic tools conflates two things that are worth keeping separate: the capability to reason and act (what the model provides) and the infrastructure for doing so safely and repeatably (what the harness provides). Improving the model does not automatically improve the harness — and a better model in a poorly constrained harness just fails more impressively.

The repository is at github.com/sebastianspicker/ralph-loop. The Bash implementation, the PRD schema, the CODEX.md policy document, and the test suite are all there.

The Problem

Every PDF acquisition pipeline eventually produces the same chaos. Journal articles downloaded from publisher sites arrive as 513194-008.pdf or 1-s2.0-S0360131520302700-main.pdf. Scanned letters from the tax authority arrive as scan0023.pdf. Invoices arrive as Rechnung.pdf — every invoice from every vendor, overwriting each other if you are not paying attention. The actual content is in the file. The filename tells you nothing.

The human solution is trivial: open the PDF, glance at the title or date or sender, type a descriptive name. Thirty seconds per file, multiplied by several hundred files accumulated over a year, becomes a task that perpetually does not get done.

The automated solution sounds equally trivial: read the text, decide what the document is, generate a filename. What could be involved?

Quite a bit, it turns out. Working through the implementation is a useful way to make concrete some things about LLMs and text processing that are easy to understand in the abstract but clearer with a specific task in front of you.

Step One: Getting Text Out of a PDF

A PDF is not a text file. It is a binary format designed for page layout and print fidelity — it encodes character positions, fonts, and rendering instructions, not a linear stream of prose. The text in a PDF has to be extracted by a parser that reassembles it from the position data.

For PDFs with embedded text (most modern documents), this works well enough. For scanned PDFs — images of pages, with no embedded text at all — you need OCR as a fallback. The pipeline handles both: native extraction first, OCR if the text yield is below a useful threshold.

The result is a string. Already there are failure modes: two-column layouts produce interleaved text if the parser reads left-to-right across both columns simultaneously; footnotes appear in the middle of sentences; tables produce gibberish unless the parser handles them specifically. These are not catastrophic — for renaming purposes, the first paragraph and the document header are usually enough, and those are less likely to be badly formatted than the body. But they are real, and they mean that the text passed to the next stage is not always clean.

Step Two: The Token Budget

Once you have a string representing the document’s text, you cannot simply pass all of it to a language model. Two reasons: context windows have hard limits, and — even when they are large enough — filling them with the full text of a thirty-page document is wasteful for a task that only needs the title, date, and category.

Language models do not process characters. They process tokens — subword units produced by the same BPE compression scheme I described in the strawberry post. A rough practical rule for English text is:

This is an approximation — technical text, non-English content, and code tokenise differently — but it is useful for budgeting. A ten-page academic paper might contain around 30,000 characters, which is approximately 7,500 tokens. The context window of a small local model (the default here is qwen2.5:3b via Ollama) is typically in the range of 8,000–32,000 tokens, depending on the version and configuration. You have room — but not unlimited room, and the LLM also needs space for the prompt itself and the response.

The tool defaults to 28,000 tokens of extracted text (DEFAULT_MAX_CONTENT_TOKENS), leaving comfortable headroom for the prompt and response in most configurations. For documents that exceed this, the extraction is truncated — typically to the first N characters, on the reasonable assumption that titles, dates, and document types appear early.

This truncation is a design decision, not a limitation to be apologised for. For the renaming task, the first two pages of a document contain everything the filename needs. A strategy that extracts the first page plus the last page (which often has a date, a signature, or a reference number) would work for some document types. The current implementation keeps it simple: take the front, stay within budget.

Step Three: Heuristics First

Here is something that improves almost any LLM pipeline for structured extraction tasks: do as much work as possible with deterministic rules before touching the model.

The AI PDF Renamer applies a scoring pass over the extracted text before deciding whether to call the LLM at all. The heuristics are regex-based rules that look for patterns likely to appear in specific document types:

• Date patterns: \d{4}-\d{2}-\d{2}, \d{2}\.\d{2}\.\d{4}, and a dozen variants
• Document type markers: “Rechnung”, “Invoice”, “Beleg”, “Gutschrift”, “Receipt”
• Author/institution lines near the document header
• Keywords from a configurable list associated with specific categories

Each rule that fires contributes a score to a candidate metadata record. If the heuristic pass produces a confident result — date found, category identified, a couple of distinguishing keywords present — the LLM call is skipped entirely. The file gets renamed from the heuristic output.

This matters for a few reasons. Heuristics are fast (microseconds vs. seconds for an LLM call), deterministic (the same input always produces the same output), and do not require a running model. For a batch of two hundred invoices from the same vendor, the heuristic pass will handle most of them without any LLM involvement.

The LLM is enrichment for the hard cases: documents with unusual formats, mixed-language content, documents where the type is not obvious from surface features. In practice this is probably 20–40% of a typical mixed-document folder.

Step Four: What to Ask the LLM, and How

When a heuristic pass does not produce a confident result, the pipeline builds a prompt from the extracted text and sends it to the local endpoint. What the prompt asks for matters enormously.

The naive approach: “Please rename this PDF. Here is the content: [text].” The response will be a sentence. Maybe several sentences. It will not be parseable as a filename without further processing, and that further processing is itself an LLM call or a fragile regex.

The better approach: ask for structured output. The prompt in llm_prompts.py requests a JSON object conforming to a schema — something like:

{
  "date": "YYYYMMDD or null",
  "category": "one of: invoice, paper, letter, contract, ...",
  "keywords": ["max 3 short keywords"],
  "summary": "max 5 words"
}

The model returns JSON. The response parser in llm_parsing.py validates it against the schema, catches malformed responses, applies fallbacks for null fields, and sanitises the individual fields before they are assembled into a filename.

This works because JSON is well-represented in LLM training data — models have seen vastly more JSON than they have seen arbitrary prose instructions to parse. A model told to return a specific JSON structure will do so reliably for most inputs. The failure rate (malformed JSON, missing fields, hallucinated values) is low enough to be handled by the fallback logic.

What counts as a hallucinated value in this context? Dates in the future. Categories not in the allowed set. Keywords that are not present in the source text. The llm_schema.py validation layer catches the obvious cases; for subtler errors (a plausible-sounding date that does not appear in the document), the tool relies on the heuristic pass having already identified any date that can be reliably extracted.

Step Five: The Filename

The output format is YYYYMMDD-category-keywords-summary.pdf. A few design decisions embedded in this:

Date first. Lexicographic sorting of filenames then gives you chronological sorting for free. This is the most useful sort order for most document types — you want to find the most recent invoice, not the alphabetically first one.

Lowercase, hyphens only. No spaces (which require escaping in many contexts), no special characters (which are illegal in some filesystems or require quoting), no uppercase (which creates case-sensitivity issues across platforms). The sanitisation step in filename.py strips or replaces anything that does not conform.

Collision resolution. Two documents with the same date, category, keywords, and summary would produce the same filename. The resolver appends a counter suffix (_01, _02, …) when a target name already exists. This is deterministic — the same set of documents always produces the same filenames, regardless of processing order — which matters for the undo log.

Local-First

The LLM endpoint defaults to http://127.0.0.1:11434/v1/completions — Ollama running locally, no external traffic. This is a deliberate choice for a document management tool. The documents being renamed are likely to include medical records, financial statements, legal correspondence — content that should not be routed through an external API by default.

A small 8B model running locally is sufficient for this task. The extraction problem does not require deep reasoning; it requires pattern recognition over a short text and the ability to return a specific JSON structure. Models at this scale handle it well. The latency is measurable (a few seconds per document on a modern laptop with a reasonably fast inference backend) but acceptable for a batch job running in the background.

For users who want to use a remote API, the endpoint is configurable — the local default is a sensible starting point, not a hard constraint.

What It Cannot Do

Renaming is a classification problem disguised as a text generation problem. The tool works well when documents have standard structure — title on page one, date near the header or footer, document type identifiable from a few keywords. It works less well for documents that are structurally atypical: a hand-written letter scanned at poor resolution, a PDF that is essentially a single large image, a document in a language the model handles badly.

The heuristic fallback means that even when the LLM produces a bad result, the file gets a usable if imperfect name rather than a broken one. And the undo log means that a bad batch run can be reversed. These are not complete solutions to the hard cases, but they are the right design response to a tool that handles real-world document noise.

The harder limit is semantic: the tool can tell you that a document is an invoice and extract its date and vendor name. It cannot tell you whether the invoice has been paid, whether it matches a purchase order, or whether the amount is correct. For those questions, renaming is just the first step in a longer pipeline.

The repository is at github.com/sebastianspicker/AI-PDF-Renamer. The tokenisation background in the extraction and budgeting sections connects to the strawberry tokenisation post and the context window post.

Changelog

• 2026-04-02: Corrected the default model name from qwen3:8b to qwen2.5:3b. The codebase default is qwen2.5:3b (apple-silicon preset) or qwen2.5:7b-instruct (gpu preset).
• 2026-04-02: Corrected DEFAULT_MAX_CONTENT_TOKENS description from “28,000 characters … roughly 7,000 tokens” to “28,000 tokens.” The variable is a token limit, not a character limit.

The Occasion

“Next level?” The question mark is doing real work. The framing HfMT chose for the day was appropriately provisional: not a declaration that AI has already transformed music education, but an invitation to ask whether, in what direction, and at what cost.

The invitations that reach me for events like this tend to come with one of two framings. The first is enthusiasm: AI is coming, we need to get ahead of it, here are tools your students are already using. The second is anxiety: AI is coming, it threatens everything we do, we need to protect students from it. Both framings are understandable. Neither is adequate to the curriculum question, which is slower-moving and more structural than either suggests.

I prepared three sets of handouts. The first covered data protection — the least glamorous topic in AI education, and the one that most directly determines what can legally be deployed in a university setting. The second covered AI tools for students: what exists, what it does, and what critical thinking skills you need to use it without being used by it. The third covered AI for instructors: where it helps, where it flatters, and where it makes things worse.

This post does not recapitulate the handouts. It addresses the question I kept returning to across all three workshops: what does this change about what a music student needs to learn?

What the Technology Actually Is

My physics training left me professionally uncomfortable with hand-waving — including my own. Before discussing curriculum implications, it is worth being specific about what these tools are.

The dominant paradigm in current AI — responsible for ChatGPT, for Whisper, for Suno.AI, for Google Magenta, for the large language models whose outputs are now visible everywhere — is the transformer architecture (Vaswani et al., 2017). A transformer is a neural network that processes sequences by computing, for each element, a weighted attention over all other elements. The attention weights are learned from data. The result is a model that can capture long-range dependencies in sequences — text, audio, musical notes — without the recurrence that made earlier architectures difficult to train at scale.

What this means practically: these models are trained on very large corpora, they learn statistical regularities, and they generate outputs that are statistically consistent with their training distribution. They are not reasoning from first principles. They do not “know” music theory the way a student who has internalised harmonic function knows it. They have learned, from enormous quantities of text and audio, what tends to follow what. For many tasks this is sufficient. For tasks that require understanding of underlying structure, it is not — and the failure modes are characteristic rather than random.

BERT (Devlin et al., 2018) showed that pre-training on large corpora and fine-tuning on specific tasks produces models that outperform task-specific architectures on a wide range of benchmarks. The same transfer-learning paradigm has spread to audio (Whisper pre-trains on 680,000 hours of labelled audio), to music generation (Magenta’s transformer-based models produce melodically coherent sequences), and to multimodal domains. The technology is mature, improving, and available to students now. Knowing what it is — not just what it produces — is the starting point for any sensible curriculum discussion about it.

The Data Protection Constraint

Before any discussion of pedagogical benefit, there is a legal boundary that most AI-in-education discussions skip over. In Germany, and in the EU more broadly, the deployment of AI tools in a university setting is governed by the GDPR (DSGVO, Regulation 2016/679) and, at state level in NRW, by the DSG NRW. The constraints are not abstract: they determine which tools can be used for which purposes with which students.

The core principle is data minimisation: only data necessary for a specific, documented purpose may be collected or processed. When a student uses a commercial AI tool to get feedback on a composition exercise and enters text that could identify them or their institution, that data may be stored, processed, and used for model improvement by an operator whose servers are outside the EU. Whether such transfers remain legally valid under GDPR after the Schrems II ruling (Court of Justice of the EU, 2020) is contested — and “contested” is not a position in which an institution can comfortably require students to use a tool.

The practical upshot for curriculum design is this: AI tools running on EU servers with documented processing agreements can be integrated into formal coursework. Commercial tools whose terms specify US-based processing and model training on user data cannot be required of students. They can be discussed and demonstrated, but making them mandatory puts students in a position where they must choose between their privacy and their grade.

This is not a reason to avoid AI in teaching. It is a reason to be honest about the regulatory landscape, to distinguish clearly between tools you can require and tools you can recommend, and to make data protection literacy part of what students learn. The skill of reading a terms-of-service document and identifying the data flows it describes is not a legal skill — it is a general literacy skill that matters for every digital tool a music professional will use.

What Changes for Students

The question I was asked most often across the three workshops was some version of: “If AI can already do X, should students still learn X?”

The question is less simple than it appears, and the answer is not uniform across skills.

Skills where automation reduces the required production threshold do exist. A student who spends weeks mastering advanced music engraving tools for score production, when AI can generate a usable first draft from a much simpler description, has arguably spent time that could have been better allocated elsewhere. Not because the underlying skill is worthless — it is not — but because the threshold of competence required to produce a working output has dropped. The student’s time might be more valuable spent on something that has not been automated.

Skills where automation creates new requirements are more interesting. Transcription is a useful example. Automatic speech recognition — using models like Whisper for spoken-word transcription, or specialised models for audio-to-score music transcription — is now accurate enough to produce usable first drafts from audio. This does not eliminate the need for transcription skill in a music student. It changes it. A student who cannot evaluate the output of an automatic transcription — who cannot hear where the model has made characteristic errors, who does not have an internalised sense of what a correct transcription looks like — is unable to use the tool productively. The required skill has shifted from production to evaluation. This is not a lesser skill; it is a different one, and it is not automatically acquired alongside the ability to run the tool.

Skills that automation cannot replace are those that depend on embodied, situated, relational knowledge: stage presence, real-time improvisation, the subtle negotiation of musical meaning in ensemble, the pedagogical relationship between teacher and student. These are not beyond AI in principle. They are far beyond it in practice, and the gap is not closing as quickly as the generative AI discourse sometimes suggests.

The curriculum implication is not “teach less” or simply “teach differently.” It is: be explicit about which category each skill falls into, and design assessment accordingly. An assignment that asks students to produce something AI can produce is now testing something different from what it was testing two years ago — not necessarily nothing, but something different. The rubric should reflect that.

What Changes for Instructors

The same three-category analysis applies symmetrically to teaching.

Routine task automation is genuinely useful. Generating first drafts of worksheets, producing exercises at different difficulty levels, transcribing a recorded lesson for later analysis — these are tasks where AI can save meaningful time without compromising the pedagogical judgment required to make use of the output. Holmes et al. (2019) identify feedback generation as one of the clearer wins for AI in education: systems that provide immediate, targeted feedback at a scale that human instructors cannot match. A transcription model listening to a student practice and flagging rhythmic inconsistencies does not replace a teacher. It extends the feedback loop beyond the lesson hour.

Content generation with limits is where AI is most seductive and most dangerous. A model like ChatGPT can produce a reading list on any topic, a summary of any debate in the literature, a set of discussion questions for any text. The outputs are fluent, plausible, and frequently wrong in ways that are difficult to detect without domain expertise. Jobin et al. (2019) and Mittelstadt et al. (2016) both document the broader concern with AI opacity and accountability: when a model produces a confident-sounding claim, the burden of verification falls on the user. An instructor who outsources the construction of course materials to a model, and who lacks enough domain knowledge to catch the errors, is not saving time — they are transferring risk to their students.

Hallucinations — outputs that are plausible in form but false in content — are not bugs in the usual sense. They are a structural consequence of how generative models work. A model trained to predict likely next tokens will produce the most statistically plausible continuation, not the most accurate one. For music education, where historical facts, composer attributions, and music-theoretic claims need to be correct, this matters. The model’s fluency is not evidence of its accuracy.

Personalisation is the most-cited promise of AI in education (Luckin et al., 2016; Roll & Wylie, 2016) and the hardest to evaluate in practice. The argument is that AI can adapt instructional content to individual learners' needs in real time, producing one-to-one tutoring at scale. The evidence in formal educational settings is more mixed than the boosters suggest. What is clear is that personalisation at scale requires data — and extensive data about individual students’ learning trajectories raises the same data protection concerns already discussed, in more acute form.

The Music-Specific Question

I want to be direct about something that came up repeatedly across the day and that the general AI-in-education literature handles badly: music education is not generic.

The skills involved — listening, performing, interpreting, composing, improvising — have a phenomenological and embodied dimension that does not map cleanly onto the text-prediction paradigm that most current AI systems instantiate. Suno.AI can generate a stylistically convincing chord progression in the manner of a named composer. It cannot explain why the progression is convincing in the way a student who has internalised tonal function can explain it. Google Magenta can generate a continuation of a melodic fragment that is locally coherent. It cannot navigate the structural expectations of a sonata form with the intentionality that a performer brings to interpreting one.

This is not a criticism of these tools. It is a description of what they are. The curriculum implication is that music education must be clear about what it is teaching: the product — a score, a performance, a composition — or the process and understanding of which the product is evidence. Where assessment focuses on the product, AI creates an obvious challenge. Where it focuses on demonstrable process and understanding — including the ability to critically evaluate AI-generated outputs — it creates new opportunities.

The more interesting question is whether AI tools can make musical process more visible and discussable. A composition student who uses a generative model, notices that the output is harmonically correct but rhythmically inert, and can articulate why it is inert — and then revise it accordingly — has demonstrated more sophisticated musical understanding than a student who produces the same output without any generative assistance. The tool does not lower the standard; it shifts where the standard is applied.

There is an analogy in music theory pedagogy. The availability of notation software that can play back a student’s harmony exercise and flag parallel fifths changed what ear training and harmony teaching emphasise — but it did not make harmony teaching obsolete. It changed the floor (students can check mechanical correctness automatically) and raised the ceiling (more class time can be spent on voice-leading logic and expressive intention). AI tools are a larger version of the same displacement: the floor rises, the ceiling rises with it, and the pedagogical question is always what you are doing between the two.

Copyright and Academic Integrity

Two issues that crossed all three workshops and deserve direct treatment.

On copyright: the training data of generative music models includes copyrighted recordings and scores, the legal status of which is actively litigated in multiple jurisdictions. When Suno.AI generates a piece “in the style of” a named composer, it is drawing on patterns extracted from that composer’s work — work that is under copyright in the case of living or recently deceased composers. The output is not a direct copy, but neither is the relationship to the training data legally settled. Music students who use these tools in professional contexts should know that they are working in a legally uncertain space, and institutions should not pretend otherwise.

On academic integrity: the issue is not that students might use AI to cheat — they will, some of them, and they have always found ways to cheat with whatever tools were available. The issue is that current AI policies at many institutions are incoherent: prohibiting AI use in assessment while providing no clear guidance on what counts as AI use, and assigning tasks where AI assistance is undetectable and arguably appropriate. The more useful approach is to design tasks where AI assistance is either irrelevant (because the task requires live performance or real-time demonstration) or visible and assessed (because the task explicitly includes reflection on how AI was used and to what effect).

Three Things I Came Away With

After a full day of workshops, discussions, and the conversations that happen in the corridors between sessions, I left with three positions that feel more settled than they did in the morning.

First: the data protection question is not separable from the pedagogical question. Any serious curriculum discussion of AI in music education has to start with what can legally be deployed, not with what would be useful if constraints were not a factor. The constraints are a factor.

Second: the skill most urgently needed — in students and in instructors — is not AI literacy in the sense of knowing which tool to use for which task. It is the critical capacity to evaluate AI-generated outputs: to notice what is wrong, to understand why it is wrong, and to correct it. This requires domain expertise first. You cannot critically evaluate an AI-generated harmonic analysis if you do not understand harmonic analysis. The tools do not lower the bar for domain knowledge. They raise the bar for its critical application.

Third: the curriculum question is not “how do we accommodate AI?” It is “what are we actually trying to teach, and does the answer change when AI can produce the visible output of that process?” Answering that honestly, skill by skill, for a full music programme, is slow work. It cannot be done at a one-day event. But a one-day event, if it is well-designed, can start the conversation in the right place.

HfMT’s Thementag started it in the right place.

References

• Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018). BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. https://arxiv.org/abs/1810.04805

• Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. https://www.deeplearningbook.org

• Holmes, W., Bialik, M., & Fadel, C. (2019). Artificial Intelligence in Education: Promises and Implications for Teaching and Learning. Center for Curriculum Redesign.

• Jobin, A., Ienca, M., & Vayena, E. (2019). The global landscape of AI ethics guidelines. Nature Machine Intelligence, 1, 389–399. https://doi.org/10.1038/s42256-019-0088-2

• LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436–444. https://doi.org/10.1038/nature14539

• Luckin, R., Holmes, W., Griffiths, M., & Forcier, L. B. (2016). Intelligence Unleashed: An Argument for AI in Education. Pearson.

• Mittelstadt, B. D., Allo, P., Taddeo, M., Wachter, S., & Floridi, L. (2016). The ethics of algorithms: Mapping the debate. Big Data & Society, 3(2). https://doi.org/10.1177/2053951716679679

• Roll, I., & Wylie, R. (2016). Evolution and revolution in artificial intelligence in education. International Journal of Artificial Intelligence in Education, 26(2), 582–599. https://doi.org/10.1007/s40593-016-0110-3

• Russell, S., & Norvig, P. (2020). Artificial Intelligence: A Modern Approach (4th ed.). Pearson.

• Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30. https://arxiv.org/abs/1706.03762

In September 2024, OpenAI publicly confirmed that their new reasoning model had been code-named “Strawberry” during development. This landed with a particular thud because “how many r’s are in strawberry?” had, by that point, become one of the canonical demonstrations of language model failure. The model named after strawberry could not count the letters in strawberry. The internet had opinions.

Before the opinions: the answer is three. s-t-r-a-w-b-e-r-r-y. One in the str- cluster, two in the -rry ending. Count carefully and you will find that most people get this right on the first try, and most large language models get it wrong, returning “two” with apparent confidence.

The question worth asking is not “why is the model stupid.” It is not stupid, and “stupid” is not a useful category here. The question is: what does this specific error reveal about the structure of the system?

The answer involves tokenisation, and it is actually interesting.

How You Count Letters (and How the Model Doesn’t)

When you count the r’s in “strawberry,” you do something like this: scan the string left to right, maintain a running count, increment it each time you see the target character. This is a sequential operation over a character array. It requires no semantic knowledge about the word — it does not matter whether “strawberry” is a fruit, a colour, or a nonsense string. The characters are the input; the count is the output.

A language model does not receive a character array. It receives a sequence of tokens — chunks produced by a compression algorithm called Byte Pair Encoding (BPE) that the model was trained with. In the tokeniser used by GPT-class models, “strawberry” is most likely split as:

Three tokens. The model’s input is these three integer IDs, each looked up in an embedding table to produce a vector. There is no character array. There is no letter “r” sitting at a known position. There are three dense vectors representing “str,” “aw,” and “berry.”

What BPE Does (and Doesn’t) Preserve

BPE is a greedy compression algorithm. Starting from individual bytes, it iteratively merges the most frequent pair of adjacent symbols into a single new token:

Applied to a large text corpus until a fixed vocabulary size is reached, this produces a vocabulary of common subwords. Frequent words and common word-parts become single tokens; rare sequences stay as multi-token fragments.

What BPE optimises for is compression efficiency, not character-level transparency. The token “straw” encodes the sequence s-t-r-a-w as a unit, but that character sequence is not explicitly represented anywhere inside the model once the embedding lookup has occurred. The model receives a vector for “straw,” not a list of its constituent letters.

The character composition of a token is only accessible to the model insofar as it was implicitly learned during training — through seeing “straw” appear in contexts where its internal structure was relevant. For most tokens, most of the time, that character structure was not relevant. The model learned what “straw” means, not how to spell it character by character.

Why the Error Is Informative

Most people say the model returns “two r’s,” not “one” or “four” or “none.” This is not random noise. It is a systematic error, and systematic errors are diagnostic.

“berry” contains two r’s: b-e-r-r-y. If you ask most models “how many r’s in berry?” they get it right. The model has seen that question, or questions closely enough related, that the right count is encoded somewhere in the weight structure.

“str” contains one r: s-t-r. But as a token it is a short, common prefix that appears in hundreds of words — string, strong, stream — contexts in which its internal letter structure is rarely attended to. “aw” contains no r’s. When the model answers “two,” it is almost certainly counting the r’s in “berry” correctly and failing to notice the one in “str.” The token boundaries are where the error lives.

This is not stupidity. It is a precise failure mode that follows directly from the tokenisation structure. You can predict where the error will occur by looking at the token split.

Chain of Thought Partially Fixes This (and Why)

If you prompt the model to “spell out the letters first, then count,” the error rate drops substantially. The reason is not mysterious: forcing the model to generate a character-by-character expansion — s, t, r, a, w, b, e, r, r, y — puts the individual characters into the context window as separate tokens. Now the model is not working from “straw” and “berry”; it is working from ten single-character tokens, and counting sequential characters in a flat list is a task the model handles much better.

This is, in effect, making the model do manually what a human does automatically: convert the compressed token representation back to an enumerable character sequence before counting. The cognitive work is the same; the scaffolding just has to be explicit.

The Right Frame

The “how many r’s” test is sometimes cited as evidence that language models don’t “really” understand text, or that they are sophisticated autocomplete engines with no genuine knowledge. These framing choices produce more heat than light.

The more precise statement is this: language models were trained to predict likely next tokens in large text corpora. That training objective produces a system that is very good at certain tasks (semantic inference, translation, summarisation, code generation) and systematically bad at others (character counting, exact arithmetic, precise spatial reasoning). The system is not doing what you are doing when you read a sentence. It is doing something different, which happens to produce similar outputs for a very wide range of inputs — and different outputs for a class of inputs where the character-level structure matters.

“Strawberry” sits squarely in that class. The model is not failing to read the word. It is succeeding at predicting what a plausible-sounding answer looks like, based on a compressed representation that does not preserve the information needed to get the count right. Those are not the same thing, and the distinction is worth keeping clear.

The tokenisation argument here is a simplified version. Real BPE vocabularies, positional encodings, and the specific way character information is or isn’t preserved in embedding tables are more complicated than this post suggests. But the core point — that the model’s input representation is not a character array and never was — holds.

A follow-up post covers a structurally different failure mode: Should I Drive to the Car Wash? — where the model understood the question perfectly but lacked access to the world state the question was about.

References

• Gage, P. (1994). A new algorithm for data compression. The C Users Journal, 12(2), 23–38.

• Sennrich, R., Haddow, B., & Birch, A. (2016). Neural machine translation of rare words with subword units. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016), 1715–1725. https://arxiv.org/abs/1508.07909

Changelog

• 2025-12-01: Corrected the tokenisation of “strawberry” from two tokens (straw|berry) to three tokens (str|aw|berry), matching the actual cl100k_base tokeniser used by GPT-4. The directional argument (token boundaries obscure character-level information) is unchanged; the specific analysis was updated accordingly.