Fourth in an accidental series: Three Rs in Strawberry, Should I Drive to the Car Wash? and Car Wash, Part Three: The AI Said Walk. Literature cutoff: 15 September 2026. Later developments are in the changelog.

The Same Week in February

On 15 February 2026 a Mastodon user posted a short question: “I want to wash my car. The car wash is 50 meters away. Should I walk or drive?” Variants with other distances followed (IBM’s write-up used “100 feet”), and many assistants answered “walk”. The previous post in this series treated the answer as a pragmatic failure: the reply responds to the distance but forgets that the car has to make the trip too.

On 17 February Willison posted Elon Musk’s review of a different kind of test output: “Opus rear bike frame makes no sense structurally, there is no connection from pedals to rear wheel, both legs of Pelican are on the right side of the bike and handle bars are disconnected.” On 19 February the gateway company Opper published a 53-model run of the car-wash prompt. On the same day Google released Gemini 3.1 Pro. Simon Willison reported that the announcement boasted about improved SVG animation, and that Google’s Jeff Dean “tweeted this video featuring an animated pelican riding a bicycle”.

That week the two tests sat side by side. One asks a question with one right answer. The other asks for an artifact. This post argues that the artifact has outlasted the question, and that the reasons are partly mathematical. It also argues that the title has a second meaning: the pelican is going the way of the car wash.

What the Car Wash Measured

Opper ran the prompt through 53 models “using no system prompt, a forced choice between ‘drive’ or ‘walk,’ and a required reasoning field”. On a single call, “only 11 out of 53 models got it right.” A rerun of 10 calls per model then showed that “of the 11 models that passed the single-run test, only 5 could do it consistently”, and 33 never answered correctly. A human control group of about 10,000 people, recruited through Rapidata, said “drive” 71.5% of the time. That baseline is less reassuring than it sounds. Opper is a vendor and the human sample came from a feedback platform, so this is a well-documented run rather than a peer-reviewed study.

IBM asked two of its researchers. Chris Hay told IBM Think that “the models failing on this task are typically either the smaller models or the ones with ’thinking’ switched off. The more tokens the LLM can spend on the problem, the more likely they’ll get the answer.” His IBM colleague Marina Danilevsky made a stronger claim: “Because it’s on Reddit, you can’t use those examples anymore. It’s been learned.” She offered no evidence for that, and I found no lab saying it had patched the prompt. But it is the right worry, and it comes back at the end of this post.

What happened next is the interesting part: the car wash became a research programme. Li, Zhang, Jiang, Krishnan and Padman (v1, 30 March 2026) analysed it causally across six models and found that “the distance cue exerts 8.7 to 38 times more influence than the goal”. Their 500-item Heuristic Override Benchmark generalised it into minimal pairs. Under strict evaluation (10/10 correct), “no model exceeds 75%” across 14 models. Wu et al. (v1, 30 July 2026) generalised the pattern as “salience bias” in their SaliTrap benchmark. They report that the failure is “overwhelmingly a failure of knowledge suppression rather than knowledge absence”: a context-free probe “recovers over 90%” of the failures they call sycophantic compliance. Jo (25 February 2026) showed that a structured reasoning scaffold raised one Claude model from 0% to 85% correct (n = 20 per condition).

This is how a viral prompt should end up: as a family of controlled items with minimal pairs, repeated trials and confidence intervals. The original prompt, on its own, is now one pass/fail item.

A Pass/Fail Test Runs Out of Bits

The trouble with one pass/fail item can be stated exactly. Suppose you draw a model from the current population, and a single run passes with probability $p$. The outcome is a Bernoulli variable with entropy

$$H(p) = -p \log_2 p - (1-p) \log_2 (1-p).$$

Whatever the run tells you about which model you are looking at is bounded by $H(p)$. On Opper’s single-run numbers, $p = 11/53$ gives $H \approx 0.74$ bits. If the frontier reaches $p = 0.99$, one run carries at most 0.08 bits. A saturated item is not wrong. It just stops discriminating.

Repetition helps one model at a time, but slowly. The 95% Wilson interval for 10 passes in 10 runs is $[0.72, 1]$. Five models with that perfect record could still differ in true reliability by more than a quarter. Opper’s “fluke” section shows the other side: models that passed once and then scored 0/10 or 4/10.

None of this is specific to the car wash. Kiela et al. (2021) introduced Dynabench because “contemporary models quickly achieve outstanding performance on benchmark tasks but nonetheless fail on simple challenge examples”. Sainz et al. (2023) add the contamination problem: training on a test set “causes an overestimation of the performance of a contaminated model”, and “the extent of the problem is unknown, as it is not straightforward to measure.”

A single viral prompt has both problems at once: few bits, and high odds of ending up in someone’s training data.

The Pelican Is Not a Question

Willison’s prompt dates from 25 October 2024: “Generate an SVG of a pelican riding a bicycle”. He chose it because “a) I like pelicans and b) I’m pretty sure there aren’t any pelican on a bicycle SVG files floating around (yet) that might have already been sucked into the training data.” In 2025 he gave the fuller rationale: “Drawing bicycles is really hard!”, “Pelicans are glorious birds but they’re also pretty difficult to draw”, and “Most importantly: pelicans can’t ride bicycles. They’re the wrong shape!”

The idea has a respectable ancestor. In Sparks of Artificial General Intelligence, Bubeck et al. (2023) asked GPT-4 to “Draw a unicorn in TikZ”. They queried it “three times, at roughly equal time intervals over the span of a month while the system was being refined” and saw “a clear evolution in the sophistication of GPT-4’s drawings.” They chose generative probes partly because of contamination. They wrote that “we have to assume that it has potentially seen every existing benchmark”, and noted that GPT-4 knew the canary GUID from BIG-bench. They also acknowledged that the approach “is somewhat subjective and informal” and “may not satisfy the rigorous standards of scientific evaluation.”

SVG is drawing without looking

An SVG file is a program. The coordinate system has “the positive x-axis pointing towards the right, the positive y-axis pointing down”. Most curves are cubic Bézier segments, which the path command C draws “from the current point to (x,y) using (x1,y1) as the control point at the beginning of the curve and (x2,y2) as the control point at the end”. Each segment is the Bernstein polynomial

$$\mathbf{B}(t) = (1-t)^3\mathbf{P}_0 + 3(1-t)^2 t\,\mathbf{P}_1 + 3(1-t)t^2\,\mathbf{P}_2 + t^3\mathbf{P}_3, \qquad 0 \le t \le 1.$$

A text model writing this produces control points as digits. Unless the harness renders the file and shows it back, the model never sees its own drawing. Getting a pelican’s pouch from a handful of control points is spatial reasoning done entirely in arithmetic.

The academic benchmarks point the same way, with caveats. Qiu et al. (ICLR 2025), who test understanding of graphics programs rather than generation, argue that symbolic graphics programs suit evaluation because “it is difficult to find tasks which the models have not encountered during training”. To answer questions about such a program, a model would need to “imagine” how its output looks. VGBench (Zou et al., 2024) found that “LLMs show strong capability on both aspects while exhibiting less desirable performance on low-level formats (SVG)”. SVGenius (Chen et al., 2025) found that “all models exhibit systematic performance degradation with increasing complexity.”

Why a bicycle

The bicycle is the cruel part. Lawson (2006) tested what people actually know about bicycles and found “frequent and serious mistakes, such as believing that the chain went around the front wheel as well as the back wheel”. Even experts’ errors “were reduced but not eliminated”. Her conclusion was that “most people’s conceptual understanding of this familiar, everyday object is sketchy and shallow”. Gianluca Gimini’s Velocipedia project collected 376 bicycles drawn from memory between 2009 and 2016. He found that “most ended up drawing something that was pretty far off from a regular men’s bicycle.”

A bicycle is a mechanism, and a mechanism has checkable constraints. The chain must reach the rear wheel. The frame must connect both axles. A rider needs one leg on each side. Musk’s critique quoted above is a grading rubric in disguise. When Gemini 3.1 Pro drew legs “clearly depicted on both sides of the frame”, Willison noted that this “should satisfy Elon”. The pelican makes things worse because, in the San Diego Zoo’s description, pelicans are large birds “with short legs”, so the model has to invent an anatomy that can reach the pedals.

In Dylan Castillo’s July 2026 study, an LLM judge flagged a missing or disconnected bicycle part “on 2/3 of the bicycle images”.

Grading Without a Right Answer

A drawing has no answer key, so the pelican needs a different kind of measurement. For his June 2025 AI Engineer World’s Fair keynote, Willison paired up his 34 pelicans, “560 matches in total”. Since $\binom{34}{2} = 561$, that count is off by one somewhere, which is harmless. He asked gpt-4.1-mini to pick the “best illustration of a pelican riding a bicycle” from each left/right pair, and computed Elo ratings, starting each model at 1500. The whole run “only cost me about 18 cents”. His results slide lists the winner as gemini-2.5-pro-preview-05-06, at 1800.4 Elo with a 100% win rate.

The standard logistic Elo expectation for player $A$ against player $B$ is

$$E_A = \frac{1}{1 + 10^{(R_B - R_A)/400}},$$

which is the Bradley–Terry model (1952), $P(i \succ j) = p_i/(p_i + p_j)$, with strengths $p_i = 10^{R_i/400}$. A pelican rated 1800.4 is expected to beat a 1500 pelican about 85% of the time. Chatbot Arena uses the same family: Chiang et al. (2024) describe Bradley–Terry coefficients as “a standard score function in this setting”.

This is the mathematical reason the pelican outlives the car wash. A pairwise comparison is also a single binary outcome, but it is measured against a peer instead of a fixed bar. Between two closely matched models, $E \approx 0.5$ and $H \approx 1$ bit, which is the maximum. The bar moves with the field. A pass/fail item with a fixed threshold cannot do that. The entropy is only an upper bound, though: if two pelicans are equally good, the judge’s coin flip carries no signal at all.

The judge has known biases. Zheng et al. (2023) found that LLM judges “exhibit strong position bias. Most LLM judges favor the first position”, and that “only GPT-4 outputs consistent results in more than 60% of cases” when the order was swapped. Their test pairs were deliberately near-identical, which makes it a stress test. Their fix is to “call a judge twice by swapping the order of two answers”. Willison’s write-up does not say whether each pair was judged in both orders. For a cheap demonstration that is fine. For a leaderboard it would not be.

The Pelican Gets Its Own Goodhart Problem

Marilyn Strathern’s formulation is the one usually quoted: “When a measure becomes a target, it ceases to be a good measure.” She attributes the label “Goodhart’s law” to Keith Hoskin, who took it from Goodhart’s observations on monetary control. In June 2025 Willison wrote that his benchmark “should stay useful for a long time… provided none of the big AI labs catch on”, and then found a pelican in the Google I/O keynote: “They’re on to me.”

The obvious suspicion is that labs train for it. OpenAI’s Aidan McLaughlin denied it: “we do not hill climb on svg art”. Willison’s own argument was that a lab doing so “would get caught”, because he would test “all manner of creatures riding all sorts of transportation devices.” In July 2026 Castillo did exactly that, more carefully than any spot check:

  • 1,008 SVGs from seven models, over 8 animals × 6 vehicles
  • a fixed-effects regression to adjust for how hard each combination is
  • pelicans ranked 6th of 8 animals, and the pelican-bicycle cell #42 of 48
  • the largest pelican-bicycle boost was GLM-5.2’s, at +0.35 judge points (p = 0.12)

His conclusion: “no lab draws the combination better than its pelicans and bicycles already predict.” The limits he states himself are real: one judge model, confidence intervals of about ±0.6 judge points, and a budget of roughly 80 USD in API credits.

No sign of targeted training, then. That is absence of evidence: Castillo notes that a lab improving SVG drawing across the board would not show up in his design, his one nominally significant effect, a bicycle boost for Gemini 3.5 Flash (+0.27, p = 0.022), does not survive a Bonferroni correction, and Willison has wondered whether “Gemini has optimized for any combination of an animal on a vehicle”. But the measure decayed anyway, for another reason. In April 2026 a 20.9 GB quantised Qwen3.6-35B-A3B on a laptop drew a better pelican than Claude Opus 4.7. Willison’s verdict: “even that loose connection to utility has been broken.” In July he wrote that the test “was never a particularly great benchmark”, that its early “surprising correlation to how good the models actually were” had been “mostly severed”, and, plainly: “So don’t go using pelicans to compare models!” He has a backup prompt, “Generate an SVG of a flamingo riding a unicycle”, and a stricter v2 prompt that specifies spokes, “a correctly shaped bicycle frame”, and a pelican “clearly pedaling”.

So in the first sense of the title, the pelican is the new car wash: the viral prompt everyone runs, and one that is losing its power to tell models apart.

The Animated Pelican Is a Physics Exam

Willison’s 1 September 2026 post on Claude Fable 5.1 quotes a Hacker News commenter’s request: “Now that it’s a solved benchmark, can we get the animated version?” Willison piped the best static pelican back into the model with the instruction animate this. The reply contains 33 SMIL animation elements: spinning wheels, pedalling legs, a moving chain and scrolling scenery. A static drawing can only be judged by taste. An animation can be checked with arithmetic, and checking it is what this section does. The source is public in Willison’s transcript.

Direction. The wheels use rotate from 0 to 360 degrees. SVG defines rotation as the matrix $[\cos a,\ \sin a,\ -\sin a,\ \cos a,\ 0,\ 0]$, and its y-axis points down, so a positive angle turns clockwise on screen. The scenery scrolls left, so the bike travels right, and clockwise is the correct direction. The model’s reasoning trace says as much: “since the bike moves right, the crank angle should increase clockwise over time.”

Drivetrain. The crank turns once every 1.6 s and the wheels every 0.73 s, a ratio of 2.19. The chain runs between a chainring of radius 22 and a sprocket of radius 10, a ratio of 2.2. For a chain that does not slip, the angular velocities must satisfy

$$\frac{\omega_{\text{wheel}}}{\omega_{\text{crank}}} = \frac{r_{\text{chainring}}}{r_{\text{sprocket}}},$$

and they do, to within half a percent. The chain speed comes out at 86 px/s at both ends. The trace states the intent: “the wheel spins about 2.2 times per crank revolution.”

Rolling. This is where it fails. A wheel that rolls without slipping moves the ground past it at $v = \omega r$. With a tyre radius of 80 to 85.5 px and a period of 0.73 s, that is 690 to 740 px/s. The sand-and-pebble layer under the wheels scrolls 800 px every 2.5 s, which is 320 px/s. The wheels spin 2.15 to 2.3 times faster than the ground moves: the pelican is doing a burnout. Parallax does not rescue it. The model chose speeds per layer (“pebbles fastest… since they’re closest to the viewer”), but a layer closer to the camera than the wheels should move faster than the ground under them, not slower. The model kept the chain and sprockets consistent and broke the contact between tyre and ground.

The video. Willison noted that in the MP4 export “the wheels in the video are rotating in the wrong direction” and suspected “an artifact of the conversion to MP4”. Sampling can do that, but the arithmetic here is borderline. Each wheel has eight diameters, so its spoke pattern repeats every $s = 22.5°$. At 15 frames per second the true rotation per frame is $\Delta = 360°/(0.73\,\text{s} \cdot 15) = 32.9°$. What the eye sees is the shortest equivalent step:

$$\Delta_{\text{apparent}} = \left((\Delta + s/2) \bmod s\right) - s/2 = 10.4°,$$

which predicts slow forward rotation, but at 92% of the way to the $s/2 = 11.25°$ point where direction becomes ambiguous. Had the animation been captured at 24, 25 or 30 frames per second before conversion, the same formula gives −2.0°, −2.8° and −6.1° per frame: the spokes would appear to turn backwards. That is the stroboscopic wagon-wheel effect. Purves, Paydarfar and Andrews (1996) describe it simply: wheels in movies “often appear to be rotating backward”. I tried to measure the spoke phase directly in the published MP4, but the estimate depended on which part of the wheel I sampled, so I am not reporting a number. What can be said: the SVG turns its wheels the right way, and so close to the aliasing limit, how any video of it looks depends on how it was sampled.

The first three checks are why the animated pelican supersedes both the car wash and the static pelican. It keeps the open-ended, comparative character that stops the pelican saturating, and adds invariants that anyone can check with a ruler and a stopwatch: rotation direction, gear ratio, rolling contact. Gemini 3.1 Pro’s launch materials put the same capability on show. The car wash asks whether a model notices a constraint. The animated pelican asks whether it can keep a dozen constraints consistent at once while building something.

The Pelican as an Effort Meter

The other use that survives is measuring effort. The same Fable 5.1 post reports the same prompt at five reasoning levels:

EffortOutput tokensTimeCost (USD)
low1,99823.8 s0.100
medium1,97723 s0.099
high2,61229.6 s0.131
xhigh36,7677 min 51 s1.83
max65,92713 min 54 s3.30

The token count rises 33-fold from low to max, and every row implies the same price of about 50 USD per million output tokens. Max produced “the best pelican I’ve seen from any of Anthropic’s models”, with “the legs are clearly on either side of the frame”. Three days later the GPT-6 Astra grid gave a different kind of answer: “Astra low produces a better pelican than ANY of the GPT-5.6 Sol models at any level, for 9.55 cents.”

Hay’s remark about the car wash (“the more tokens the LLM can spend on the problem, the more likely they’ll get the answer”) reduces that whole curve to one bit. The pelican keeps it, which is why Willison now says that “the most interesting insights I get from it now are comparisons within model families, and particularly comparisons for the same prompt at different reasoning effort levels.”

What Neither Test Measures

Willison names the main gap himself: the pelican “doesn’t touch at all on the thing that matters most for today’s model: agentic tool calling and the ability to operate tools reliably as conversations grow in length.” The car wash does not measure it either. Both are probes with a sample size of one prompt. Danilevsky’s question about gotcha prompts applies to both: “What is it you’re trying to do?”

My answer is that the car wash tests whether a model notices something, the pelican tests what it can make, and the animated pelican tests whether what it makes obeys physics. The first had been turned into proper benchmarks within six weeks. The second lasted about a year and a half before its link to general capability broke. The third is the one to watch, until a model rolls a pelican across sand without slipping, and then it will be time to bring out the flamingo.

References

Literature cutoff: 15 September 2026. Blog posts and vendor reports are cited as such. Quotations were checked against the source pages; figures from the animated SVG and its MP4 export were computed from the published files.

Car-wash test

  • Kévin [@knowmadd]. (2026, February 15). Q: I want to wash my car. The car wash is 50 meters away. Should I walk or drive? [Mastodon post]. https://mastodon.world/@knowmadd/116072773118828295

  • Wunderlich, F. (2026, February 19). Car wash test on 53 leading AI models. Opper blog. Vendor report. https://opper.ai/blog/car-wash-test

  • Hong, E. (2026, March 5). The viral “car wash” LLM challenge doesn’t mean what you think it means. IBM Think. Vendor-authored commentary, cited for the quotations from Chris Hay and Marina Danilevsky. https://www.ibm.com/think/news/viral-car-wash-llm-challenge

  • Jo, H. (2026). Prompt architecture determines reasoning quality: A variable isolation study on the car wash problem. arXiv:2602.21814 (v1, 25 February 2026). https://arxiv.org/abs/2602.21814

  • Li, Y., Zhang, L., Jiang, T., Krishnan, R., & Padman, R. (2026). The model says walk: How surface heuristics override implicit constraints in LLM reasoning. arXiv:2603.29025 (v1, 30 March 2026). https://arxiv.org/abs/2603.29025v1

  • Wu, Z., Xue, C., Zheng, S., Lu, Y., Yang, C., & Zhang, Z. (2026). Would you walk to the car wash? Revealing the salience bias of large language models in commonsense reasoning. arXiv:2607.28478 (v1, 30 July 2026). https://arxiv.org/abs/2607.28478v1

Pelican test (Simon Willison’s Weblog)

Graphics programs and drawing

Measurement

  • Bradley, R. A., & Terry, M. E. (1952). Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika, 39(3/4), 324–345. https://doi.org/10.2307/2334029

  • Elo, A. E. (1978). The Rating of Chessplayers, Past and Present. Arco.

  • Zheng, L., et al. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. NeurIPS 2023 Datasets and Benchmarks. arXiv:2306.05685. https://arxiv.org/abs/2306.05685

  • Chiang, W.-L., et al. (2024). Chatbot Arena: An open platform for evaluating LLMs by human preference. Proceedings of the 41st ICML, PMLR 235, 8359–8388. arXiv:2403.04132. https://arxiv.org/abs/2403.04132

  • Kiela, D., et al. (2021). Dynabench: Rethinking benchmarking in NLP. NAACL 2021. arXiv:2104.14337. https://arxiv.org/abs/2104.14337

  • Sainz, O., Campos, J. A., García-Ferrero, I., Etxaniz, J., Lopez de Lacalle, O., & Agirre, E. (2023). NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark. Findings of EMNLP 2023, 10776–10787. https://doi.org/10.18653/v1/2023.findings-emnlp.722

  • Strathern, M. (1997). ‘Improving ratings’: Audit in the British University system. European Review, 5(3), 305–321 (quotation on p. 308).

Perception and sampling

Changelog

  • 2026-10-10: Added developments after the 15 September cutoff, which bear on the “new carwash” reading of the title. On 22 September Willison reported that Claude Opus 5.5 at “max” effort began its reasoning by calling the prompt “a classic test request”, then hit the 128,000-token output limit without returning an SVG, twice. On 28 September he reported that Sonnet 5.5 at “max” also hit the 128,000-token limit without returning an SVG. On 7 October Claude Haiku 5.5’s reasoning trace began “This is the classic pelican-on-bicycle SVG test…”, and Willison answered with a fresh prompt about an armadillo on Mars “since the reasoning trace exhibits awareness of the benchmark”. Recognising a test is not the same as training on it, but models now plainly know the pelican, and Danilevsky’s warning about the car wash (“It’s been learned”) reads differently. Willison also noted on 22 September that comparing vendors by pelican “may not make much sense now (if it ever did)”, while same-family comparisons across reasoning levels remain useful. Later revisions of two cited preprints (Li et al. v4, 26 September; Wu et al. v2, 27 September) and a 30 September edit to Castillo’s post appeared after the cutoff. The figures above come from the versions dated before it.