When a Larger Window Is Not Better Use
There is a popular intuition in LLM engineering that context is a resource to spend freely: more background, more history, more examples, inevitably better answers. Experiments with distractors, long-document retrieval, and positional bias show that this is not a safe default. Irrelevant context can reduce accuracy on some tasks, and advertised window length is not the same as effective use of every token. This post argues for focused context as an engineering heuristic: supply the evidence needed for the current decision, while preserving a route back to details that retrieval or summarisation might miss.
What the Evidence Shows
The “more is more” assumption has an obvious origin. Transformers condition on sequences, and product specifications now advertise very long input windows. That specification is a capacity limit, not a guarantee of constant accuracy across positions, tasks, or context lengths.
Two lines of empirical and theoretical work complicate this story.
The lost-in-the-middle problem. Liu et al. [[1]] showed that performance on multi-document question answering can degrade when the relevant passage appears in the middle of a long context, compared to the beginning or end. Performance on 20-document prompts dropped by more than 20 percentage points relative to the single-document condition in some of their evaluations even though the information was present. This is a result for the paper’s models and tasks, not a universal law. Later work finds some newer models more robust while still detecting position and distance biases [[4]].
The complexity argument. Standard scaled dot-product attention [[2]] is
\[ \text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{QK^{\top}}{\sqrt{d_k}}\right) V \]For full self-attention over an input, the \( QK^{\top} \) product has \( O(n^2) \) time and attention-matrix memory in sequence length \( n \). During autoregressive generation, a KV cache avoids recomputing earlier keys and values; cache memory still grows linearly, and each new query attends over the cached positions. These compute facts do not themselves prove an accuracy loss: attention is weighted, not a uniform average.
A Useful Heuristic
Relevance density, not Shannon SNR
Let \( \mathcal{S} \) be a task-specific set of relevant token positions. A simple descriptive quantity is their density in the supplied context:
\[ \rho = \frac{|\mathcal{S}|}{n} \]where \( n \) is total context length. If \( |\mathcal{S}| \) stays fixed while irrelevant material is added, \( \rho \) decreases. This is not Shannon signal-to-noise ratio, does not assign information content to tokens, and does not predict model accuracy. It is bookkeeping for the retrieval burden.
Softmax attention distributes unit mass across positions, but adding positions does not imply that every old weight is halved: the result depends on all query– key logits. The empirical question is whether a particular model suppresses the added material and retrieves the needed evidence on a particular task.
Position bias compounds the problem
Several evaluations report a U-shaped recall curve: relevant material near the start or end is used more reliably than material in the middle. The magnitude and even shape vary by model and task; 2025 results report improved robustness in newer models alongside remaining biases when relevant pieces are separated across the context [[4]].
The original mixture-of-priors story was not fitted to any model and has been removed. Attention weights, positional representation, training distribution, and task structure can all contribute; the cited behavioural results do not identify one universal cause.
What focused context looks like in practice
The aim is not minimum context at any price. It is to control what enters the working context while retaining recoverability and provenance. Three practical patterns, each with a failure mode:
Retrieval before bulk inclusion. Retrieve candidate passages at query time and attach source identifiers. This can keep \( n \) smaller, but a retrieval miss removes evidence the generator cannot recover.
Rolling summarisation with access to the record. Compress history for routine use, but preserve the raw transcript outside the active window so a disputed detail can be retrieved. A summary is lossy and may erase the very condition that matters later.
Phased orchestration. Give each phase a task contract, selected prior outputs, and retrieval over earlier evidence. Passing only the immediately preceding output is too brittle: an early omission would become permanent.
Agent Context Is a Control Problem
The argument applies to agentic systems with particular force. A single-shot prompt has a fixed, author-controlled context. An agent can accumulate tool outputs, prior messages, and retrieved documents across a long trajectory. That raises cost and may reduce performance, but the effect must be measured for the model, task, ordering, and compression policy in use.
A larger window can be useful when the task genuinely needs distant evidence. The design problem is deciding what stays active, what is summarised, what is retrievable, and how omissions are detected.
Limitations. Some irrelevant tokens are actively misleading distractors [[3]]; others are benign fillers. Full, sparse, and linear-attention systems have different compute profiles. “Relevant” depends on the task and may only become clear after later evidence arrives. Focused context can therefore fail by omission just as bulk context can fail by distraction.
What would test the engineering claim. Fix a task and model version; vary length, distractor type, and evidence position independently; compare full context, retrieval, and summary policies; measure accuracy, omissions, cost, and latency. Repeat across models rather than inferring a universal context law from one benchmark.
References
Literature checked through 2026-07-11.
[1] Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024). Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12, 157–173. https://doi.org/10.1162/tacl_a_00638
[2] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30. https://arxiv.org/abs/1706.03762
[3] Shi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E., Schärli, N., & Zhou, D. (2023). Large language models can be easily distracted by irrelevant context. Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202, 31210–31227. https://proceedings.mlr.press/v202/shi23a.html
[4] Tian, R., et al. (2025). Distance between relevant information pieces causes bias in long-context LLMs. Findings of ACL 2025, 521–533. https://doi.org/10.18653/v1/2025.findings-acl.28
The phased orchestration argument in the Discussion section is not just theoretical hand-waving — I have been building a concrete implementation of it. The current state lives at sebastianspicker/phased-agent-orchestration. It is rough, but the core idea is there: each agent phase gets a bounded, purpose-built context rather than the full accumulated history. Feedback very welcome.
Changelog
- 2026-07-11: Replaced the non-Shannon “SNR” equation with a descriptive relevance-density measure; removed the incorrect claim that doubling context halves relevant attention weights; corrected full-attention/KV-cache scaling; made lost-in-the-middle claims model- and task-specific; and added omission risks to retrieval, summarisation, and phased orchestration.