When a Larger Window Is Not Better Use

There is a popular intuition in LLM engineering that context is a resource to spend freely: more background, more history, more examples, inevitably better answers. Experiments with distractors, long-document retrieval, and positional bias show that this is not a safe default. Irrelevant context can reduce accuracy on some tasks, and advertised window length is not the same as effective use of every token. This post argues for focused context as an engineering heuristic: supply the evidence needed for the current decision, while preserving a route back to details that retrieval or summarisation might miss.


What the Evidence Shows

The “more is more” assumption has an obvious origin. Transformers condition on sequences, and product specifications now advertise very long input windows. That specification is a capacity limit, not a guarantee of constant accuracy across positions, tasks, or context lengths. RULER measured the gap in 2024: of 17 models claiming 32K to a million tokens, 16 fell below the authors' threshold, Llama 2 7B’s 85.6% at 4K, before their claimed length; Gemini 1.5 Pro still passed at 128K, the longest length tested [[5]].

Two further lines of work, one empirical and one theoretical, complicate this story.

The lost-in-the-middle problem. Liu et al. [[1]] showed that performance on multi-document question answering can degrade when the relevant passage appears in the middle of a long context, compared to the beginning or end. With 20 retrieved documents, GPT-3.5-Turbo was correct 75.8% of the time with the answer in the first document, 53.8% with it in the tenth and 63.2% in the last (Appendix G of the arXiv version). The middle score is below its closed-book accuracy, 56.1% with no documents at all, even though the information was present; the same holds for its 16K-token variant with 30 documents. This is a result for the paper’s models and tasks, not a universal law. Later work finds some newer models more robust while still detecting position and distance biases [[4]].

The complexity argument. Standard scaled dot-product attention [[2]] is

\[ \text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{QK^{\top}}{\sqrt{d_k}}\right) V \tag{1} \]

Each row of \( Q \) is the query vector of a position that attends, the rows of \( K \) and \( V \) are the key and value vectors of the positions it attends to, and \( d_k \) is the query and key dimension. The softmax turns each query’s row of attention logits, the entries of \( QK^{\top}/\sqrt{d_k} \), into weights that sum to one; the output is the weighted average of the value vectors.

For full self-attention over an input, the \( QK^{\top} \) product has \( O(n^2) \) time and attention-matrix memory in sequence length \( n \). During autoregressive generation, a KV cache avoids recomputing earlier keys and values; cache memory still grows linearly, and each new query attends over the cached positions. These compute facts do not themselves prove an accuracy loss: attention is weighted, not a uniform average.


A Useful Heuristic

Relevance density, not Shannon SNR

Let \( \mathcal{S} \) be a task-specific set of relevant token positions. A simple descriptive quantity is their density in the supplied context:

\[ \rho = \frac{|\mathcal{S}|}{n} \]

where \( n \) is total context length. If \( |\mathcal{S}| \) stays fixed while irrelevant material is added, \( \rho \) decreases. This is not Shannon signal-to-noise ratio, does not assign information content to tokens, and does not predict model accuracy. It is bookkeeping for the retrieval burden.

The empirical question is whether a particular model suppresses the added material and retrieves the needed evidence on a particular task.

A toy model: the gap must grow like ln n

How much weight does an added key take? The softmax in Eq. (1) depends only on logit differences. Take the toy case of one relevant key and \( n - 1 \) identical distractors, each a logit gap \( \Delta \) below it. The weight on the relevant key is

\[ w(n, \Delta) = \frac{e^{\Delta}}{e^{\Delta} + n - 1}. \tag{2} \]

On a logarithmic \( n \) axis every gap gives the same S-shaped curve, shifted along by the factor \( e^{\Delta} \) (Fig. 1). An earlier version of this post said that doubling the context roughly halves each relevant weight unless the model suppresses the other positions to near zero; I removed it in July. Eq. (2) says when each case holds. While \( n - 1 \ll e^{\Delta} \), extra distractors cost little: at \( \Delta = 8 \), going from 100 to 200 keys lowers \( w \) from 0.97 to 0.94. Once \( n - 1 \gg e^{\Delta} \), \( w \approx e^{\Delta}/n \) and doubling does halve it: from 100,000 to 200,000 keys, 0.029 becomes 0.015. Holding \( w \) fixed requires

\[ \Delta = \ln(n - 1) + \ln\frac{w}{1 - w}, \tag{3} \]

a gap that grows like \( \ln n \): for half the weight, about 6.9 at a thousand keys and 13.8 at a million.

Figure 1. At a fixed gap, the relevant key keeps its weight until the distractors outnumber \( e^{\Delta} \), then its weight falls as \( 1/n \). Curves: the toy model, Eq. (2), for \( \Delta = 4 \), 8 and 12. Dots: half weight, at \( n = e^{\Delta} + 1 \); each step of 4 in \( \Delta \) multiplies the half-weight \( n \) by \( e^{4} \approx 55 \). Read the other way, the gap needed for half weight grows like \( \ln n \), as Eq. (3) states.

Chiang and Cholak met the same denominator in a one-layer transformer that checks whether a bit string starts with a 1. It is guaranteed correct only while its logit gap exceeds \( \ln(n - 1) \), so a gap learned on short strings can fail on long ones; they proposed multiplying attention logits by \( \log n \) [[6]].

The toy stops well short of a trained model. Real distractors do not share one logit, logits can sharpen with length, and many heads and layers can carry the evidence. Nor is dilution the only cost of length: Du et al. found accuracy falling with input length even when the added tokens were masked out of attention [[7]].

Position bias compounds the problem

Liu et al. found a U-shaped performance curve [[1]]: relevant material near the start or end is used more reliably than material in the middle. The toy model has no position term, so it does not account for this bias.

The original mixture-of-priors story was not fitted to any model and has been removed. Attention weights, positional representation, training distribution, and task structure can all contribute; the cited behavioural results do not identify one universal cause.

What focused context looks like in practice

The aim is not minimum context at any price. It is to control what enters the working context while retaining recoverability and provenance. Three practical patterns, each with a failure mode:

  1. Retrieval before bulk inclusion. Retrieve candidate passages at query time and attach source identifiers. This can keep \( n \) smaller, but a retrieval miss removes evidence the generator cannot recover.

  2. Rolling summarisation with access to the record. Compress history for routine use, but preserve the raw transcript outside the active window so a disputed detail can be retrieved. A summary is lossy and may erase the very condition that matters later.

  3. Phased orchestration. Give each phase a task contract, selected prior outputs, and retrieval over earlier evidence. Passing only the immediately preceding output is too brittle: an early omission would become permanent.


Agent Context Is a Control Problem

The argument applies to agentic systems with particular force. A single-shot prompt has a fixed, author-controlled context. An agent can accumulate tool outputs, prior messages, and retrieved documents across a long trajectory. That raises cost and may reduce performance, but the effect must be measured for the model, task, ordering, and compression policy in use.

A larger window can be useful when the task genuinely needs distant evidence. The design problem is deciding what stays active, what is summarised, what is retrievable, and how omissions are detected.

Limitations. Some irrelevant tokens actively distract, as one irrelevant sentence in a grade-school maths problem does [[3]]; others are benign fillers. Full, sparse, and linear-attention systems have different compute profiles. “Relevant” depends on the task and may only become clear after later evidence arrives. Focused context can therefore fail by omission just as bulk context can fail by distraction.

What would test the engineering claim. Fix a task and model version; vary length, distractor type, and evidence position independently; compare full context, retrieval, and summary policies; measure accuracy, omissions, cost, and latency. Repeat across models rather than inferring a universal context law from one benchmark.

The phased orchestration pattern under “What focused context looks like in practice” is not just theoretical hand-waving — I have been building a concrete implementation of it. The current state lives at sebastianspicker/phased-agent-orchestration. It is rough, but the core idea is there: each agent phase gets a bounded, purpose-built context rather than the full accumulated history. Feedback very welcome.

Literature and calculations checked through 2026-10-25.

Follow-ups: Token Saving as Noise Reduction carries this argument into agent tooling, and Token Saving, Revisited tests it against newer evidence, including the length-alone result above.


References

[1] Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024). Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12, 157–173. https://doi.org/10.1162/tacl_a_00638. Accuracies by position are tabulated (Appendix G, Tables 6–7) only in the arXiv version; the journal version plots them: arXiv:2307.03172v3, https://arxiv.org/abs/2307.03172v3

[2] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30. https://arxiv.org/abs/1706.03762

[3] Shi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E., Schärli, N., & Zhou, D. (2023). Large language models can be easily distracted by irrelevant context. Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202, 31210–31227. https://proceedings.mlr.press/v202/shi23a.html

[4] Tian, R., Li, Y., Fu, Y., Deng, S., Luo, Q., Qian, C., Wang, S., Cong, X., Zhang, Z., Wu, Y., Lin, Y., Wang, H., & Liu, X. (2025). Distance between relevant information pieces causes bias in long-context LLMs. Findings of ACL 2025, 521–533. https://doi.org/10.18653/v1/2025.findings-acl.28

[5] Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., & Ginsburg, B. (2024). RULER: What’s the real context size of your long-context language models? First Conference on Language Modeling (COLM 2024). https://arxiv.org/abs/2404.06654

[6] Chiang, D., & Cholak, P. (2022). Overcoming a theoretical limitation of self-attention. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 7654–7664. https://doi.org/10.18653/v1/2022.acl-long.527

[7] Du, Y., Tian, M., Ronanki, S., Rongali, S., Bodapati, S. B., Galstyan, A., Wells, A., Schwartz, R., Huerta, E. A., & Peng, H. (2025). Context length alone hurts LLM performance despite perfect retrieval. Findings of the Association for Computational Linguistics: EMNLP 2025, 23281–23298. https://doi.org/10.18653/v1/2025.findings-emnlp.1264


Changelog

  • 2026-07-11: Replaced the non-Shannon “SNR” equation with a descriptive relevance-density measure; removed the incorrect claim that doubling context halves relevant attention weights; corrected full-attention/KV-cache scaling; made lost-in-the-middle claims model- and task-specific; and added omission risks to retrieval, summarisation, and phased orchestration.

  • 2026-10-03: Restated the lost-in-the-middle figure as the paper reports it (GPT-3.5-Turbo drops by more than 20% when the relevant document moves to the middle, in the worst case below the 56.1% closed-book accuracy), not as a drop relative to the single-document condition; fixed the closing note’s pointer to a nonexistent “Discussion section”.

  • 2026-10-25: Defined the queries, keys, values and key dimension of the attention equation, now Eq. (1). Added a toy model of attention dilution, one relevant key among identical distractors (Eqs. (2)–(3), Fig. 1): the logit gap needed to hold the relevant weight fixed grows like the logarithm of the number of keys, and doubling the context halves the weight only once the distractors far outnumber the exponential of the gap. It replaces the bare statement that adding positions need not halve every weight, and says when the hedged halving claim removed on 2026-07-11 holds. Linked the model to Chiang and Cholak (2022), who met the same expression and proposed log-length scaled attention, and bounded it with Du et al. (2025), where length hurt even with the added tokens masked. Replaced “dropped by more than 20% when the relevant document moved from the edges to the middle” with the 20-document numbers for GPT-3.5-Turbo from the appendix of the arXiv version (75.8% with the answer first, 53.8% tenth, 63.2% last; closed-book 56.1%): the drop of more than 20 points is from the first position; from the last it is 9.4 points. Narrowed “with 20 or 30 documents”: the 4K-token model was not run with 30 documents, so that case is its 16K-token variant. Attributed the U-shaped curve, previously “several evaluations” without a source, to Liu et al., and called it a performance curve, not a recall curve. Changed “actively misleading distractors” to distracting ones: Shi et al. add a sentence that does not affect the solution. Added RULER (2024) on claimed against effective context length. Corrected the closing note’s pointer again: phased orchestration is described under “What focused context looks like in practice”, not in a section on agent context; moved the note out of the references and linked the two follow-up posts.