Status note, 24 October 2026: the tool stack described below is the one I used when I wrote this post in June 2026, and I no longer use it. The follow-up, Token Saving, Revisited, corrects the cost half of the argument and counts what the stack itself added to every request. The argument about relevance does not depend on the stack.

Summary

This is a follow-up to More Context Is Not Always Better. The earlier point was theoretical: long context is not automatically useful context. The practical consequence is sharper. In agentic systems, token saving should not be understood primarily as thrift. It is a form of noise reduction.

Every additional tool output, log fragment, stale plan, old reasoning trace, retrieved document, or configuration dump becomes part of the input from which the model must make its next decision. Some of those tokens are evidence. Many are not. The problem is not that large context windows are useless; the problem is that large context windows make it easy to stop curating context.

The workflow I found most robust in June treats the agent harness as a sequence of filters. Headroom sits at the model/API boundary. RTK reduces shell output. lean-ctx reduces file-reading and search noise. Serena and CodeGraph provide project-local structure instead of undifferentiated text. AGENTS.md, hooks, MCP configuration, and subagent configuration encode routing policy. Codacy, OpenSpec, impeccable, and Ponytail reduce different kinds of decision surface.

That sounds like a tooling inventory. It is not. The unifying idea is that each layer should improve the signal-to-noise ratio of the context before the model is asked to reason over it.

The Mathematical Point

Let the available context be a sequence of chunks or tokens

$$ C = \{t_1, t_2, \ldots, t_n\}, $$

and let \(q\) denote the current task. For each \(t_i\), define a task-dependent relevance score

$$ r_i(q) \in [0,1]. $$

This is not meant as a claim that relevance is easy to measure exactly. It is a useful abstraction. A token that contains a failing assertion, an API contract, or the line currently being edited has high relevance. A token from an unrelated passing test, an old plan, or a verbose progress log has low relevance.

A crude relevance-density analogy is then

$$ R(C,q)= \frac{\sum_{i=1}^{n} r_i(q)} {\epsilon + \sum_{i=1}^{n}(1-r_i(q))}, \tag{1} $$

where \(\epsilon\) avoids division by zero. Write \(\rho = \frac{1}{n}\sum_i r_i(q)\) for the mean relevance per token or chunk, the relevance density in the strict sense. The relevant mass in the numerator is then \(n\rho\), the irrelevant mass in the denominator \(n(1 - \rho)\), and for \(\epsilon = 0\), \(R = \rho/(1 - \rho)\): \(R\) is the odds of the relevance density and rises whenever the density does. This is not Shannon information, an empirical attention law, or a measurable property of a prompt without a specified relevance estimator. It encodes the design preference under discussion: adding material judged irrelevant increases the denominator without increasing the numerator. In a real model, whether that hurts a task must be measured.

The practical objective is not to maximise context length. It is to select a bounded working context \(C'\) that preserves decision-relevant information:

$$ C_B^* = \arg\max_{C' \subseteq C,\ |C'|\le B} \left( \sum_{t_i \in C'} r_i(q) - \lambda |C'| \right), \tag{2} $$

where \(B\) is a token budget and \(\lambda\) is the penalty for carrying additional context. Take the \(t_i\) to be tokens here, so that \(|C'|\) is a token count. Because the objective is a sum of independent terms \(r_i(q) - \lambda\), its maximiser is a threshold rule: keep every token with \(r_i(q) > \lambda\), the most relevant first, until the budget is reached. \(\lambda\) is a relevance threshold, and \(B\) binds only when more than \(B\) tokens clear it. If \(\lambda\) is too small, the system keeps material because it is available. If \(\lambda\) is too large, the system deletes evidence. The engineering task is to make the filtering good enough that the context becomes smaller and more informative at the same time.

The additive form is also the objective’s main simplification. A second copy of the same stack trace scores as high as the first, although it adds nothing; in a real context, what a token is worth depends on what else is already there.

This is why “token saving” is a slightly misleading name. The goal is not merely fewer tokens. The goal is a better working set.

Why This Is Not Just Personal Preference

Several recent preprints treat context as something to be selected, compressed, or compiled, rather than simply enlarged. They support the feasibility of particular compression methods; they do not establish a universal law that shorter prompts are more accurate.

LongAttnComp, for example, frames long-context compression as a token-level selection problem. It introduces token-level chunking, token-budget selection, positional reordering, and a learned cross-attention scoring layer. Its authors report that on InfiniteBench Code-Debug, against full-context baselines of 120k to 152k tokens depending on the target model, a 16k-token compressed context matches or exceeds full-context accuracy for their main target, DeepSeek-R1-0528. On the other three target models, their Stage 1 checkpoint stays 1.8 to 3.0 points below full context (1.5 to 5.8 across all three checkpoints) [[1]]. The Code-Debug result is compatible with the presence of distractors or imperfect use of long context. It does not isolate either mechanism. On LongBench v2, a long-context benchmark in six task categories that the authors use to test multi-document reasoning, every compressed variant in the paper stays below full context.

CompressKV makes the same point inside the KV cache. Like other eviction methods, it keeps only some of the cached key-value pairs. Unlike most of them, it does not score tokens with all attention heads, but only with the heads it identifies as semantic retrieval heads, which attend to the start and end of a prompt and to important evidence in between. The reported results are strong: more than 97 percent of full-cache performance with 3 percent of the KV cache on LongBench question-answering tasks and, for Llama-3.1-8B-Instruct, 90 percent of full-cache Needle-in-a-Haystack accuracy with 0.7 percent of the KV storage [[2]]. The precise numbers may move with model family and benchmark, but the direction is hard to ignore. Contextual information has very uneven value.

Latent Context Compilation approaches the issue differently. Its authors report preserved detail and reasoning at 16x compression in experiments with Llama-3.1-8B [[3]]. That is conceptually close to what good agent memory should do: not replay the entire transcript, but carry forward a smaller representation of what remains useful. The analogy is mine; the paper does not evaluate this agent setup.

Work on adaptive context compression for long-running interactions makes the agent connection explicit. The authors evaluate importance-aware memory selection, coherence-sensitive filtering, and dynamic budget allocation on LOCOMO, LOCCO, and LongBench [[4]]. Its reported gains are method- and benchmark-specific. The engineering analogy is still useful: a system must decide what to retain as history grows.

These papers differ technically. One compresses long prompts, one compresses KV cache, one compiles memory, and one adapts context across long-running interactions. But they share the same structural assumption: not all context deserves equal access to the model’s attention.

The Harness as Applied Context Selection

This is the frame in which my June setup made sense.

Headroom is the system-level boundary for model and agent traffic. It is not just a proxy. It is the place where request and response handling, compression, health checks, and traffic visibility become part of the reasoning environment. If it is healthy, memory-aware, and compressing effectively, it reduces waste before individual tool choices even matter. If behaviour looks wrong, that has to be verified with concrete endpoints such as /health, /readyz, and /stats, not inferred from the absence of visible failure.

RTK is the default shell-output filter. Git, tests, logs, builds, package managers, Docker, and similar commands produce a lot of text whose main value is usually concentrated in a few lines: the failing assertion, the changed file, the non-zero exit, the warning that actually matters. Using RTK by default keeps the shell available while preventing routine commands from flooding the transcript.

lean-ctx belongs primarily to file reading, search, and compact tree views. When it works well, it solves a very common failure mode: reading whole files before knowing which parts matter. A compact ctx_read or ctx_search result is often a better context object than a raw dump of every line. At the same time, this is not ideology. If shell execution through ctx_shell is blocked, then it should not be treated as the primary shell path until that integration is fixed. Compression that hides evidence or blocks execution is not context engineering. It is just failure with fewer tokens.

Serena and CodeGraph are different again. They are useful inside actual code projects because they retrieve structure: symbols, references, call paths, project-local knowledge. They are not replacements for shell routing or log filtering, and they are not especially meaningful for home-directory configuration work unless a project has been activated. Their value is that they answer structural questions without forcing the model to infer project topology from raw text.

AGENTS.md, hooks, MCP configuration, subagent configuration, and token-saving configuration form the policy layer. They encode defaults that otherwise have to be renegotiated in every task: when to use RTK, when to use lean-ctx, when raw shell is justified, when to avoid subagents, when to preserve exact deterministic evidence. Good policy reduces prompt mass because the same routing decisions no longer have to be restated manually.

Codacy, OpenSpec, impeccable, and Ponytail reduce context in a broader sense. Codacy turns some code-quality questions into deterministic findings instead of free-form model judgement. OpenSpec keeps implementation attached to a declared behavioural slice. impeccable narrows UI critique toward visible, user-facing quality rather than vague aesthetic preference. Ponytail asks whether a feature, abstraction, dependency, or explanation needs to exist at all. The best token saving is sometimes deletion before the model ever sees the next problem.

Failure Modes

The obvious failure mode is over-compression. If a filtered command removes the exact compiler error, the failing line, the relevant HTTP status code, or the one stack frame that matters, then the signal-to-noise ratio that counts, the one for the next decision, has not improved. It has worsened, even if the ratio \(R\) defined above goes up.

This gives a practical rule in the notation of Eq. (1):

$$ \text{compress only if } \Delta R > 0. $$

Here \(\Delta R = R' - R\) is the change caused by the compression, with \(R'\) the value of Eq. (1) for the compressed context. Suppose the compression removes a share \(\alpha\) of the relevant mass \(n\rho\) and a share \(\beta\) of the irrelevant mass \(n(1 - \rho)\). The two shares are linked: a removed token of relevance \(r\) takes \(r\) from the relevant mass and \(1 - r\) from the irrelevant mass, so a filter reaches \(\alpha = 0\) only by removing tokens with \(r = 0\). Ignoring \(\epsilon\), Eq. (1) gives

$$ \frac{R'}{R} = \frac{(1-\alpha)\,n\rho}{(1-\beta)\,n(1-\rho)} \cdot \frac{1-\rho}{\rho} = \frac{1-\alpha}{1-\beta}, \tag{3} $$

so \(R\) rises exactly when \(\alpha < \beta\): when the compression removes a smaller share of the relevant mass than of the irrelevant mass. Equivalently, the removed tokens must be less relevant on average than the context they came from. (With \(\epsilon > 0\), the condition tightens slightly, to \(\beta > \alpha\,\bigl(1 + \frac{\epsilon}{n(1 - \rho)}\bigr)\).) Compression is not justified merely because the output is shorter.

The rule has a blind spot, and it is the failure mode this section began with. Eq. (3) depends only on shares, so it cannot see how much evidence was lost. A filter that removes half of the irrelevant mass and nothing relevant doubles \(R\). So does a filter that removes half of the relevant mass and three quarters of the irrelevant mass: \((1 - 0.5)/(1 - 0.75) = 2\) (Fig. 1). The rule accepts both. Only the first is lossless; the second may have deleted the failing assertion.

Figure 1. The rule \(\Delta R > 0\) sees shares, not amounts: a lossless filter and one that deletes half the evidence can raise \(R\) by the same factor. Lines of constant \(R'/R = (1 - \alpha)/(1 - \beta)\) from Eq. (3), with \(\epsilon = 0\), labelled by that factor (2 solid, 10 dashed), against the shares \(\alpha\) of relevant and \(\beta\) of irrelevant mass removed; every such line ends in the corner where everything is removed. Below the diagonal, \(R\) falls. The two points are the filters in the text.

The selection objective, Eq. (2), counts relevant mass in absolute terms: a removal raises it exactly when it drops more than \(1/\lambda\) tokens for each unit of relevance it loses. Removing the two shares from the full context changes the objective by \(n[\lambda\beta(1 - \rho) - (1 - \lambda)\alpha\rho]\). At \(\lambda = \rho\) this is \(n\rho(1 - \rho)(\beta - \alpha)\), positive exactly when \(\alpha < \beta\), so the \(\Delta R\) rule is the objective’s test with the penalty set to the current relevance density. A smaller \(\lambda\) demands more noise removed per unit of evidence lost: the lossless filter passes at any \(\lambda > 0\), the lossy one only if \(\lambda > 2\rho/(3 - \rho)\), about two thirds of the relevance density.

Subagents have the same problem. A subagent is useful when it returns a smaller, sharper evidence object than the work it replaces. It is not useful when it produces another long narrative that must itself be audited. The same applies to summaries. A summary that preserves exact file names, commands, failures, and decisions can be valuable. A summary that says “various changes were made” is not compression. It is information loss.

There is also a privacy boundary. A workflow can describe AGENTS.md, hooks, MCP configuration, subagent configuration, and token-saving setup without publishing raw local configuration. The article-level claim is about routing and filtering, not about exposing secrets, tokens, private endpoints, environment variables, or credentials.

Capacity Is Not Selection

The strongest argument for token saving is not only economic. It is epistemic: when a filter preserves the evidence and removes material irrelevant to the decision, it gives the model a cleaner working set. Whether a particular filter actually does that remains a validation question.

This is the practical continuation of the “more context is not always better” argument. Long context gives capacity. It does not solve selection. In real agent work, selection happens through the harness: proxies, shell filters, file readers, symbol tools, hooks, specs, quality gates, and deliberately minimal scope.

The mature version of context engineering is therefore not a race toward larger windows. It is the design and testing of systems that know what not to include.

Literature cutoff: 2026-07-11; sources and calculations checked through 2026-10-24. All four research references below are arXiv preprints without a journal reference on that date; one, CompressKV, is marked as a workshop paper. Their performance claims are the authors’ reported benchmark results, not independent replications.

Follow-up: Token Saving, Revisited: Right About the Noise, Wrong About the Bill works through what prompt caching and tool-definition overhead do to the cost half of this argument.

References

[1] Ji, M., Raju, R. S., Li, J. L., & Wu, C. (2026). LongAttnComp: Cross-family context compression for long-context reasoning. arXiv:2606.01336v2 [cs.CL], preprint. https://arxiv.org/abs/2606.01336

[2] Lin, X., Wang, J., Kondrateva, O., Shi, Y., Li, B., & Zhang, G. L. (2026). CompressKV: Semantic-retrieval-guided KV-cache compression for resource-efficient long-context LLM inference. arXiv:2606.24467 [cs.AI], preprint; marked as a paper for SuRE'26: Workshop on Sustainability and Resource-Efficiency of Artificial Intelligence, 17 August 2026, Bremen (proceedings not located). https://arxiv.org/abs/2606.24467 (arXiv notes substantial text overlap with the same authors’ arXiv:2508.02401, August 2025.)

[3] Li, Z., Zhou, Y., & Xu, Q. (2026). Latent context compilation: Distilling long context into compact portable memory. arXiv:2602.21221 [cs.LG], preprint. https://arxiv.org/abs/2602.21221

[4] Fofadiya, P., & Tiwari, S. (2026). Developing adaptive context compression techniques for large language models (LLMs) in long-running interactions. arXiv:2603.29193 [cs.CV], preprint. https://arxiv.org/abs/2603.29193

Changelog

  • 2026-07-11: Recast the proposed signal-to-noise quantity as an explicit relevance-density analogy; bounded four preprint results to their evaluated models and benchmarks; and separated the papers’ findings from the article’s agent-workflow recommendations.

  • 2026-10-03: Corrected the reading of the compression rule: a positive change in R means a smaller share of relevant than irrelevant mass is removed (not “more irrelevant than relevant material” in absolute terms), and defined ΔR.

  • 2026-10-10: Added a pointer to the follow-up, which partly retracts this post’s cost framing. The quality argument stands. The implied claim that every removed token saves money in proportion does not hold under prompt caching, and the tool stack described here put its own tool definitions into every request. I no longer use that stack.

  • 2026-10-24: Added a status note: I no longer use the tool stack described here. Corrected LongAttnComp’s Code-Debug result: it matches full context only for DeepSeek-R1-0528, not “across their evaluated models”; added its LongBench v2 shortfall. Corrected CompressKV’s Needle-in-a-Haystack result to 90 percent of full-cache accuracy and added its contrast with all-head eviction. Derived the ΔR rule, R′/R = (1 − α)/(1 − β), with its ε > 0 form, its blind spot for lost evidence (Fig. 1) and its link to the selection objective, whose threshold form and additivity are now stated. Linked the citations; corrected three arXiv categories, pinned LongAttnComp to v2, and noted CompressKV’s earlier version and workshop venue. Renamed “Conclusion” to “Capacity Is Not Selection”.