Context Compaction
Context compaction is a serving-time rewrite that replaces a long agent prompt with a shorter surrogate before the next model call. It cuts prefill work by summarising, pruning, retrieving, or extracting state, but can also break prefix-cache reuse because edited history changes the token positions of everything after it.
Agents tend to resend their whole working history: system instructions, tool schemas, user turns, tool outputs, retrieved documents, errors, and partial plans. Making attention kernels faster helps, but it does not change the fact that the server must prefill a large prompt and write KV for it. Context compaction exists because repeated long histories turn prefill into a major serving cost, especially when much of the old text is no longer needed verbatim.
Mechanically, compaction takes some span of the previous transcript and replaces it with something shorter, or removes it and later retrieves only relevant pieces. A summariser might turn several tool calls into a short state note; a pruner might drop noisy logs; a retrieval system might keep old evidence outside the prompt and append selected passages near the end. The model then receives this rewritten prompt and computes fresh KV for the resulting token sequence.
The trap is that cache keys are not based on meaning alone. Prefix caches reuse KV only when the token IDs and their positions match what was computed before. If you delete or shorten text near the start or middle, every later token moves to a different position, so its cached KV is no longer valid. A smaller prompt can therefore cost more if it destroys a large reusable suffix and forces a new prefill.
The tradeoff depends on where the edit happens, how much text is removed, whether the workload is prefill-bound, and whether exact old wording matters. Summaries can corrupt details; pruning can remove dependencies that only become important later; retrieval can miss relevant context. Compaction is safest for cold, low-value history and riskiest for code, legal text, tool outputs, or hidden state that must be preserved exactly.
Engineers meet context compaction in agent serving stacks that use paged KV caches, prefix caching, radix-style prefix sharing, or KV-aware routing. The practical pattern is to keep system prompts, tool schemas, and recent dialogue stable and append-only, then place volatile retrieved or summarised state late in the prompt. When compaction is needed, treat it as starting a new cache lineage rather than a free prompt cleanup.
Common questions
- Is context compaction just summarisation?
- No. Summarisation is one way to compact, but the serving problem is broader. Pruning, retrieval, state extraction, and prompt layout can all reduce the next prefill. The important engineering question is not whether the text is shorter, but whether the saved tokens outweigh summary cost, quality risk, and lost prefix-cache reuse.
- Why does editing old history break the prefix cache?
- A prefix cache stores KV for a particular token sequence at particular positions. If an early span is shortened or deleted, later tokens may have the same text but different positions. For position-aware attention, that is a different computation, so the server cannot safely reuse the old KV blocks for the shifted suffix.
- When is context compaction a good idea?
- It is usually attractive when the prompt is long, prefill is expensive, the removed material is cold, and little useful cache would have been reused anyway. It is usually poor when most of the prompt is a stable reusable prefix, the edit is early, outputs are short, or the task depends on exact previous wording.
- How should an agent be structured to make compaction safer?
- Keep stable material append-only: system instructions, developer rules, tool schemas, and recent dialogue should not be rewritten casually. Put retrieved documents, summaries, and other volatile state near the end where changes damage less cache. If you must compact older history, do it deliberately and assume subsequent requests may need a fresh prefix.