What is context compaction in agent serving?

Context compaction is a serving-time rewrite of an agent prompt to avoid resending the full history. It may summarise old turns, remove low-value content, store observations externally, or extract state. The goal is lower end-to-end inference cost, not just a shorter prompt.

Why can shortening a prompt hurt prefix caching?

Prefix caches depend on the exact token sequence and positions. If you delete or rewrite material near the start or middle of a conversation, every later token moves. The cached KV for that suffix no longer matches, so the server must recompute work it might otherwise have reused.

What is a safer pattern for compacting agent context?

Keep stable material append-only where possible: system prompts, tool schemas, and recent dialogue should remain in fixed positions. Put retrieved evidence or summarised state near the end, and compact cold sections only when the saved prefill work is larger than the cacheable suffix you will lose.

Serving Agents

07.11 · Concept

Context Compaction

Cut the prompt instead of speeding it up, summarising, pruning or retrieving rather than resending. Understand the trap that pays for it: rewriting history shifts every token position after the edit, so a compaction that saves tokens can destroy the prefix cache it was meant to help.

Context compaction reduces serving cost by replacing long agent history with a shorter form, but it can backfire by invalidating prefix-cache reuse. Deleting or rewriting earlier tokens changes later token positions, so the system may prefill a large suffix again even though the visible prompt is shorter.

What this lesson answers

how does context compaction affect prefix cache
when should agents summarise conversation history
why can pruning prompts make inference slower

Notes

Context compaction is a serving-time transformation that replaces a long agent history $x_{1 : n}$ by a shorter surrogate $x ~_{1 : m}$ , $m < n$ , before the next prefill, using summarisation, pruning, retrieval, or state extraction rather than trying to make attention over $n$ tokens faster. The immediate token saving is $Δ T = n - m$ , but the engineering objective is not just fewer prompt tokens; it is lower end-to-end cost after accounting for lost prefix-cache reuse:…

References

Practical online KV cache compaction for LLM agents

Common questions

What is context compaction in agent serving?: Context compaction is a serving-time rewrite of an agent prompt to avoid resending the full history. It may summarise old turns, remove low-value content, store observations externally, or extract state. The goal is lower end-to-end inference cost, not just a shorter prompt.
Why can shortening a prompt hurt prefix caching?: Prefix caches depend on the exact token sequence and positions. If you delete or rewrite material near the start or middle of a conversation, every later token moves. The cached KV for that suffix no longer matches, so the server must recompute work it might otherwise have reused.
What is a safer pattern for compacting agent context?: Keep stable material append-only where possible: system prompts, tool schemas, and recent dialogue should remain in fixed positions. Put retrieved evidence or summarised state near the end, and compact cold sections only when the saved prefill work is larger than the cacheable suffix you will lose.

Short definition: what is Context Compaction?

07.10 · Serving AgentsLong-Context Prefill 07.12 · Serving AgentsAgents per Megawatt