What is prefix caching in LLM inference?

Prefix caching stores KV blocks produced while processing a prompt prefix, then reuses those blocks when another request starts with the same token sequence and compatible runtime settings. The engine can skip prefill for the matched prefix and continue computation from the first unmatched block.

Why does prompt order affect prefix cache hits?

Prefix caching matches from the left. If stable content comes first, many requests can share the same initial blocks. If the user query, timestamp or request-specific metadata comes before the system prompt or tool schema, the first tokens differ and the reusable prefix disappears.

What must be identical for cached KV blocks to be safe?

The token prefix is not enough. The model, tokenizer behaviour, position encoding convention, adapters, attention mask, template version and other prefix-affecting execution settings must match. If any of these change, reusing stored KV would produce numerically wrong results, so the cache must miss.

The KV Cache

03.08 · Walkthrough

Prefix Caching

Implement prefix caching: cache processed KV blocks and reuse them when a new request shares the same prefix. Explain the design rule that follows --- stable content first, variable content last. A learner who puts the user query before the system prompt has destroyed the cache without knowing it.

Prefix caching reuses previously computed KV pages when a new request begins with the same tokens under the same execution conditions. It saves prefill work only for a shared left prefix, so prompt layout matters: put stable system text, tools and examples first, then append retrieved content and the user query.

What this lesson answers

how does prefix caching work for KV cache
why put system prompt before user query
when does prefix caching fail in LLM serving

Notes

Prefix caching is reuse of already computed key/value tensors for an identical token prefix: if request $r_{2}$ has tokens $x_{1 : m}$ and an earlier request $r_{1}$ computed KV for the same $x_{1 : m}$ under the same model weights, tokenizer, RoPE position convention, LoRA/adapters, sampling-independent prefill path, and attention mask, the engine skips prefill for those $m$ tokens and begins at position $m + 1$ .

References

Common questions

What is prefix caching in LLM inference?: Prefix caching stores KV blocks produced while processing a prompt prefix, then reuses those blocks when another request starts with the same token sequence and compatible runtime settings. The engine can skip prefill for the matched prefix and continue computation from the first unmatched block.
Why does prompt order affect prefix cache hits?: Prefix caching matches from the left. If stable content comes first, many requests can share the same initial blocks. If the user query, timestamp or request-specific metadata comes before the system prompt or tool schema, the first tokens differ and the reusable prefix disappears.
What must be identical for cached KV blocks to be safe?: The token prefix is not enough. The model, tokenizer behaviour, position encoding convention, adapters, attention mask, template version and other prefix-affecting execution settings must match. If any of these change, reusing stored KV would produce numerically wrong results, so the cache must miss.

Short definition: what is Prefix Caching?

03.07 · The KV CachePaged Attention 03.09 · The KV CacheRadixAttention