Prefix Caching
Prefix caching is an inference optimisation that reuses the KV cache for an identical prompt prefix instead of recomputing it. When a new request begins with the same tokens under the same execution settings, the server attaches the cached KV blocks and starts prefill after the shared prefix.
The need comes from repeated prompt boilerplate. Production requests often share system instructions, tool schemas, safety policy, examples, or tenant context, then differ only in the user question. Without prefix caching, the model still runs prefill over that stable text for every request. The common misunderstanding is to treat prompt order as cosmetic. It is not: if variable text appears first, the shared left prefix disappears and the cache cannot match.
Mechanically, the engine tokenises the request and compares its leftmost token blocks with previously computed KV blocks. Cache keys include not just token IDs, but also execution details that affect the tensors, such as model, tokenizer behaviour, position encoding settings, adapters, KV format, attention mask, and template version. On a hit, the request’s block table points to existing KV pages; on the first miss, prefill resumes from that boundary and new blocks are inserted.
The trade-off is that reuse is exact and stateful. Semantically equivalent prompts do not match if their tokens differ, so whitespace, JSON field order, timestamps, request IDs, or locale formatting can defeat the cache. The server also pays for hashing, lookup, reference counting, memory residency, and eviction policy. It helps most when long prefixes recur soon enough to stay resident; it can be neutral or harmful for short, rare, or highly fragmented prefixes.
Engineers meet prefix caching in LLM serving systems built on paged KV caches, prefix-aware routers, or radix-style prompt caches. The design rule is simple: put stable content first and variable content last. A robust chat template serialises policy, tools, and examples deterministically, then appends retrieved documents and the user query. For multimodal or adapter-based serving, cache keys must include metadata beyond placeholder token IDs, otherwise reused KV can be wrong.
Common questions
- Is prefix caching the same as the normal KV cache?
- No. The normal KV cache stores keys and values for tokens within one active sequence so decoding does not recompute earlier context. Prefix caching reuses already computed KV blocks across different requests that start with the same tokens and compatible execution settings. It is a sharing layer on top of the ordinary per-sequence KV mechanism.
- Why does prompt ordering matter so much?
- Prefix caching only matches from the beginning of the token sequence. If every request starts with a different user question, the longest shared prefix may be empty, even if the same system prompt appears later. Stable content should lead the prompt, while user-specific content, timestamps, request IDs, and other volatile fields should come after the reusable portion.
- Can it reuse prompts that mean the same thing but are written differently?
- Usually not. Prefix caching is based on tensor correctness, not semantic similarity. Different tokens, different template formatting, changed tool serialisation, different position settings, or an enabled adapter can produce different KV tensors. A safe implementation must miss in those cases unless it can prove the cached tensors were produced under equivalent conditions.
- When does prefix caching not help?
- It depends on prefix length, repetition rate, memory pressure, and implementation overhead. If the shared prefix is short, appears rarely, or is evicted before another request uses it, lookup and cache management may cost more than recomputation. It also helps less when long retrieved documents are unique per request and placed before any shared material.