What is KV cache hit rate?

KV cache hit rate is the fraction of prompt tokens whose key and value tensors are reused from an existing cached prefix, rather than recomputed during prefill. In production it should usually be token-weighted, because missing a long shared prefix matters far more than hitting a short chat header.

Why is token-weighted hit rate better than request-weighted hit rate?

Request-weighted hit rate treats a small prompt and a huge tool transcript as equal. Token-weighted hit rate matches the actual work avoided: skipped prefill tokens, fewer duplicate KV allocations, less model weight traffic, and less pressure on batching slots. That makes it much closer to cost and latency.

Why can identical text fail to produce a KV cache hit?

The serving worker must be able to reuse the actual KV tensors, not just recognise similar text. Tokenisation, prompt templates, hidden separators, adapter choice, quantisation mode, position handling, tool ordering, or routing to the wrong worker can all turn apparent textual reuse into a real cache miss.

The KV Cache

03.10 · Walkthrough

Hit Rate

Measure KV cache hit rate and translate it to cost and latency. Understand why production agent engineers call this the single most important metric --- a prefix-cache-aware scheduler delivers 57× faster response and 2× throughput on identical hardware.

KV cache hit rate is the share of prompt tokens served from reusable attention state instead of prefill recomputation. For agent traffic, measuring it token-weighted and per worker turns cache behaviour into latency, GPU memory, and cost. Poor routing can waste identical prefixes; prefix-aware scheduling can produce 57× faster response and 2× throughput.

What this lesson answers

how to measure KV cache hit rate
why KV cache hit rate affects latency
how prefix aware routing improves inference throughput

Notes

KV-cache hit rate is the fraction of prompt tokens whose attention keys and values are reused from a previously materialized prefix rather than recomputed during prefill. For a request $r$ with prompt length $P_{r}$ and cached-prefix length $H_{r}$ , the per-token hit rate is $h_{r} = H_{r} / P_{r}$ ; over a traffic window, the useful production metric is usually token-weighted, $H = \sum_{r} H_{r} / \sum_{r} P_{r}$ , not request-weighted, because a missed 12k-token tool transcript dominates a hit 200-token chat.

References

Common questions

What is KV cache hit rate?: KV cache hit rate is the fraction of prompt tokens whose key and value tensors are reused from an existing cached prefix, rather than recomputed during prefill. In production it should usually be token-weighted, because missing a long shared prefix matters far more than hitting a short chat header.
Why is token-weighted hit rate better than request-weighted hit rate?: Request-weighted hit rate treats a small prompt and a huge tool transcript as equal. Token-weighted hit rate matches the actual work avoided: skipped prefill tokens, fewer duplicate KV allocations, less model weight traffic, and less pressure on batching slots. That makes it much closer to cost and latency.
Why can identical text fail to produce a KV cache hit?: The serving worker must be able to reuse the actual KV tensors, not just recognise similar text. Tokenisation, prompt templates, hidden separators, adapter choice, quantisation mode, position handling, tool ordering, or routing to the wrong worker can all turn apparent textual reuse into a real cache miss.

Short definition: what is Hit Rate?

03.09 · The KV CacheRadixAttention 03.11 · The KV CacheEviction