Is hit rate measured per request or per token?

For production inference, per-token measurement is usually the more honest view. Request-weighted hit rate can make a system look healthy while a few very long prompts miss and consume most of the prefill work. Token-weighted hit rate counts how many prompt tokens actually reused KV entries out of all prompt tokens served.

Why can identical-looking prompts still miss the KV cache?

KV entries are tied to exact token sequences and execution conditions, not just visible text. A different chat template, tokenizer, tool order, adapter, quantisation setup, position convention, or hidden separator can make stored keys and values invalid for the new request. The cache can only reuse prefixes that are semantically and mechanically identical for the model run.

Does a higher hit rate always mean lower latency?

Usually, but not always. It depends on prefix length, lookup overhead, memory pressure, eviction behaviour, and load balancing. Reusing a long stable prefix can remove substantial prefill work. Chasing tiny or rare matches can add routing and cache-management cost, or overload a worker that happens to hold a marginally useful prefix.

How is hit rate different from FlashAttention?

Hit rate is about avoiding prefill work by reusing KV tensors that were already created for a matching prefix. FlashAttention makes attention computation more IO-efficient while the computation is happening. They optimise different parts of the system: one skips repeated prefix computation, the other executes attention more efficiently when computation is still required.

Hit Rate

Hit rate is the share of prompt tokens for which a serving engine reuses already-built KV cache entries instead of running prefill again. In LLM inference it measures realised prefix reuse, not textual similarity, and it directly affects time to first token, memory use, batching pressure, and serving cost.

The need for hit rate appears when many requests contain the same beginning: system prompts, tool definitions, policy text, retrieved documents, or a long conversation trunk. Without reuse, the engine repeatedly computes attention keys and values for identical prompt tokens and stores duplicate KV data. That turns agent traffic into a prefill-heavy workload, where hardware spends much of its time rereading weights and building cache state before any useful decoding starts.

Concretely, the engine looks up the incoming prompt against prefixes whose KV tensors already exist. A block-based system hashes token blocks and reuses matching KV pages; a radix-tree system walks shared token spans and reuses the matching prefix. The useful metric is usually token-weighted: cached prompt tokens divided by total prompt tokens over a window. That matters because missing one long transcript can dominate many small successful hits.

The trade-off is that cache lookup, page tracking, eviction, and routing are now on the request path. A prefix is reusable only if the tokens, positions, model configuration, adapter, template details, and attention rules are compatible. It is common to confuse apparent text reuse with KV reuse. Timestamps, reordered tools, hidden separators, different tokenisers, or sliding-window position changes can turn a visually identical prompt into a miss.

Engineers meet hit rate in vLLM automatic prefix caching, SGLang RadixAttention, TensorRT-LLM KV reuse, and distributed serving systems that route requests towards workers holding useful pages. In practice, dashboards should separate local realised hits from globally possible hits, track avoided prefill tokens, cache residency, eviction reasons, and the latency split across cache lookup, remaining prefill, and decode. A low realised hit rate is often a scheduling or prompt-canonicalisation problem.

Go deeper

Inference EngineeringHit RateMeasure KV cache hit rate and translate it to cost and latency. Understand why production agent engineers call this the single most important metric --- a prefix-cache-aware scheduler delivers 57× faster response and 2× throughput on identical hardware.Open the lesson

Common questions

Is hit rate measured per request or per token?: For production inference, per-token measurement is usually the more honest view. Request-weighted hit rate can make a system look healthy while a few very long prompts miss and consume most of the prefill work. Token-weighted hit rate counts how many prompt tokens actually reused KV entries out of all prompt tokens served.
Why can identical-looking prompts still miss the KV cache?: KV entries are tied to exact token sequences and execution conditions, not just visible text. A different chat template, tokenizer, tool order, adapter, quantisation setup, position convention, or hidden separator can make stored keys and values invalid for the new request. The cache can only reuse prefixes that are semantically and mechanically identical for the model run.
Does a higher hit rate always mean lower latency?: Usually, but not always. It depends on prefix length, lookup overhead, memory pressure, eviction behaviour, and load balancing. Reusing a long stable prefix can remove substantial prefill work. Chasing tiny or rare matches can add routing and cache-management cost, or overload a worker that happens to hold a marginally useful prefix.
How is hit rate different from FlashAttention?: Hit rate is about avoiding prefill work by reusing KV tensors that were already created for a matching prefix. FlashAttention makes attention computation more IO-efficient while the computation is happening. They optimise different parts of the system: one skips repeated prefix computation, the other executes attention more efficiently when computation is still required.

Every term