Is KV cache quantisation the same as weight quantisation?

No. Weight quantisation compresses the model parameters, mainly affecting how weights are stored and loaded. KV cache quantisation compresses the per-request attention state produced during inference. That state grows with context and batch, and it is reread during decode, so it attacks a different bottleneck.

Why can KV cache quantisation hurt long-context recall?

Attention chooses between many previous tokens by comparing query-key scores. Quantisation noise in keys can slightly shift those scores. In a retrieval task, the correct span may be separated from a distractor by a small margin, so a small key error can change the winner even if overall language modelling metrics look fine.

When is FP8 or NVFP4 KV cache worth using?

It is most useful when decode is dominated by reading a large KV cache, typically long context, larger batches, or memory-bandwidth-bound serving. It may be neutral or worse for short prompts, small batches, poor page locality, or kernels that lose a fused fast path. The answer depends on measurement at the target workload.

How should an engineer validate a KV cache format?

Compare the full-precision cache with candidate FP8 and NVFP4 layouts on the actual serving stack. Include long-context recall, RAG, citation, and prefix-cache reuse tests, not just perplexity. Sweep scale granularity and context length, then choose the lowest-precision format whose recall curve remains acceptable.

KV Cache Quantisation

KV cache quantisation is the practice of storing attention keys and values in a lower-precision format, such as FP8 or NVFP4, during LLM inference. It reduces the bytes read at each decode step, which matters because long-context decoding repeatedly streams the accumulated KV cache from GPU memory.

The need appears when decoding long prompts or serving large batches. After prefill, each new token forms a query that attends over all earlier keys and values. The model may not be limited by arithmetic at that point; it is often waiting on memory bandwidth. Weight quantisation helps load model parameters, but decode keeps rereading the growing KV cache, so compressing that cache can be the larger lever.

Mechanically, the server does not simply run all attention in a tiny integer type. It writes each layer’s K and V tensors into a compact representation, stores scale information, and either dequantises inside the attention kernel or uses a kernel that consumes the low-precision layout directly. Scale granularity matters: one scale is cheap but crude, while per-head, per-page, or per-token scaling better follows changing distributions.

The trade-off is accuracy, kernel complexity, and sometimes latency. Key error can change attention logits before softmax, so a relevant span can lose to a distractor; value error affects the weighted result after attention has been chosen. Lower precision also adds scale reads, packing or unpacking work, and layout constraints. For short contexts or small batches, those costs can outweigh the saved memory traffic.

Engineers meet KV cache quantisation in inference servers and low-precision attention backends: vLLM-style paged caches, TensorRT-LLM kernels, SGLang serving paths, and disaggregated prefill/decode systems. The practical question is not whether FP8 or NVFP4 is universally safe. It depends on the model, context length, batch shape, scale layout, and workload. Retrieval-heavy tests are usually more revealing than average perplexity alone.

Go deeper

Inference EngineeringKV Cache QuantisationQuantise the KV cache to FP8 or NVFP4, the largest win available at long context and large batch, because the cache is what decode is actually reading. Then find the point where recall starts to degrade.Open the lesson

Common questions

Is KV cache quantisation the same as weight quantisation?: No. Weight quantisation compresses the model parameters, mainly affecting how weights are stored and loaded. KV cache quantisation compresses the per-request attention state produced during inference. That state grows with context and batch, and it is reread during decode, so it attacks a different bottleneck.
Why can KV cache quantisation hurt long-context recall?: Attention chooses between many previous tokens by comparing query-key scores. Quantisation noise in keys can slightly shift those scores. In a retrieval task, the correct span may be separated from a distractor by a small margin, so a small key error can change the winner even if overall language modelling metrics look fine.
When is FP8 or NVFP4 KV cache worth using?: It is most useful when decode is dominated by reading a large KV cache, typically long context, larger batches, or memory-bandwidth-bound serving. It may be neutral or worse for short prompts, small batches, poor page locality, or kernels that lose a fused fast path. The answer depends on measurement at the target workload.
How should an engineer validate a KV cache format?: Compare the full-precision cache with candidate FP8 and NVFP4 layouts on the actual serving stack. Include long-context recall, RAG, citation, and prefix-cache reuse tests, not just perplexity. Sweep scale granularity and context length, then choose the lowest-precision format whose recall curve remains acceptable.

Every term