05.06 · Walkthrough
KV Cache Quantisation
Quantise the KV cache to FP8 or NVFP4, the largest win available at long context and large batch, because the cache is what decode is actually reading. Then find the point where recall starts to degrade.
KV cache quantisation cuts the bytes read by attention during decode by storing keys and values in FP8 or NVFP4, then using suitable scales and kernels at read time. It matters most at long context and large batch, where HBM traffic from the cache dominates latency and capacity.
What this lesson answers
- how does KV cache quantisation speed up decode
- when should I use FP8 KV cache
- how do I test recall loss from NVFP4
Notes
KV-cache quantisation stores the per-layer keys and values used by attention in fewer bits than the model weights or activations, so each decode step reads fewer cache bytes. For a decoder with layers, KV heads, head dimension , context length , batch , and element size bytes, the resident cache is , where the is for K and V.
Common questions
- Why is KV cache quantisation often a bigger win than weight quantisation during decode?
- During decode, each new token attends over the existing context, so the engine repeatedly reads stored keys and values. At long context or large batch, those reads can dominate HBM traffic. Shrinking the KV cache reduces the bytes moved on the hot path, while weight quantisation may not address the main bottleneck.
- Is FP8 KV cache always faster than FP16?
- No. FP8 helps when KV reads dominate the decode step and the attention kernel can hide conversion and scale handling. For short prompts, tiny batches, poor page locality, or unfused kernels, the extra scale reads and dequantisation work can offset the bandwidth saving. The right test is end-to-end decode latency on the target workload.
- How should I check whether NVFP4 hurts model quality?
- Use retrieval-heavy long-context tests, not only perplexity. Quantisation noise in keys can change attention ranking between the correct span and distractors, causing abrupt recall loss. Sweep FP16, FP8 scale granularities, and NVFP4 layouts across the intended context and batch, then choose the lowest-bit format whose recall curve stays flat.
Short definition: what is KV Cache Quantisation?