05.12 · Walkthrough
Stacking the Wins
Compose quantisation, cache reuse and speculation, and measure the combination rather than adding up the individual claims: they contend for the same bandwidth and the same batch slots.
Inference optimisations do not stack as independent multipliers. Quantisation, KV cache reuse and speculative decoding all shift pressure between memory bandwidth, compute, scheduling and batch capacity. The reliable method is to run controlled ablations on the same traffic and measure accepted tokens per target step, latency and occupancy together.
What this lesson answers
- do inference speedups from quantisation and speculation multiply
- how does KV cache reuse affect decode latency
- what should I measure when stacking inference optimisations
Notes
Stacking inference optimisations means changing both the bytes moved per accepted token and the number of model steps required per accepted token, then measuring the combined fixed point rather than multiplying isolated speedups. A useful first-order latency model for decode is , where is weight traffic per target-model step, is KV-cache traffic, is activation and metadata traffic, is sustained memory bandwidth, is FLOPs per step, is sustained compute, and is accepted…
References
Common questions
- Why can quantisation make KV cache costs more important?
- Weight quantisation cuts the traffic needed to read model weights during decode. Once that traffic shrinks, other costs that were previously hidden become visible: KV reads, activation traffic, dequantisation work, scheduling overhead and batch contention. The bottleneck may move, so the next optimisation is no longer attacking the same limiting factor.
- Does prefix caching reduce per-token decode cost?
- Prefix caching mainly saves prefill work by avoiding repeated computation and KV writes for shared prompts. During decode, each active sequence still attends over its context, including reused prefix tokens. Unless attention is windowed, compressed or otherwise changed, cache reuse improves admission and time to first token more than steady per-token decode.
- When does speculative decoding stop helping?
- Speculation loses value when the draft model's proposed tokens are often rejected or when draft generation and verification consume capacity that could serve other requests. High-entropy sampling, tool boundaries, brittle code tokens and distribution mismatch can reduce acceptance. The relevant metric is accepted tokens per target step after including draft cost and scheduler effects.
Short definition: what is Stacking the Wins?