Why can’t I multiply the speedups from quantisation, caching and speculation?

Because they often attack the same bottleneck or move the bottleneck somewhere else. If quantisation reduces weight bandwidth enough, KV-cache reads, scheduling overhead or draft-model work may dominate. If prefix reuse removes prefill, decode may still be limited by long-context attention. Multiplication assumes independence, and serving optimisations are usually not independent.

What is the common misunderstanding about speculative decoding in this stack?

The common mistake is to count only the target-model verification saving. Speculation also runs a draft model, consumes batch and attention capacity during verification, and depends on acceptance rate. If the draft distribution is mismatched, or sampling is difficult, accepted tokens per target step can fall enough that ordinary decoding is faster.

Does prefix caching make long-context decode cheap?

Not by itself. Prefix caching avoids recomputing shared prefill and rewriting the same KV blocks, which can greatly improve admission and time to first token. During decode, however, each active sequence may still attend over the cached context. To reduce that cost, you need another mechanism such as windowing, compression, offload or a different attention strategy.

What should I benchmark when stacking inference optimisations?

Benchmark the actual serving mix with the same prompts, sampling settings, concurrency and latency target. Compare an ablation table rather than isolated claims: baseline, each optimisation alone, and the combined system. Include accepted tokens per target step, cache hit rate, batch occupancy and hardware utilisation, because throughput alone can hide queueing or latency regressions.

Stacking the Wins

Stacking the Wins is the practice of combining inference optimisations such as quantisation, KV-cache reuse and speculative decoding, then benchmarking the combined serving system instead of multiplying separate speedup claims. It treats latency as a shared bottleneck problem: the optimisations compete for memory bandwidth, compute, scheduler capacity and batch slots.

This is necessary because inference optimisations rarely act on independent parts of the system. Quantisation may cut weight traffic, cache reuse may remove repeated prefill, and speculation may reduce target-model steps per accepted token. But all of them still run through the same memory hierarchy, kernels, queueing policy and scheduler. Once one bottleneck is relieved, another often becomes the limiter, so isolated benchmark results can be directionally useful but numerically misleading.

Mechanically, stacking the wins means decomposing generation into bytes moved, FLOPs executed, cache blocks read or written, and accepted tokens produced per target forward. Quantisation changes the weight and sometimes KV-cache byte terms. Prefix caching avoids recomputing shared prompt work and reuses stored KV blocks. Speculation has a draft model propose several tokens, while the target model verifies them in a parallel block and accepts only the valid prefix.

The trade-off is that each optimisation adds its own overhead and can consume the resource another optimisation needs. Quantised kernels may pay for dequantisation or packing. Prefix caches need lookup, eviction and resident memory, and they do not remove decode attention over the context. Speculation adds draft-model work and verification slots, and can be worse when acceptance is poor. The honest answer is usually: it depends on traffic, prompts, context length, sampling and batching.

Engineers meet this in serving stacks that combine continuous batching, paged KV caches, prefix sharing, quantised kernels and speculative decoding. The useful experiment is an ablation under identical traffic: baseline, each optimisation alone, then all together. Measure time to first token, inter-token latency, accepted tokens per target step, KV hit rate, batch occupancy and hardware utilisation. The combined result is the product you ship, not the arithmetic product of marketing claims.

Go deeper

Inference EngineeringStacking the WinsCompose quantisation, cache reuse and speculation, and measure the combination rather than adding up the individual claims: they contend for the same bandwidth and the same batch slots.Open the lesson

Common questions

Why can’t I multiply the speedups from quantisation, caching and speculation?: Because they often attack the same bottleneck or move the bottleneck somewhere else. If quantisation reduces weight bandwidth enough, KV-cache reads, scheduling overhead or draft-model work may dominate. If prefix reuse removes prefill, decode may still be limited by long-context attention. Multiplication assumes independence, and serving optimisations are usually not independent.
What is the common misunderstanding about speculative decoding in this stack?: The common mistake is to count only the target-model verification saving. Speculation also runs a draft model, consumes batch and attention capacity during verification, and depends on acceptance rate. If the draft distribution is mismatched, or sampling is difficult, accepted tokens per target step can fall enough that ordinary decoding is faster.
Does prefix caching make long-context decode cheap?: Not by itself. Prefix caching avoids recomputing shared prefill and rewriting the same KV blocks, which can greatly improve admission and time to first token. During decode, however, each active sequence may still attend over the cached context. To reduce that cost, you need another mechanism such as windowing, compression, offload or a different attention strategy.
What should I benchmark when stacking inference optimisations?: Benchmark the actual serving mix with the same prompts, sampling settings, concurrency and latency target. Compare an ablation table rather than isolated claims: baseline, each optimisation alone, and the combined system. Include accepted tokens per target step, cache hit rate, batch occupancy and hardware utilisation, because throughput alone can hide queueing or latency regressions.

Every term