05.11 · Concept
When Speculation Loses
Explain why speculative decoding is a latency technique and not a throughput one: at high batch size the GPU is already compute-saturated, so verification costs real time and a low acceptance rate makes the server slower.
Speculative decoding reduces interactive latency when decode is under-utilising the GPU, but it can hurt throughput once batching has filled the tensor cores. Verification is then real target-model compute, not spare work. If the drafter’s tokens are rejected often enough, the server spends more time per emitted token than ordinary decoding.
What this lesson answers
- why speculative decoding hurts throughput at high batch
- when does speculative decoding make inference slower
- is speculative decoding a latency or throughput optimisation
Notes
Speculative decoding is a two-model or two-path decoding mechanism: a cheap drafter proposes future tokens , then the target model verifies them in one forward pass and accepts the longest prefix consistent with the target distribution. In exact speculative sampling from Leviathan et al., 2023, with drafter distribution and target , token is accepted with probability , and the expected number of tokens emitted per target verification is $E[L]=\sum_{i=1}^{\gamma}\Pr(\text{first }i\text{ draft tokens…
Common questions
- Why does speculative decoding help at low batch size?
- At low batch size, decode is often limited by memory bandwidth and launch overhead rather than raw arithmetic. The target model may have unused compute capacity during each step. Verifying several drafted positions in one target pass can turn that idle capacity into fewer sequential target invocations, reducing the critical path for a single request.
- Why does speculative decoding fail at high batch size?
- With high continuous batching, the GPU is already busy doing large matrix multiplies for normal decoding. A speculative verification pass must evaluate extra target positions, so it consumes additional compute. If the accepted draft length is not high enough, the extra verification and drafter work exceeds the steps saved.
- What should be measured before enabling speculative decoding in production?
- Measure accepted tokens per verification alongside GPU utilisation. High acceptance with unused compute indicates speculation may reduce latency. Low acceptance while the GPU is compute-saturated means verification is adding work on the bottleneck path. In that case, speculation can reduce capacity and increase per-token serving time.