Speculative Decoding
Speculative decoding is an LLM inference technique where a cheap draft model proposes k future tokens, the target model scores those proposed positions in one causal forward pass, and a rejection-sampling accept or repair step preserves the exact output distribution of normal target-model sampling.
Autoregressive decoding is slow because the target model normally emits tokens one after another, rereading the large model weights for each step. In the common memory-bandwidth-limited decode regime, the expensive part is often not the arithmetic for a single token but moving the model through HBM repeatedly. Speculative decoding exists to spend a little work on a cheaper predictor so the large model can validate several possible next steps at once.
The mechanism is draft then verify. A smaller model, extra proposal heads, or a feature-based drafter generates k candidate tokens. The target model then runs over the current prefix plus that proposed suffix using a causal mask, producing target probabilities for each proposed position. Tokens are checked in order: if the target probability supports the draft token sufficiently relative to the draft probability, it is accepted; at the first failure, a replacement is sampled from the remaining target probability mass.
The important subtlety is that this is not merely an approximation to the larger model. With the right rejection-sampling formula, and with temperature, top-k, top-p, grammar masks, and other sampling transforms applied consistently to both distributions, the emitted sequence has the same distribution as ordinary sampling from the target model. If all draft tokens are accepted, the target can also provide the next token, giving an extra step from the same verification pass.
The trade-off is that speculation only helps when the draft is cheap and usually agrees with the target. If the draft distribution is poor, the system pays for serial draft work, a multi-token target verification pass, and rollback handling while emitting few accepted tokens. Large batches, short generations, domain shift, high temperature, constrained decoding, and attention-bound serving can all erase the gain. Exactness also requires careful KV-cache handling: accepted target KV may be committed, rejected suffix KV must not persist.
Engineers meet speculative decoding in inference servers such as vLLM, TensorRT-LLM, SGLang, and newer disaggregated runtimes. It may appear as a separate draft model, Medusa-style proposal heads, or EAGLE-style feature prediction. In practice, the hard parts are not just modelling quality but scheduling, paged KV allocation, continuous batching, grammar-mask consistency, and deciding dynamically when speculation is worth enabling for a request.
Common questions
- Is speculative decoding approximate?
- Not necessarily. The proposer can be approximate, but the final sampling need not be. When the verifier uses the target distribution and the rejection-sampling correction is implemented correctly, the output distribution matches target-model sampling. The common misunderstanding is to treat the draft tokens as trusted predictions rather than candidates that must be statistically accepted or repaired.
- Why can one target forward pass verify several tokens?
- The target is run on the proposed suffix with a causal mask. Each proposed position can attend to earlier proposed tokens exactly as it would have if those tokens had already been generated. That makes the pass larger than a single-token decode, but it avoids streaming the target weights separately for every proposed token.
- When does speculative decoding fail to speed things up?
- It depends on acceptance rate, draft cost, batching, and the server bottleneck. If the draft often disagrees with the target, most proposed tokens are rejected and the extra work is wasted. If serving is compute-bound, attention-bound, dominated by short outputs, or complicated by constrained decoding, the expected weight-read saving may not translate into lower latency.