05.08 · Concept
Speculative Decoding
Explain draft-then-verify: a cheap model proposes k tokens, the target model verifies them in one forward pass, and the rejection-sampling step makes the output distribution identical to decoding without it.
Speculative decoding speeds autoregressive inference by having a cheaper draft model propose several future tokens, then asking the target model to verify that suffix in a single pass. Accepted tokens are committed, rejected ones are repaired with rejection sampling, so the final samples match the target model’s normal decoding distribution.
What this lesson answers
- how does speculative decoding preserve exact sampling
- when does speculative decoding improve inference latency
- why can speculative decoding fail in production
Notes
Speculative decoding is the draft-then-verify mechanism from Leviathan et al., “Fast Inference from Transformers via Speculative Decoding” (ICML 2023), and Chen et al., “Accelerating Large Language Model Decoding with Speculative Sampling” (2023): a cheap draft distribution proposes tokens , then the target distribution is evaluated for all proposed positions in one target forward pass.
Common questions
- What is the basic idea behind speculative decoding?
- A smaller or cheaper proposer guesses a short continuation. The target model then scores those proposed positions together, instead of decoding them one at a time. Tokens that agree statistically with the target distribution are accepted. When a token is rejected, a corrected token is sampled and the speculative suffix after it is discarded.
- Does speculative decoding change the model’s output distribution?
- Not when implemented with the rejection-sampling correction and matching sampling rules. The draft model only proposes candidates; it does not define the final distribution. The accept, reject, and residual sampling steps ensure the emitted tokens are distributed as if they had been sampled directly from the target model.
- Why does speculative decoding sometimes make inference slower?
- It depends on the draft being cheap and usually close to the target. If the proposer is weak, many proposed tokens are rejected after both models have done work. The benefit also shrinks when decoding is not dominated by target weight reads, or when batching, cache management, kernel launches, and rollback costs dominate.
Short definition: what is Speculative Decoding?