What is a draft model in speculative decoding?

A draft model is a cheaper proposal generator. It predicts several possible next tokens, then the target model checks them in a single verification step. Speedup comes from committing more than one token per expensive target pass, not from the draft being generally accurate in the abstract.

When is an n-gram or prompt-lookup draft the right choice?

Use prompt lookup when output is likely to copy text from the prompt, retrieved documents, recent conversation, code context, or a structured template. Its main advantage is that it adds almost no GPU cost, so even modest acceptance can help. It performs poorly when generation is creative, translated, or reasoning-heavy.

Why can self-speculation fail to speed up inference?

Self-speculation still has a draft cost, even though there is no separate model. If the model uses too much of the full network to propose tokens, the accepted length must be very high to break even. It works best when the auxiliary predictor is much cheaper than a normal target pass.

Fewer Bytes, Fewer Steps

05.09 · Walkthrough

Draft Models

Choose a draft: a small model from the same family, an n-gram or prompt-lookup draft that costs no GPU at all, or self-speculation. Compute the acceptance rate you need for each to break even.

Draft models speed speculative decoding only when accepted tokens repay their cost. The practical choice is between a same-family small model, a nearly free n-gram or prompt-lookup draft, and self-speculation. The key calculation is break-even: verifier cost plus draft cost divided by committed progress must beat normal decoding.

What this lesson answers

how to choose a speculative decoding draft model
when does prompt lookup decoding beat normal decoding
how to calculate speculative decoding break even

Notes

Draft models are the proposal mechanism in speculative decoding: a cheap generator $q$ produces $k$ candidate tokens, the target model $p$ verifies them in one forward pass, and the sampler accepts the longest prefix whose sampled tokens are consistent with $p$ . In the Leviathan et al. 2023 / Chen et al. 2023 algorithm, if $x_{i} \sim q (\cdot ∣ x_{< i})$ , accept token $x_{i}$ with probability $min (1, p (x_{i}) / q (x_{i}))$ and, after the first rejection, sample a correction from the positive part of $p - q$ .

References

Fast Inference from Transformers via Speculative Decoding

Common questions

What is a draft model in speculative decoding?: A draft model is a cheaper proposal generator. It predicts several possible next tokens, then the target model checks them in a single verification step. Speedup comes from committing more than one token per expensive target pass, not from the draft being generally accurate in the abstract.
When is an n-gram or prompt-lookup draft the right choice?: Use prompt lookup when output is likely to copy text from the prompt, retrieved documents, recent conversation, code context, or a structured template. Its main advantage is that it adds almost no GPU cost, so even modest acceptance can help. It performs poorly when generation is creative, translated, or reasoning-heavy.
Why can self-speculation fail to speed up inference?: Self-speculation still has a draft cost, even though there is no separate model. If the model uses too much of the full network to propose tokens, the accepted length must be very high to break even. It works best when the auxiliary predictor is much cheaper than a normal target pass.

Short definition: what is Draft Models?

05.08 · Fewer Bytes, Fewer StepsSpeculative Decoding 05.10 · Fewer Bytes, Fewer StepsEAGLE and MTP