Draft Models
Draft models are cheap token proposers used in speculative decoding, where a fast approximation guesses several next tokens and the full target model verifies them in a single pass. They can be smaller sibling models, prompt or n-gram lookup mechanisms, or internal early-exit predictors inside the target model itself.
Autoregressive decoding is expensive because the target model normally advances by one token per full model pass. For large models, that pass is often dominated by moving weights and KV data rather than by arithmetic alone. Drafting exists to amortise that expensive verification work: if the target can safely commit several proposed tokens after one pass, latency per emitted token falls. The key question is not whether the draft sounds good, but whether enough of its proposals survive verification.
A draft produces k candidate tokens from the current prefix. The target model then scores those proposed positions in one verification pass. In the standard speculative sampler, each proposed token is accepted only when it is statistically compatible with the target distribution; after the first rejection, the sampler draws a correction from the target-side residual distribution. This preserves the target model’s sampling distribution while letting accepted prefixes advance the sequence faster than ordinary one-token decoding.
The trade-off is that drafting is not free. A small transformer draft consumes memory bandwidth and scheduling capacity; self-speculation spends skipped layers or auxiliary heads; prompt lookup is cheap but only works when future text repeats nearby text. Break-even depends on draft cost, verification cost, and mean accepted length. Commonly misunderstood: a low perplexity draft is not automatically useful. It must reduce critical-path time after all verification and batching overheads are included.
Engineers meet draft models in inference servers and decoding backends such as TensorRT-LLM, vLLM, SGLang, Dynamo-style serving stacks, and Kubernetes-oriented LLM deployments. Same-family small models are the usual choice when semantic match matters. N-gram and prompt-lookup drafts are attractive for retrieval, code, JSON, and long copied passages. Self-speculative schemes avoid cross-model coordination, but they need very cheap internal predictors or very high acceptance to pay off.
Common questions
- Is a draft model allowed to change the model’s output distribution?
- In speculative decoding as usually defined, no. The draft only proposes candidates. The target model verifies them with an accept/reject rule and supplies the correction after rejection, so sampling remains distributed as if the target model had decoded normally. Approximate production variants may relax this, but then they are trading exactness for speed.
- How do I choose between a small model, prompt lookup, and self-speculation?
- It depends on where the next tokens come from. Use a small same-family model when you need broad semantic prediction. Use prompt or n-gram lookup when outputs often copy input, recent context, code, or structured text. Use self-speculation when operating another model is operationally awkward, but check that the internal draft is genuinely cheap.
- What acceptance rate is good enough?
- There is no universal threshold. Compute break-even from the cost of verification plus the cost of producing draft tokens, divided by the mean number of committed tokens per cycle. If the verifier is close to one ordinary target pass and the draft is nearly free, almost any accepted token helps. If the draft is costly, acceptance must be much higher.
- Why can draft models make latency worse?
- They add work before verification. If proposed tokens are rejected often, the system pays for the draft and still advances by only the correction token. Long-context attention, uneven accepted lengths across a batch, KV cache repacking, and cross-device scheduling can also make the verification pass more expensive than the simplified mental model assumes.