05.10 · Concept
EAGLE and MTP
Describe trained draft heads: Medusa's parallel heads, EAGLE's feature-level autoregression, and multi-token prediction built into the model. Explain why these, not separate draft models, are what production engines ship.
Trained draft heads make speculative decoding practical by proposing future tokens inside the target model’s own execution path. Medusa predicts parallel token candidates, EAGLE predicts future features, and MTP trains future-token prediction into the checkpoint. Production engines prefer these designs because separate draft models add memory traffic, cache complexity, and scheduling cost.
What this lesson answers
- how do EAGLE and MTP speed up decoding
- why use draft heads instead of draft models
- what is Medusa speculative decoding in LLM serving
Notes
A trained draft head is an extra predictor attached to the target LLM so that one forward pass proposes several future tokens, after which the unmodified target distribution verifies them with speculative decoding. If the draft proposes and the target model gives , exact rejection sampling accepts token with probability for a draft distribution , while many engine implementations use greedy or lossless-compatible variants for deterministic serving.
References
Common questions
- What is a trained draft head in speculative decoding?
- A trained draft head is an extra predictor attached to the main language model. It proposes upcoming tokens or representations, then the target model checks those proposals using its normal distribution. The speedup comes from accepting several tokens per expensive target pass, without running a separate transformer as the proposer.
- How is EAGLE different from Medusa?
- Medusa predicts several future tokens from the current hidden state using parallel heads, which is simple but weak when later tokens depend heavily on earlier sampled tokens. EAGLE instead predicts future hidden features autoregressively, then uses the target model’s language head, keeping proposals closer to the model’s own internal trajectory.
- Why do production engines prefer in-model draft mechanisms?
- Large-model decoding is often limited by moving weights and KV data, not raw arithmetic. A separate draft model brings another serving graph, another cache, extra placement decisions, and more memory reads. In-model heads keep the proposer near the target forward path, so acceptance can reduce target steps without doubling operational complexity.
Short definition: what is EAGLE and MTP?