How is EAGLE different from Medusa?

Medusa predicts several future tokens from the same current hidden state, often as parallel heads and sometimes as a candidate tree. EAGLE drafts future hidden features step by step, then maps those features through the original LM head. That makes later draft positions condition on earlier draft state, which usually matches autoregressive generation better.

Is MTP just speculative decoding?

No. Speculative decoding is the serving procedure: propose candidate tokens, verify them with the target, and accept a valid prefix. MTP is a way to train the model so its own representations can predict multiple future tokens. An MTP model can be used for speculative decoding, but the runtime still needs explicit verifier and scheduling support.

Why not use a separate small draft model?

A separate draft model can work, but it has to read its own weights, maintain its own KV cache, fit into the serving graph, and be placed across hardware. In-model heads reuse the target model’s state and avoid a second model replica. That is why production systems often prefer trained draft heads when the checkpoint supports them.

When can EAGLE or MTP make serving slower?

They can lose when acceptance is low or batching is already highly efficient. Verification still consumes attention, memory bandwidth, launch overhead, and scheduler complexity. If speculative branches expand the batch more than accepted tokens shorten the decode loop, latency and throughput can degrade. The honest answer is workload-dependent and must be measured.

EAGLE and MTP

EAGLE and MTP are in-model speculative decoding methods that attach or train future-token predictors inside the main language model, so a serving engine can propose several continuations and verify them with the target model. They reduce repeated weight and KV reads by accepting multiple tokens from one target pass.

Autoregressive decoding is expensive because the same large model is invoked for every new token. In the decode phase, the bottleneck is often moving model weights and cached attention state through GPU memory, not doing the arithmetic itself. A separate small draft model can propose tokens, but it brings its own weights, cache, scheduling, and placement problems. EAGLE and MTP exist to get speculative decoding benefits without running another transformer beside the target.

The serving loop is propose, score, then verify. A draft mechanism suggests a short continuation. The target model then evaluates those candidate tokens under its own distribution, accepting the longest valid prefix and rolling back the rest. Medusa-style heads predict future tokens in parallel from the current hidden state. EAGLE instead predicts future hidden features autoregressively and reuses the target LM head for logits. MTP trains the base model itself to predict multiple future tokens, rather than bolting the ability on later.

The common misunderstanding is that these methods make several tokens free. They do not. They add heads, verification work, extra candidate positions, more complex batching, and sometimes more KV traffic. The win depends on accepted tokens per target pass being high enough to amortise that overhead. Acceptance is workload-sensitive: deterministic formats, code, and low-entropy continuations tend to work better than high-temperature creative sampling or sharp distribution shifts.

Engineers meet EAGLE and MTP in inference runtimes, not just papers: speculative decoding paths in systems such as vLLM, SGLang, TensorRT-LLM, and model families that export MTP modules. Practical support means more than a checkpoint label. The runtime needs the extra heads or modules in the graph, a verifier, rollback support for KV state, quantisation coverage, and batching rules that avoid making all requests wait for the largest speculative branch.

Go deeper

Inference EngineeringEAGLE and MTPDescribe trained draft heads: Medusa's parallel heads, EAGLE's feature-level autoregression, and multi-token prediction built into the model. Explain why these, not separate draft models, are what production engines ship.Open the lesson

Common questions

How is EAGLE different from Medusa?: Medusa predicts several future tokens from the same current hidden state, often as parallel heads and sometimes as a candidate tree. EAGLE drafts future hidden features step by step, then maps those features through the original LM head. That makes later draft positions condition on earlier draft state, which usually matches autoregressive generation better.
Is MTP just speculative decoding?: No. Speculative decoding is the serving procedure: propose candidate tokens, verify them with the target, and accept a valid prefix. MTP is a way to train the model so its own representations can predict multiple future tokens. An MTP model can be used for speculative decoding, but the runtime still needs explicit verifier and scheduling support.
Why not use a separate small draft model?: A separate draft model can work, but it has to read its own weights, maintain its own KV cache, fit into the serving graph, and be placed across hardware. In-model heads reuse the target model’s state and avoid a second model replica. That is why production systems often prefer trained draft heads when the checkpoint supports them.
When can EAGLE or MTP make serving slower?: They can lose when acceptance is low or batching is already highly efficient. Verification still consumes attention, memory bandwidth, launch overhead, and scheduler complexity. If speculative branches expand the batch more than accepted tokens shorten the decode loop, latency and throughput can degrade. The honest answer is workload-dependent and must be measured.

Every term