06.05 · Concept
Mixture of Experts
Explain why MoE changed the serving problem rather than just the training one: compute per token stays small while the memory that must be resident stays large, so the bottleneck moves.
Mixture-of-Experts makes inference sparse in compute but not in memory. Each token activates only selected experts, so per-token work can look like a smaller dense model, while the serving fleet still needs all expert weights resident, routable, balanced, and close enough to avoid communication dominating latency.
What this lesson answers
- why MoE inference is memory bottlenecked
- how expert routing affects serving latency
- why MoE needs expert parallelism
Notes
A Mixture-of-Experts transformer replaces each dense feed-forward block with separate MLP “experts” and a router that selects only of them per token, usually by and . The key serving fact is that activated FLOPs scale with of the expert MLPs, while resident parameters scale with all experts: roughly but .
References
Common questions
- Why does MoE reduce compute but not model memory?
- An MoE layer contains many expert networks, but the router sends each token to only a small subset. That cuts the work done for that token. The unused experts still have parameters, and production serving must keep those weights available somewhere in the fleet, so resident memory remains tied to the full expert set.
- Why is MoE serving harder during decode than prefill?
- Prefill has many tokens available at once, so routed tokens can be grouped by expert and run as useful batched matrix multiplies. Decode produces one token per active sequence per step. That fragments expert batches, increases routing and all-to-all overhead, and can make fabric latency more important than raw GPU arithmetic.
- What makes expert imbalance a production problem?
- If the router sends too many tokens to one expert, the device holding that expert becomes the slowest point in the step. Other devices may be idle while waiting for it. The advertised sparse compute saving then matters less than routing skew, batching quality, and the latency of moving activations between workers.
Short definition: what is Mixture of Experts?