06.06 · Walkthrough
Expert Parallelism
Place experts across devices, trace the all-to-all that routing requires, and explain why expert load imbalance, not raw FLOPs, is what limits a wide-EP deployment.
Expert parallelism shards sparse MoE experts across devices, then routes token activations through dispatch and combine collectives. The hard part is not usually expert MLP throughput, but uneven router buckets and cross-device all-to-all traffic. A wide deployment is gated by tail latency from hot experts, small batches and poor placement.
What this lesson answers
- how does expert parallelism route MoE tokens
- why does MoE expert imbalance limit inference
- when does expert parallelism become slower than replication
Notes
Expert parallelism for a sparse MoE layer shards the expert MLPs, not the tokens or weights of a dense layer: a router computes logits for token , selects experts by , and the layer output is . With devices and experts, a common placement is experts per GPU; after routing, each rank must send the hidden states of tokens whose selected experts live elsewhere, execute its resident experts on the received token buckets, then send the expert outputs back for weighted…
References
Common questions
- What is expert parallelism in MoE inference?
- Expert parallelism places different MoE expert MLPs on different devices. The router chooses experts for each token, activations are sent to the devices that own those experts, the expert outputs are computed there, and results are sent back for weighted combine. It shards the sparse experts, not the attention cache or a dense layer.
- Why is all-to-all communication central to expert parallelism?
- After routing, many tokens need experts that are not local to the device holding the token batch. Each rank must exchange activation slices with other ranks, run its resident experts, then return outputs for combine. That dispatch and return path creates collective traffic, synchronisation, padding and queueing, especially when experts are spread across nodes.
- Why does load imbalance hurt MoE serving more than raw FLOPs?
- An MoE layer finishes when the slowest rank finishes, not when the average expert finishes. If one expert receives many more token copies than others, its owning device becomes the critical path while other devices wait. In inference, dropping overflow tokens is usually unacceptable, so placement, batching, replicated hot experts and routing behaviour matter more than peak compute.
Short definition: what is Expert Parallelism?