Expert Parallelism
Expert parallelism is a way to run sparse mixture-of-experts layers by placing different expert networks on different devices and routing each token to the devices that own its chosen experts. It saves memory and compute per token, but introduces collective communication and makes latency depend on the busiest expert or rank.
Sparse MoE models contain many expert MLPs, but each token uses only a small subset of them. Replicating every expert on every device wastes memory, especially in large serving fleets. Expert parallelism exists because the experts, not the attention block or KV cache, become the large shardable object. The hard part is that tokens in a batch do not naturally arrive already grouped by the devices that own their selected experts.
In an MoE layer, a router scores each token against the available experts, chooses the top experts, and produces weights for combining their outputs. Each device owns some experts. After routing, the system permutes and sends token hidden states to the owning devices, runs local expert MLPs on the received token buckets, then sends the results back so the original token order can be restored and weighted together.
The cost is not just extra network traffic. Expert parallelism requires all-to-all dispatch and another all-to-all for the returned outputs, plus local permutation, padding, grouped GEMMs, and synchronisation. The common misunderstanding is to compare only total FLOPs. The layer finishes when the slowest rank finishes, so one hot expert or one unlucky bucket can dominate latency while other devices sit idle.
In practice, engineers meet expert parallelism in MoE inference stacks, distributed schedulers, and kernels that fuse routing, permutation, grouped matrix multiplies, and finalisation. It interacts with batching because larger prefill or decode batches give each expert enough tokens to run efficiently. It also interacts with placement: keeping frequent expert traffic within a fast interconnect can matter more than spreading experts as widely as possible.
Common questions
- How is expert parallelism different from tensor parallelism?
- Tensor parallelism splits the computation of one dense layer across devices, so all participating devices work on the same layer weights. Expert parallelism assigns whole experts to devices and sends tokens to the devices that own the selected experts. They can be combined, but they solve different scaling problems.
- Why does expert load imbalance matter so much?
- An MoE layer is gated by the slowest expert bucket or rank, not by the average amount of work. If routing sends many tokens to one expert, the owning device must process that larger bucket while others finish early. Unless the expert is replicated or placement changes, idle devices cannot easily absorb the excess.
- Is expert parallelism always faster for MoE inference?
- No. It depends on batch size, router skew, network topology, expert placement, and whether expert GEMMs are large enough to use the hardware well. With small decode batches or cross-node all-to-all traffic, communication and imbalance can outweigh the memory benefit of sharding experts.