Why does MoE help if all the weights still exist?

Because a token only runs through a few selected experts, not every expert. That lowers the active computation for that token. The catch is that serving must still keep the unchosen experts in memory somewhere, because the next token may route to them. MoE trades dense compute for residency, routing, and communication complexity.

Is MoE mainly a training technique or an inference technique?

It affects both, but in different ways. In training, sparsity helps scale parameter count without activating everything for every token. In inference, the important shift is operational: active FLOPs may be small, while the serving system must place all experts, route tokens to them, and handle skew and latency at each MoE layer.

When can MoE be slower than a dense model?

It depends on batch size, interconnect, expert placement, routing balance, and kernel efficiency. Low-batch decode is especially risky: routed expert batches can be under-filled, and small all-to-all transfers can cost more in latency than they save in compute. If expert communication and imbalance dominate, sparsity no longer translates into lower token latency.

Mixture of Experts

A Mixture of Experts is a transformer design where each token is sent by a learned router to a small subset of feed-forward expert networks instead of through one dense feed-forward block. It reduces active computation per token, but all experts’ weights still have to be resident and reachable during serving.

Dense transformers make inference expensive in a straightforward way: each token tends to touch the same large set of weights. MoE changes that shape. It lets a model have many more parameters than it activates for any one token, so the arithmetic per token can look modest. The serving problem does not disappear, though. It moves from pure matrix-multiplication throughput to keeping a much larger collection of weights placed, addressable, and balanced across devices.

In an MoE layer, a router scores the token representation, chooses the top-k experts, sends the token to those expert MLPs, then combines their outputs using the router’s weights. In a distributed implementation, that means permuting tokens by destination expert, running grouped GEMMs on each expert’s batch, then undoing the permutation. With expert parallelism, different devices hold different experts, so every MoE block contains routing and communication as well as compute.

The trade-off is that sparse activation saves expert FLOPs but introduces memory residency, communication, and batching problems. During prefill, many tokens can often be grouped into useful expert batches. During decode, each sequence contributes only one new token at a step, so expert batches can be tiny and remote sends can dominate. Router imbalance is another common failure mode: if many tokens choose the same expert, one device becomes the barrier while others wait.

Engineers meet MoE in inference servers, model-parallel runtimes, and GPU kernels for models such as Mixtral, Switch Transformer, GShard-style systems, and DeepSeek-style sparse models. The practical questions are placement, expert parallelism, all-to-all latency, fused routing, grouped GEMM efficiency, KV-cache pressure, and scheduling. A common misunderstanding is that MoE is simply a cheaper large model. It is cheaper only on the activated path; the fleet still pays to host the experts.

Go deeper

Inference EngineeringMixture of ExpertsExplain why MoE changed the serving problem rather than just the training one: compute per token stays small while the memory that must be resident stays large, so the bottleneck moves.Open the lesson

Common questions

Why does MoE help if all the weights still exist?: Because a token only runs through a few selected experts, not every expert. That lowers the active computation for that token. The catch is that serving must still keep the unchosen experts in memory somewhere, because the next token may route to them. MoE trades dense compute for residency, routing, and communication complexity.
Is MoE mainly a training technique or an inference technique?: It affects both, but in different ways. In training, sparsity helps scale parameter count without activating everything for every token. In inference, the important shift is operational: active FLOPs may be small, while the serving system must place all experts, route tokens to them, and handle skew and latency at each MoE layer.
When can MoE be slower than a dense model?: It depends on batch size, interconnect, expert placement, routing balance, and kernel efficiency. Low-batch decode is especially risky: routed expert batches can be under-filled, and small all-to-all transfers can cost more in latency than they save in compute. If expert communication and imbalance dominate, sparsity no longer translates into lower token latency.

Every term