Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, et al.2017ICLR 2017
Read it on arxiv.org(opens in a new tab)Why this one
Read this after Scaling Laws for Neural Language Models and before Switch Transformers. The useful idea is not just “make the model huge.” It is that parameter count and compute can be partly separated if each token only visits a small routed subset of the network. That sounds simple until you try to train it: the router collapses, experts go unused, batches become uneven, and the hardware starts punishing your cleverness. This paper is worth your evening because it treats sparsity as a systems problem as much as a modeling trick. People often remember MoE as free capacity, but the real lesson is load balancing, routing noise, expert utilization, and communication cost. If you are building or fine-tuning large models, this is the ancestor of many modern sparse LLM choices, including why they are powerful and why they are annoying to serve.
What to take away
- Sparse routing lets a model add many parameters while activating only a few experts per token.
- The gating network needs explicit load balancing or training collapses into a few popular experts.
- MoE gains depend on hardware layout and communication, not just parameter counts in a table.
Reads with
- Scaling Laws for Neural Language Models
sets up why sparse capacity is attractive when dense compute scaling gets expensive
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
turns this routed expert layer into the simpler sparse transformer recipe
- DeepSeek-V3 Technical Report
shows the same routing and load balancing problems in a modern production scale llm