Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

Why this one

Read this after Outrageously Large Neural Networks, because Switch is the version of mixture of experts that made the idea feel buildable instead of exotic. The core move is brutal simplification: route each token to one expert, not several, then spend the saved complexity on making the system train and run. People often talk about MoE as if it is free capacity, but the paper is really about the tax: routing collapse, load balancing, communication between devices, precision trouble, and batches that stop looking like normal dense training. If you are used to thinking of a model as one fixed pile of weights, this changes the accounting. A trillion parameters can be mostly idle on any given token, which is why parameter count no longer maps cleanly to inference or training cost. Worth your evening because modern frontier model specs are unreadable without this distinction.