The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
Jonathan Frankle, Michael Carbin2019ICLR 2019
Read it on arxiv.org(opens in a new tab)Why this one
Read this after Dropout: A Simple Way to Prevent Neural Networks from Overfitting and Understanding the difficulty of training deep feedforward neural networks. The useful idea here is not that small networks can replace big ones, or that pruning is free compression. It is stranger: the dense model may be acting as a search space for trainable subnetworks, and the original random weights are part of what makes a sparse winner work. People often remember this as a neat sparsity story, then miss the uncomfortable implication for engineering: architecture, initialization, optimizer, and training path are entangled. If you are building models, this paper gives you a better mental model for why reruns differ, why pruning after training is not the same as training small from scratch, and why scale can help even when much of the final model looks unnecessary.
What to take away
- Resetting a pruned subnetwork to its original initialization is the key experiment.
- The sparse mask alone is not enough, the matching initial weights matter.
- The paper turns pruning from mere compression into evidence about how training searches.
Reads with
- Dropout: A Simple Way to Prevent Neural Networks from Overfitting
dropout trains many implicit subnetworks, lottery tickets ask which fixed subnetwork was trainable
- Understanding the difficulty of training deep feedforward neural networks
winning tickets depend on the original random initialization, not just sparse architecture
- Understanding deep learning requires rethinking generalization
both show overparameterization changes training dynamics rather than merely adding capacity