Dropout: A Simple Way to Prevent Neural Networks from Overfitting
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, et al.2014JMLR 2014
Read it on jmlr.org(opens in a new tab)Why this one
Read this after ImageNet Classification with Deep Convolutional Neural Networks and Understanding the difficulty of training deep feedforward neural networks. Dropout is often taught as a little trick: randomly turn neurons off so the model does not memorize. The better idea is stranger and more useful: you are training a huge family of thinned networks that share weights, then using one averaged network at test time. That ensemble view explains why the scaling rule matters, why dropout can hurt in the wrong place, and why it became less central once Batch Normalization and very large datasets changed the training recipe. If you are building models, this paper is worth an evening because it turns regularization from a checkbox into a design question: what dependencies are you trying to break, and what approximate ensemble are you really buying?
What to take away
- Dropout trains many subnetworks with shared weights, then approximates their average at inference.
- The test-time weight scaling is not cosmetic, it is the cheap ensemble approximation.
- Dropout is strongest when co-adaptation is the problem, not when capacity or optimization is the bottleneck.
Reads with
- ImageNet Classification with Deep Convolutional Neural Networks
dropout is one of the tricks that made alexnet train and generalize at imagenet scale
- Understanding the difficulty of training deep feedforward neural networks
initialization and dropout are paired answers to why deep nets fail to train cleanly
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
batch norm changed the training recipe that made dropout feel essential