Deep Double Descent: Where Bigger Models and More Data Hurt
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, et al.2020ICLR 2020
Read it on arxiv.org(opens in a new tab)Why this one
Read this after Understanding deep learning requires rethinking generalization and before Scaling Laws for Neural Language Models. The old bias versus variance picture says bigger models overfit more, so capacity should be rationed. Deep learning made that advice strangely unreliable: once a model can fit the training set exactly, making it even larger can improve test error again. The trap is to treat double descent as permission to ignore overfitting. It is not. It says the dangerous region is often near the interpolation threshold, and that model size, data size, training time, label noise, and optimization all change where that threshold sits. For a working engineer, this paper is useful because it replaces a childish rule with a map: if validation gets worse when you add data or train longer, you may not be seeing a bug, you may be crossing a complexity peak.
What to take away
- Worst test error can occur near the point where the model first fits the training set perfectly.
- More data can hurt when it moves the task into a harder effective complexity regime.
- Training longer can act like increasing model complexity, not just like reducing loss.
Reads with
- Understanding deep learning requires rethinking generalization
sets up why classical capacity control failed to explain deep nets
- Scaling Laws for Neural Language Models
continues the story once overparameterized models enter smoother scaling regimes
- Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets
shows another case where more training after overfit can improve generalization