Skip to content
Klay

All papers

ML Mathematics18 min read

Understanding deep learning requires rethinking generalization

Chiyuan Zhang, Samy Bengio, Moritz Hardt, et al.2017ICLR 2017

Read it on arxiv.org(opens in a new tab)
  • generalization
  • capacity
  • regularization

Why this one

Read this after No Free Lunch Theorems for Optimization, and before Deep Double Descent. The useful shock here is not that neural nets are big, it is that the usual story for why big models should not overfit stops working. Zhang and colleagues show that the same training recipe that learns real images can also memorize nonsense labels, so you cannot explain test accuracy by saying the architecture simply lacks the capacity to memorize. People often misread this as nihilism, or as proof that regularization is fake. It is neither. It says your mental model is missing the role of data structure, optimization bias, and the path SGD takes through an overparameterized space. If you are building models, this paper is worth an evening because it inoculates you against tidy but wrong explanations. It makes modern deep learning look less like classical curve fitting and more like a system whose behavior depends on the interaction between data, parameterization, and training dynamics.

What to take away

  • A modern convnet can fit random labels, so low training error alone says almost nothing about learned structure.
  • Explicit regularizers help in practice, but they are not the main reason these networks generalize.
  • Capacity bounds that ignore the data distribution and training process miss the central phenomenon.

Reads with