Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets

Why this one

Read this after Understanding deep learning requires rethinking generalization and Deep Double Descent: Where Bigger Models and More Data Hurt. The useful shock here is not that networks can memorize, you already know that. It is that memorization and generalization can be separated in time by a long, boring stretch where every normal training dashboard tells you to stop. On these toy algorithmic tasks, the model first fits the examples, then much later settles into the rule. People often turn grokking into folklore about magic phase changes. Treat it more practically: optimization path matters, weight decay matters, data size matters, and validation curves can lie if your patience encodes the wrong theory of learning. It is worth your evening because it gives you a clean mental counterexample to the lazy rule that overfitting means learning is over.