Scaling Laws for Neural Language Models
Jared Kaplan, Sam McCandlish, Tom Henighan, et al.2020arXiv 2020
Read it on arxiv.org(opens in a new tab)Why this one
Read this after A Mathematical Theory of Communication if cross-entropy still feels like a metric someone picked from a menu, and before Training Compute-Optimal Large Language Models, which revises the budget advice with better data scaling. The useful idea here is not “bigger is better.” It is that loss, parameters, data, and compute can behave regularly enough that you can make engineering plans before spending the money. People often quote this paper as destiny, as if a fitted line on log-log axes explains intelligence. Do not read it that way. Read it as a discipline for experiments: vary one resource, hold the others honest, measure loss, then ask which bottleneck you are actually paying for. The surprising part is how much architectural taste recedes once scale is controlled. For a builder, that is liberating and brutal: your clever trick has to beat a power law and a purchase order.
What to take away
- Fit scaling curves on loss, not vibes, then use them to choose model size, data size, and training compute.
- The paper’s original optimum favors larger models trained short, which Chinchilla later corrects toward more data.
- Architecture details matter less here than whether parameters, tokens, and compute are the active bottleneck.
Reads with
- A Mathematical Theory of Communication
cross entropy and bits per token come from shannon's coding view of prediction
- Training Compute-Optimal Large Language Models
revises kaplan's compute advice by showing data was the hidden bottleneck
- Language Models are Few-Shot Learners
turns the scaling curve into a product bet on few shot behavior