Training Compute-Optimal Large Language Models
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, et al.2022NeurIPS 2022
Read it on arxiv.org(opens in a new tab)Why this one
Big models did not get better just because someone made them bigger. The useful idea here is compute optimality: for a fixed training budget, there is a balance between parameters and data, and many early LLMs were too large and undertrained. Engineers often quote parameter count as if it were capacity, but this paper makes you ask the better question: how many tokens bought that capacity, and was the compute spent in the right place? Read it if you want a practical mental model for why a smaller model trained longer can beat a larger one, why dataset quality and repetition matter, and why training plans are budget allocation problems before they are architecture problems.
What to take away
- For a fixed compute budget, model size and training tokens must be chosen together.
- The paper argues many large language models were undertrained relative to their parameter count.
- Chinchilla's result makes token budget a first-class design variable, not a leftover detail.
Reads with
- Scaling Laws for Neural Language Models
sets up the empirical scaling curves that chinchilla revises with a new compute optimum
- Language Models are Few-Shot Learners
the undertrained large model that made chinchilla's parameter versus data tradeoff obvious
- The Llama 3 Herd of Models
shows compute optimal data scaling becoming standard practice in modern open model training