The Llama 3 Herd of Models
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al.2024arXiv 2024
Read it on arxiv.org(opens in a new tab)Why this one
Read this after Scaling Laws for Neural Language Models, Training Compute-Optimal Large Language Models, and Training language models to follow instructions with human feedback. Most model papers show you the trophy. This one shows more of the factory: data filtering, scaling choices, infrastructure constraints, post-training recipes, safety models, evals, and the compromises that turn a pretrained transformer into a product model. The common mistake is to read Llama 3 as a leaderboard entry or an open weights announcement. Its real value is operational. It teaches that frontier quality is not one trick, it is thousands of boring decisions made consistently, with measurement loops around every stage. If you are building your own smaller model, fine-tuning one, or just trying to understand why training runs fail in expensive ways, this is worth an evening because it gives you a map of the whole pipeline.
What to take away
- The paper makes data curation feel like model architecture: filtering, deduping, mixing, and quality scoring drive the final model.
- Post-training is treated as a production pipeline, not a single RLHF step, with separate work for chat, tools, safety, and refusals.
- The 405B model matters less as a size record than as a teacher, evaluator, and data generator for the rest of the herd.
Reads with
- Scaling Laws for Neural Language Models
gives the empirical scaling frame behind llama 3's data and compute choices
- Training Compute-Optimal Large Language Models
explains why llama 3 spends so much of the factory on token quality and quantity
- Training language models to follow instructions with human feedback
is the post-training template that turns a base transformer into an assistant