Training language models to follow instructions with human feedback
Long Ouyang, Jeff Wu, Xu Jiang, et al.2022NeurIPS 2022
Read it on arxiv.org(opens in a new tab)Why this one
A base language model predicts text. An assistant has to follow an instruction, refuse some requests, prefer useful answers over plausible junk, and keep doing that across prompts it has never seen. The durable idea here is not just RLHF as a recipe, but the separation of three jobs: teach the model the task format, learn a human preference signal, then optimize against that signal without letting the model drift into reward hacking. People often talk about this paper as if human feedback magically adds values to a model. It does not. It turns messy judgments into a training interface, with all the usual product tradeoffs around data quality, labeler instructions, distribution shift, and evaluation. If you are fine-tuning models for real users, this is the paper that explains why a smaller instructed model can beat a larger raw one in practice.
What to take away
- Supervised instruction tuning gives the model the interface before preference optimization begins.
- The reward model is trained on pairwise human rankings, not absolute scores or gold answers.
- PPO improves helpfulness, but the paper tracks regressions on public NLP tasks to catch alignment tax.
Reads with
- Language Models are Few-Shot Learners
it is the raw base model behavior instruction tuning is trying to turn into a product
- Proximal Policy Optimization Algorithms
it is the policy optimizer used to push the model toward the learned reward signal
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model
it removes the separate reward model and rl loop while keeping the preference data interface