Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Rafael Rafailov, Archit Sharma, Eric Mitchell, et al.2023NeurIPS 2023
Read it on arxiv.org(opens in a new tab)Why this one
Most engineers first meet RLHF as a pile of moving parts: collect rankings, train a reward model, run policy optimization, hope nothing drifts. The useful idea here is that preference tuning can be written as a much simpler supervised objective if you start from the right view of the model. The paper turns a pairwise human preference into a direct update on the policy, with the reference model acting as the anchor that keeps the tuned model from wandering too far. What people get wrong is treating DPO as magic alignment sauce. It is really a clean loss function for one specific kind of data: chosen versus rejected outputs. Read it because it makes post-training feel like something you can implement, inspect, and debug, instead of a ritual involving a separate reward model and a fragile RL loop.
What to take away
- DPO trains directly on chosen and rejected responses without fitting a separate reward model first.
- The reference model is not optional, it is the constraint that limits how far preference tuning can move the policy.
- The loss is useful because it turns preference learning into ordinary gradient descent over paired examples.
Reads with
- Training language models to follow instructions with human feedback
it is the reward-model-plus-ppo pipeline that dpo collapses into one loss
- Proximal Policy Optimization Algorithms
dpo removes the policy-optimization step that made rlhf fragile to run
- Constitutional AI: Harmlessness from AI Feedback
ai feedback supplies the same chosen-versus-rejected data that dpo can train on