Proximal Policy Optimization Algorithms
John Schulman, Filip Wolski, Prafulla Dhariwal, et al.2017arXiv 2017
Read it on arxiv.org(opens in a new tab)Why this one
Read this after Playing Atari with Deep Reinforcement Learning and before Training language models to follow instructions with human feedback. The useful idea is not that PPO is the best RL algorithm. It is that you can let a policy improve itself while putting a leash on how far each update is allowed to move it. That leash, the clipped objective, is why PPO became the default workhorse for messy environments and later for the RL step in RLHF. People often treat it as a bag of tricks: advantage estimates, value loss, entropy bonus, clipping, minibatches. The paper is worth your evening because it turns those tricks into one engineering instinct: when your training data comes from the model you are changing, optimization stability is a product feature, not a nicety. If InstructGPT or DeepSeek-R1 feels like magic, PPO is one of the gears you need to see.
What to take away
- PPO reuses rollout data for several minibatch updates without letting the new policy drift too far from the old one.
- The clipped probability ratio is the core safeguard, not an incidental implementation detail.
- RLHF uses this pattern when a language model is optimized against a reward model while staying near its starting policy.
Reads with
- Playing Atari with Deep Reinforcement Learning
shows the earlier value-learning route before ppo makes policy updates the workhorse
- Training language models to follow instructions with human feedback
uses ppo as the stability mechanism for the reinforcement learning step in rlhf
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
scales the same constrained policy-improvement instinct into reasoning-focused llm training