Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
David Silver, Thomas Hubert, Julian Schrittwieser, et al.2017arXiv 2017
Read it on arxiv.org(opens in a new tab)Why this one
Read this after Playing Atari with Deep Reinforcement Learning. Atari showed that a neural net could learn behavior from reward, but AlphaZero shows the cleaner engineering pattern: use search at decision time, use the search results as training targets, then make the model strong enough that tomorrow's search starts from a better prior. People often remember this paper as magic self-play, or as proof that handcrafted knowledge is dead. That misses the useful lesson. The rules still define the world, Monte Carlo tree search still supplies the improvement operator, and the neural net is trained to imitate a stronger version of itself. If you are building agents, planners, or evaluators, this is the paper that makes reinforcement learning feel less like gradient folklore and more like an iterative systems loop.
What to take away
- The policy network does not just choose moves, it guides which branches the search should spend compute on.
- The value head replaces handcrafted evaluation, turning leaf positions into trainable judgments.
- Self-play works here because the environment is exact, cheap to simulate, and has a clear win condition.
Reads with
- Playing Atari with Deep Reinforcement Learning
atari is the reward-learning baseline alphazero turns into search-guided self-improvement
- Proximal Policy Optimization Algorithms
shows the other dominant rl engineering pattern, optimizing a policy without tree search
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
reuses the self-improvement loop idea for reasoning traces instead of game positions