Playing Atari with Deep Reinforcement Learning
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, et al.2013NeurIPS Deep Learning Workshop 2013
Read it on arxiv.org(opens in a new tab)Why this one
Read this after ImageNet Classification with Deep Convolutional Neural Networks if you want to see convnets stop being image classifiers and start being controllers. The useful idea is not just raw pixels in, joystick actions out. It is the engineering trick that makes unstable bootstrapping barely trainable: keep old experience around, sample it out of order, and use it to break the feedback loop between the current policy and the current batch. People often remember DQN as a triumph of end-to-end learning, but the replay buffer is doing much of the practical work. The paper is worth your evening because it shows deep RL at the moment it becomes a system you can build, debug, and improve. Mastering Chess and Shogi by Self-Play and Proximal Policy Optimization Algorithms both make more sense after you have seen this simpler, rougher machine.
What to take away
- Experience replay turns correlated gameplay into reusable training data for Q-learning.
- A single convnet architecture was used across games, which made the result feel like a method, not a demo.
- The network predicts action values, so choosing an action is just an argmax over learned future reward estimates.
Reads with
- ImageNet Classification with Deep Convolutional Neural Networks
shows the convnet machinery dqn turns from perception into control
- Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
takes deep value learning from games to self-play search at superhuman scale
- Proximal Policy Optimization Algorithms
is the later stabilization path when replayed q-learning is not the right fit