Adam: A Method for Stochastic Optimization
Diederik P. Kingma, Jimmy Ba2014ICLR 2015
Read it on arxiv.org(opens in a new tab)Why this one
Adam is the default in every framework, which is exactly why it is worth spending an evening on rather than accepting. The method keeps two running averages per parameter - one of the gradient and one of its square - and divides one by the root of the other, so a parameter with consistently small gradients still gets a meaningful step. The bias-correction terms that look like clutter in the pseudocode are doing real work in the first few hundred steps, when both averages start at zero and would otherwise pull the step size to nothing. Read it after the moving-averages lesson; the entire algorithm is exponential moving averages applied twice, and it stops being mysterious the moment you see that.
What to take away
- Per-parameter step sizes are the point. That is what SGD does not give you.
- Beta1 and beta2 are memory lengths, not magic constants - 0.9 and 0.999 are roughly 10 and 1000 steps of history.
- If training is unstable in the first epoch, look at bias correction and warmup before you blame the learning rate.
Reads with
- Attention Is All You Need
shows adam becoming part of the transformer training recipe
- ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
shards adam's two moment vectors when optimizer state becomes the bottleneck
- LoRA: Low-Rank Adaptation of Large Language Models
reduces the number of tensors adam has to update during llm fine tuning