Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun2015CVPR 2016
Read it on arxiv.org(opens in a new tab)Why this one
The observation that opens this paper is more interesting than the fix: past a certain depth, adding layers made training error go up, not down. That is not overfitting - a deeper network can always represent a shallower one by making the extra layers the identity - so it is an optimisation failure, and naming it that way is what leads to the answer. Let each block learn a residual and add the input back, so the identity is what you get for free rather than what you have to learn. Skip connections are now in essentially every deep architecture including the transformer, and the reason is the one in this paper: they give the gradient a clean route backwards through a very long stack.
What to take away
- Diagnose before you fix. 'Deeper is worse' was an optimisation problem wearing a capacity problem's clothes.
- The residual stream in a transformer block is this idea, unchanged.
- Identity mappings are cheap and load-bearing. Notice them in every architecture diagram from here on.
Reads with
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
resnets rely on it to make very deep convolutional stacks trainable
- Attention Is All You Need
transformers keep the residual path as the default route through depth
- Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
the initialization piece that made rectifier based deep vision nets easier to optimize