Layer Normalization
Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E. Hinton2016arXiv 2016
Read it on arxiv.org(opens in a new tab)Why this one
Read this after Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. Batch norm made normalization feel like a property of the training run, tied to whatever examples happened to share a mini-batch. The useful turn here is to make normalization a property of one example as it flows through one layer. That sounds small, but it is why the trick fits recurrent nets, variable batch sizes, and later the transformer stack without special casing inference. What people get wrong is treating layer norm as just a safer batch norm. It changes the failure mode: you give up batch statistics and instead control the scale of each token or hidden state locally. If you build deep networks, this paper is worth the evening because residual connections, attention blocks, and stable training all become easier to reason about once you see what is being normalized and what is learned back with gain and bias.
What to take away
- Layer norm computes mean and variance across features of one example, not across examples in a mini-batch.
- The same computation runs at train and test time, so inference has no moving averages to maintain.
- Its learned gain and bias restore representational freedom after forcing activations onto a stable scale.
Reads with
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
the contrast case shows why normalizing across a batch breaks for sequences
- Attention Is All You Need
transformer blocks rely on layer norm to keep residual attention stacks trainable
- On the difficulty of training Recurrent Neural Networks
the recurrent instability problem explains why per example normalization mattered