Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun2015ICCV 2015
Read it on arxiv.org(opens in a new tab)Why this one
If you have ever copied `kaiming_normal_` without knowing why, spend an evening here. The durable idea is not PReLU or the ImageNet headline, it is that depth turns small variance mistakes into broken signal propagation. With ReLUs, half the activations vanish in expectation, so Xavier style initialization is no longer neutral. He initialization is the fix: choose weight variance so forward activations and backward gradients do not quietly shrink or explode before learning has a chance. People often treat initialization as a cosmetic default, below optimizers and architectures in importance. This paper shows it is part of the model's math. Read this after Understanding the difficulty of training deep feedforward neural networks, then before Deep Residual Learning for Image Recognition. It is the bridge from vague training instability to a concrete calculation you can use in your own network.
What to take away
- For ReLU layers, preserving variance means scaling weights by fan-in with a factor of 2, not using Xavier unchanged.
- Initialization affects both forward activations and backward gradients, so bad variance can kill training before the optimizer matters.
- PReLU is less important than the method: model the activation's effect on signal scale, then initialize from that calculation.
Reads with
- Understanding the difficulty of training deep feedforward neural networks
he initialization is the relu specific correction to xavier's variance argument
- Deep Residual Learning for Image Recognition
stable rectifier initialization helps make the much deeper resnet stack trainable
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
both attack depth induced signal drift before the optimizer can learn around it