Understanding the difficulty of training deep feedforward neural networks

Why this one

Read this after The Matrix Calculus You Need For Deep Learning and before Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. The useful idea is that a deep net is not just a pile of nonlinear functions, it is also a signal pipeline, and bad initialization quietly turns that pipeline into amplification or extinction. People often remember Xavier initialization as a formula to paste into code. That misses the point. The formula is the consequence of asking a practical engineering question: what scale keeps activations and backpropagated gradients from changing wildly as they cross layers? Once you see that, initialization stops being folklore and becomes a constraint you can reason about when you change activations, widths, or architecture. Worth your evening because it gives you the first real diagnostic lens for why a network that is mathematically expressive still refuses to train.