The Matrix Calculus You Need For Deep Learning
Terence Parr, Jeremy Howard2018arXiv 2018
Read it on arxiv.org(opens in a new tab)Why this one
Most engineers can use PyTorch long before they can explain why the gradients have the shapes they do. This is the bridge. Parr and Howard give you the small set of matrix derivative rules that make backprop feel like ordinary code instead of notation theater. What people get wrong is treating matrix calculus as a separate math subject, full of conventions to memorize. Here it is a debugging tool: if you know which object is scalar, vector, or matrix, and you track shapes carefully, the chain rule becomes mechanical. Read this before Adam: A Method for Stochastic Optimization, Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, or The Matrix Calculus You Need For Deep Learning's natural companion in practice, any neural network from scratch exercise. It is worth your evening because it turns autograd from magic into something you can inspect, derive, and distrust when necessary.
What to take away
- Shape checking is the main discipline: most gradient mistakes show up as impossible matrix dimensions.
- The paper teaches numerator-layout derivatives, so be consistent when comparing formulas from other sources.
- Backprop is just the chain rule over matrix expressions, not a special neural network trick.
Reads with
- Adam: A Method for Stochastic Optimization
turns the gradients you can now derive into practical parameter updates
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
its backward pass is a compact stress test for shape tracked chain rule
- On the difficulty of training Recurrent Neural Networks
uses gradient flow to explain why recurrent nets forget or explode