On the difficulty of training Recurrent Neural Networks

Why this one

Read this after Long Short-Term Memory if you want the math behind why the gate was not just clever architecture. The useful idea here is that backprop through time is repeated multiplication by Jacobians, so sequence learning becomes a question about eigenvalues, norms, and curvature. People often remember this paper as the origin story for gradient clipping, but clipping is the least interesting part if you are building models. The better lesson is diagnostic: when training blows up or silently forgets early tokens, you should look for a dynamical system whose state either stretches or contracts information at every step. That framing survives RNNs. It shows up again in residual nets, normalization, state space models, and long context attention. Spend the evening here if you want vanishing gradients to stop being folklore and start being a thing you can see in the algebra.