Neural Ordinary Differential Equations
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, David Duvenaud2018NeurIPS 2018
Read it on arxiv.org(opens in a new tab)Why this one
Read this after Deep Residual Learning for Image Recognition and The Matrix Calculus You Need For Deep Learning. ResNets made it natural to view a layer as a small update to a state. This paper asks what happens if you take that idea literally and let depth become time. The useful lesson is not that every model should call an ODE solver, because most production models should not. The lesson is that numerical methods, memory, gradients, and model architecture are the same engineering problem once your network is a dynamical system. People often remember the adjoint trick as free memory. It is not free, it trades stored activations for recomputation and solver error. Read it because it gives you a clean mental bridge from calculus to deep learning systems, and because many later ideas in flows, diffusion, continuous time models, and efficient backprop make more sense after this one.
What to take away
- A residual block can be read as one Euler step of a learned dynamical system.
- The adjoint method saves activations by solving another ODE backward, but accuracy and stability now matter.
- Continuous normalizing flows use ODE structure to make density changes tractable.
Reads with
- Deep Residual Learning for Image Recognition
residual blocks are the discrete update that this turns into continuous time
- The Matrix Calculus You Need For Deep Learning
the adjoint method is backprop through an ODE written in matrix calculus
- Denoising Diffusion Probabilistic Models
diffusion samplers reuse the same dynamics, solver, and gradient tradeoffs