Gradient descent on a surface

Condition number 12. The gradient points across the valley rather than along it, which is what momentum fixes and what makes a single learning rate a compromise.

This surface is a quadratic, so the stability threshold is exact rather than a rule of thumb: plain gradient descent converges while lr < 2/L, and here L = 24, so the boundary is 0.0833. You are at 0.0200, below it. SGD ended at a loss of 0.0193. Walk the slider across 0.0833 and watch the prediction and the behaviour change together.

At a learning rate of 0.0200 over 60 steps on the Elliptical bowl (ill-conditioned), the best final loss is 1.21e-5 from Momentum.

All three start at the same point and see the same gradients. Where they end up differs only in how they turn a gradient into a step, which is the entire content of an optimiser.

The last two columns are the whole point. SGD multiplies both gradients by the same learning rate, so a coordinate with a gradient twelve times larger gets a step twelve times larger. Adam divides each coordinate by the square root of its own running squared gradient, so the two steps come out close to the same size whatever the gradients are. That is what "adapts per parameter" means, and it is these two numbers being different from each other in a way SGD's never are.

Real: every gradient is the analytic derivative of the formula shown, not a finite difference. All three update rules are the textbook ones, Adam's bias correction included and visible in the table. The stability bound on the quadratic is exact, and the divergence step beside it is measured by running until the loss stops being finite.

Standing in, and this is the big one: there is no data here. This is full-batch descent on a closed-form function, so the gradient is exact every time and there is no noise at all - the "S" in SGD is missing, and half of what momentum is for in practice is smoothing exactly that noise.

Also standing in: two dimensions. A real network has millions of parameters and a surface nobody can draw, and the mechanics generalise while the geometry does not - in high dimensions saddle points vastly outnumber local minima, which is the most important fact about a real loss landscape and the one a picture like this cannot show you. Momentum here is heavy ball rather than Nesterov, and there is no weight decay, no learning-rate schedule and no gradient clipping.