What a fixed learning rate actually does to four optimizers

Every model you train, from a two-parameter regression to a trillion-parameter transformer, is fit the same way: take a step opposite the gradient, and repeat until the loss stops moving. The rule is one line of algebra. Almost everything interesting about optimization happens in how that one line behaves on a real loss surface, and that behavior is easiest to see, not derive.

The lab above puts three surfaces under four optimizers. Drag the starting point, switch surfaces, and watch what a fixed learning rate does to each method. That contrast is the fastest way to build the intuition this article tries to write down.

The update rule and why it works locally #

Gradient descent minimizes a differentiable loss $L(\theta)$ by repeating

The justification is a first-order Taylor expansion: near $\theta_t$, $L(\theta_t - \eta g) \approx L(\theta_t) - \eta \lVert g \rVert^2$ for small $\eta$, so moving against the gradient decreases the loss — provided $\eta$ is small enough that the linear approximation still holds. Every optimizer on this page is a variation on that one idea: reshape the step, not the objective.

The bowl surface in the lab, $f(x, y) = x^2 + y^2$, is the case where the first-order story is the whole story. Curvature is identical along both axes, so a single learning rate serves both directions equally, and plain gradient descent converges directly to the minimum. Push the learning rate past roughly $1.0$ on this surface and you'll see it overshoot and diverge — the same instability that shows up as NaN loss in a real training run, just isolated down to two dimensions.

Curvature imbalance is where plain descent breaks #

Switch to the ravine: $f(x, y) = x^2 + 25y^2$. The two axes now have a curvature ratio, or condition number, of 25. A learning rate that safely descends along $x$ overshoots along $y$, and a learning rate calm enough for $y$ crawls along $x$. Plain gradient descent has no way to reconcile the two — it uses one scalar $\eta$ for every coordinate. The lab's default rate is right on that edge: watch the gradient-descent trace and $y$ overshoots a little further on every step — $0.95 \to -1.43 \to 2.14 \to -3.21 \to 4.81\dots$ — doubling in magnitude each time until the point is flung off the grid, even though the same step size is calmly walking $x$ toward zero the whole time. Nothing about the update rule changed between steps; the step size that was safe for one axis was never safe for the other.

This is the practical reason a raw loss landscape matters more than its value at the current point. Second-order curvature — the Hessian — determines how a fixed step size behaves in every direction at once, and real loss surfaces are rarely isotropic. A million-parameter network has a Hessian with a spectrum spanning many orders of magnitude; the ravine is that phenomenon in a form you can watch converge or fail to.

Momentum accumulates, it doesn't accelerate on command #

Momentum keeps a running velocity and lets the gradient nudge it rather than replace it:

With $\beta = 0.9$, roughly the last ten gradients are averaged into the current step, weighted toward recent ones. On the ravine at the lab's default learning rate, this is the difference between surviving and not: where plain gradient descent's step size is already past the stability limit and explodes, momentum's velocity term acts like a low-pass filter on the sign-flipping steep-axis gradient, and it survives — but survives loudly. Watch its trace: $x$ swings from $2.7$ past zero to $-1.6$, back past zero to $0.9$, and keeps crossing before it settles, four oscillations before the ensemble calms down around round 45. That overshoot is velocity built up while descending steeply, still carrying the point forward after the local gradient would have said "slow down." Momentum is not free stability; it converts one failure mode (divergence) into a milder one (ringing), and the same accumulated velocity that rescues it here is what would make it overshoot a sharp turn in a differently-shaped valley.

That is the entire mechanism — there is no separate "acceleration" term, only a longer effective memory of gradient direction, applied the same way regardless of what the surface looks like underneath it.

RMSProp rescales each coordinate by its own history #

RMSProp keeps a decaying average of squared gradients per coordinate and divides by its square root:

This is a direct answer to the ravine problem: a coordinate with large, persistent gradients gets its effective step size shrunk; a coordinate with small gradients gets its step size grown. At the same learning rate where plain gradient descent diverges outright, RMSProp converges — its $y$-step is throttled almost immediately, while its $x$-step keeps moving at close to the raw learning rate. That per-coordinate rescaling, done without ever computing or storing a Hessian, is the entire re-conditioning effect.

It's worth being precise about what that rescaling actually buys here, because it's easy to overstate: RMSProp doesn't converge faster than an untroubled gradient descent would on this surface, it converges at all where plain descent can't. Its $x$-progress is capped near the learning rate itself once the per-coordinate normalization kicks in, so a rate tuned to keep the steep axis stable makes the shallow axis move about as slowly as gradient descent's own $x$-steps did before it diverged — normalizing curvature away doesn't make the surface flatter, it makes every direction move at the speed of the most cautious one.

The cost is in $\epsilon$ and in what happens once $s_t$ has been small for a while: a coordinate that has been quiet accumulates a large effective step, so a late, large gradient in that direction can produce an outsized update. This is a real failure mode in sparse-gradient training (embeddings, attention over long sequences), not just an artifact of the toy surfaces here.

Adam combines both, with a correction that matters early #

Adam keeps both a momentum term and an RMSProp-style variance term, and corrects both for the fact that they start at zero:

The bias correction is not a footnote. Without it, $m_0 = v_0 = 0$ means the first several updates are shrunk toward zero exactly when you want them closest to a plain, informative gradient step — an easy bug to introduce if you ever hand-roll Adam instead of trusting a library implementation. With correction, Adam behaves like momentum with per-coordinate rescaling from step one, which is why it is a reasonable default on unfamiliar loss surfaces: it degrades gracefully whether the problem looks more like a ravine or more like a valley.

That default status has limits worth stating plainly, in the spirit of not overselling a tool because it is convenient. Adam's adaptive step sizes can converge to sharper minima than SGD with momentum on some architectures, and the generalization gap this produces on image classifiers is documented, not folklore. AdamW — decoupling weight decay from the adaptive step rather than folding it into the gradient — closes part of that gap and is the version worth reaching for by default now, even though the lab above implements the original coupled form for clarity.

Reading the valley: where momentum earns its keep #

The valley surface is a scaled Rosenbrock function, $f(x, y) = (1-x)^2 + 10(y - x^2)^2$, the standard stress test for optimizers because its minimum sits inside a curved, narrow trough rather than a straight one. The lab's default learning rate here is conservative, chosen to keep every method stable against the steep initial gradient near the starting point — and at that rate, run the default 120 rounds and look at where each method ends up: gradient descent, RMSProp, and Adam are all still clustered near the entrance to the valley, having barely covered a third of the distance to the minimum. Momentum alone gets there, reaching a loss an order of magnitude lower than the rest.

That is not RMSProp or Adam malfunctioning — their per-coordinate rescaling has nothing to correct here, because the valley's curvature isn't axis-aligned the way the ravine's was; normalizing each coordinate's step separately doesn't help you turn. What actually moves a point through a curved, narrow trough is sustained velocity in a consistent direction, accumulated over many steps — exactly the mechanism the momentum section above described, and exactly why momentum methods (and their better-known descendant, Nesterov accelerated gradient) were originally motivated by problems shaped like this one, not by the ravine. Watching all four methods stall except the one built to carry velocity through a curve is a more honest demonstration of "why momentum exists" than any bowl or ravine can give: on a straight ravine, momentum is optional insurance against a bad learning rate; on a curved valley, at a learning rate conservative enough for everything to stay stable, it's the only one that arrives.

The broader lesson generalizes past this toy function: a per-coordinate adaptive method's rescaling is diagonal by construction, so it cannot correct curvature that runs across coordinates, only curvature that differs between them. Full second-order methods (Newton, L-BFGS) or trust-region methods model that cross-coordinate curvature directly, at the cost of computing or approximating a Hessian that is intractable at neural-network scale — which is why, in practice, "Adam has stalled" is often information about the geometry a per-coordinate method structurally cannot see, not a sign to raise the learning rate and hope.

What the toy surfaces don't show #

Every trajectory in this lab is a full-batch, noise-free gradient. Real training computes gradients on mini-batches, and that stochasticity does two things none of these three surfaces capture: it adds variance that momentum and Adam's moment estimates are partly designed to smooth, and it can help a noisy step escape a sharp local minimum that a noiseless step would settle into. The bowl, ravine, and valley are here to make curvature legible, not to stand in for a training run — treat the qualitative comparison between optimizers as the transferable part, and the exact step counts as specific to these three functions.