What each optimizer is doing
All six read the same gradient at the same point. The difference is entirely in what they do with it.
| Method | Update | The idea |
|---|---|---|
| SGD | x ← x − η g | Step straight downhill. In a narrow valley it bounces off the walls instead of moving along the floor. |
| Momentum | v ← βv − ηg; x ← x + v | Accumulate a velocity. Perpendicular bounces cancel out, the consistent downhill component adds up. |
| Nesterov | v ← βv − η∇f(x + βv) | Momentum, but the gradient is read where you are about to be. It brakes earlier going into a turn. |
| AdaGrad | s ← s + g²; x ← x − ηg/√s | Divide by the running sum of squared gradients. The step size only ever shrinks, which eventually stalls it. |
| RMSProp | s ← ρs + (1−ρ)g² | AdaGrad with a forgetting factor, so the step size can recover. |
| Adam | m, v with bias correction | Momentum and RMSProp together. Usually the safe default — but see below. |
“Adam is always best” is not what happens
Run the Rosenbrock surface at the default settings and watch the ranking. Adam moves confidently at first, because normalising by √v makes it take near-constant-size steps regardless of how flat the floor is — and then that same normalisation stops it from accelerating along the long, gentle valley floor where there is nothing to be cautious about. Momentum, which does accelerate there, tends to arrive first.
The same race written up with a 3-D surface and NumPy code is here in Japanese. The conclusion there was the same: Momentum won, and “just use Adam” is advice about robustness, not about speed.
The learning rate is shared on purpose
One slider drives all six. That is not a simplification — it is the comparison. Adam, RMSProp and AdaGrad rescale the gradient before stepping, so at a learning rate where SGD explodes they are still stable, and at a learning rate where SGD is fine they crawl. Sweep the slider across its range on the ravine surface and you can watch the stable interval for each method: SGD’s is the narrowest, Adam’s is the widest, and that width is the actual reason Adam is the default in most frameworks.
Try the saddle
The saddle surface is flat at the origin with two minima above and below. Drop the optimizers near the middle. Plain SGD sits there almost motionless — the gradient is tiny in every direction — while the adaptive methods, which divide by the magnitude of recent gradients, treat a small consistent gradient as a large one and leave quickly. In high-dimensional networks saddles vastly outnumber local minima, so this behaviour matters more than the cartoon of “getting stuck in a local minimum”.
And the local minima
Rastrigin and the two-well surface are the ones where gradient descent simply cannot help you: every method here only ever goes downhill, so whichever basin you start in is the one you finish in. Turning up Gradient noise is the cheapest fix — enough jitter and a runner can be shaken out of a shallow well into a deeper one. That is the seed of simulated annealing, which in a run of 200 random starts reached the global optimum 200 times out of 200, where plain gradient descent reached it zero times.