Training is optimization: w ← w − lr · ∇L, repeated a few million times. Every optimizer in this lesson is that one line plus an idea for walking downhill faster and more reliably.
Every weight is a coordinate, and the loss assigns a height to every combination. The gradient gives the local uphill direction — so one step goes the other way: w ← w − lr · ∇L. Repeat. That loop is training.
w ← w − lr · ∇LMomentum accumulates past gradients into a velocity, so a consistent downhill direction adds up while a zigzag cancels out. Adam tracks each weight's average gradient and average squared gradient, then divides — large gradients get small steps, small ones get large steps.
v ← β·v + g w ← w − lr·vA mini-batch gradient is the true gradient plus noise, and the noise shrinks like 1/√batch. On a real loss it can push the optimizer off a saddle point or out of a sharp minimum — which is also why the final answer often generalizes better.
batch 1 → noisy · batch 256 → smoothEvery optimizer is w ← w − lr · gradient plus one idea for walking downhill faster: momentum remembers where you were heading, Adam gives every weight its own step size, and a schedule shrinks the steps as you approach the bottom.
By the end you will be able to read the update rule in any optimizer, predict when a learning rate will oscillate or diverge, explain why mini-batch noise helps, and pick a starting optimizer, learning rate and schedule for a real training run.
Optimization means finding the inputs that make a function small. When the function is a loss and the inputs are weights, optimization is training — and the terrain it walks is called the loss landscape.
Collect every weight of a model into one long vector w. The loss L(w) assigns a single number to every possible w: how wrong the model is on the training data. That number is a height, so L is a surface over weight space, and training is the search for its lowest point:
Nobody can see the whole surface. A model with one million weights lives in a 1,000,001-dimensional space, and at each point the optimizer only gets the local slope — the gradient. Every strategy in this lesson is a different way to use that one local reading to get down the hill faster and more reliably.
Click anywhere to drop a start point, then step downhill. The path is computed live from w ← w − lr·∇L. Two valleys, one saddle, one rule.
This two-basin landscape is a simplified teaching model. A real network’s loss lives in millions of dimensions; researchers slice it along two random directions to draw pictures like this.
The classic benchmark valley is the Rosenbrock function: f(x, y) = (1 − x)² + 100(y − x²)². Its minimum sits at (1, 1) inside a long curved trough. Differentiate each term with the chain rule from Lesson 04.
The factor 100 is the whole story of why plain gradient descent zigzags here: the steep direction demands tiny steps, while the long shallow floor would reward big ones. A single learning rate has to compromise — unless momentum or Adam changes the picture, as the rest of the lesson shows.
Gradient descent computes a gradient, steps the other way, and does it again. Everything else in this lesson is a refinement of this one line — and a convex bowl is where you can see exactly when it works.
The rule is w ← w − lr · ∂L/∂w, applied to every weight at once. The partial derivative says which way to nudge that weight to make the loss bigger, so subtracting moves it the other way. The learning rate lr scales the step. Near a minimum the slope flattens, so the steps shrink on their own — the update rule has automatic braking built in.
A function with a single valley is called convex: any local minimum is the global one, and gradient descent is guaranteed to find it. A parabola is convex. Neural-network losses are not convex — they have many valleys, plateaus and saddles — but the convex bowl is still the right place to learn what the learning rate does, because there the behavior is completely predictable.
Pick a loss, a start and a learning rate. Every row applies the one update rule — next x = x − lr · f′(x) — and shows the arithmetic.
curvature 2 — safe lr < 1
| step | x | f(x) | f′(x) | next x |
|---|---|---|---|---|
| 0 | 4.0000 | 16.0000 | 8.0000 | 3.2000 |
Convexity is also why a loss like L(w) = (prediction − target)² for a linear model is easy: there is exactly one bottom, so stopping anywhere flat means you found the answer. For a deep network the same rule still runs, but “a flat spot” might be a saddle rather than a solution. That is a chapter-07 problem; the update rule itself is unchanged.
On the bowl f(x) = x², which learning rate makes the loss grow without bound?
Take the simplest loss, f(x) = x², whose slope is 2x. One step of gradient descent is x_new = x − lr·2x = (1 − 2·lr)·x. Every step multiplies x by the same factor, so the whole story lives in that factor.
Second worked example — curvature changes the limit. Take f(x) = 2x², slope 4x. One step is x_new = (1 − 4·lr)·x, so the safe range halves: 0 < lr < 0.5.
This is the deepest practical fact about learning rates: the steepest direction of a loss sets the largest safe step, not the average direction. That is why narrow valleys are hard — a step safe for the steep walls is tiny along the long floor.
The learning rate is the single most important hyperparameter in training. It has three regimes, and you can see all of them on one parabola.
Too small and you creep toward the answer for thousands of wasted steps. Too large and each step overshoots the valley; the loss bounces between the walls, and past a certain point it climbs without bound. There is no formula for the right value — you find it by experiment, starting from common defaults: 0.001 for Adam and 0.01 for SGD with momentum.
The safe limit is set by the curvature of the steepest direction, as the previous derivation showed. Real losses have different curvature in different directions, so the compromise is unavoidable — and that is the motivation for everything that follows: schedules shrink the steps late, momentum and Adam adapt them per direction and per weight.
Same bowl, same start, one slider. Push lr until convergence turns into oscillation, then into divergence. Watch the factor (1 − lr·curvature).
On x², lr = 0.5 lands on the minimum in one step, lr = 1 oscillates forever, and lr > 1 diverges. Now switch to 2x² — the safe limit halves to 0.5 because the curvature doubled.
A fixed learning rate is a compromise between fast early progress and careful late convergence. Schedules change it over time. The modern favorite is cosine annealing:
The half-cosine falls slowly at first, fastest in the middle, and flattens at the end — exactly the “big steps early, gentle landing late” shape you want. Warmup is the mirror trick at the start: for the first few hundred steps the learning rate ramps up linearly from 0, because a freshly initialized model has noisy gradient estimates and a full-size first step can wreck it.
Batch gradient descent uses every example for one exact step. SGD uses one. Mini-batch uses a handful — and that noisy estimate is what everyone actually trains with.
The gradient of a loss over a dataset is the average of the gradients on each example. Computing all of them gives the exact direction, but on a million-example dataset that is a million backward passes for a single step. Computing one gives a wildly noisy direction that is nonetheless correct on average — and costs one pass. A mini-batch of 32–256 examples splits the difference. Each full pass through the data is called an epoch.
| variant | gradient computed on | gradient quality | speed per step | noise |
|---|---|---|---|---|
| Batch gradient descent | the entire dataset | exact | slow | none |
| Stochastic GD (SGD) | 1 random example | very noisy | fastest | high |
| Mini-batch SGD | 32–256 random examples | good estimate | balanced | moderate |
The noise scales like 1/√batch: a batch of 256 is 16× quieter than a single example, but still not exact. In practice “SGD” almost always means mini-batch SGD. The noise is not a defect to be minimized away — it is a feature that shakes the optimizer out of places where a perfectly smooth descent would sit forever.
The same start, two update streams: one uses the exact gradient, one uses a mini-batch estimate (true gradient + random error that shrinks as the batch grows). Run it and compare the paths.
Noise does not make the average step bigger — it adds a random jolt on top. That jolt is what shakes an optimizer out of a shallow dip or off a saddle, and it is why “SGD” in practice means a noisy mini-batch walk, not exact gradient descent.
Why can random mini-batch noise actually help training?
Plain descent has no memory: it feels only the slope at its feet. In a narrow valley that means zigzagging across the walls while creeping along the floor. Momentum fixes the memory problem.
Momentum keeps a running velocity that accumulates past gradients, then steps by the velocity instead of the raw gradient:
In a long narrow valley the across-the-valley gradients alternate sign every step, so they cancel in the running sum. The along-the-valley gradient keeps the same sign, so those contributions add up. The result is a path that damps the zigzag and accelerates down the floor — visible in the difference between these two walks.
Unroll the update. v₁ = g₁, then v₂ = β·g₁ + g₂, then v₃ = β²·g₁ + β·g₂ + g₃. In general:
Higher β means longer memory: smoother paths and more acceleration, but slower reaction when the valley turns. β = 0.9 is the standard compromise, and it is the default in the momentum optimizer you met in Lesson 04’s exercise.
With β = 0.9 and a gradient that stays constant at g, the momentum step is about:
Some weights receive huge gradients every step; others receive tiny ones. One global learning rate is wrong for both — so Adam divides each weight’s gradient by a running measure of how big that weight’s gradients have been.
The first adaptive idea is RMSProp: keep a running average of the squared gradient per weight and divide the update by its square root. A weight with habitually large gradients gets divided by a large number (small effective steps); a weight with small gradients gets divided by a small number (large effective steps).
Adam is literally momentum’s first moment plus RMSProp’s second moment, with one fix for the cold start. On modern transformers it is the default, usually as AdamW — Adam with decoupled weight decay.
All three start at the same point and run their real update rules. The table counts the steps each needed to get below a loss threshold.
| optimizer | loss now | steps to L < 1 | steps to L < 0.1 | steps to L < 0.01 |
|---|---|---|---|---|
| SGD | 4.000e+0 | not in budget | not in budget | not in budget |
| Momentum β=0.9 | 4.000e+0 | not in budget | not in budget | not in budget |
| Adam | 4.000e+0 | not in budget | not in budget | not in budget |
On the Rosenbrock valley Adam reaches moderate losses first because its steps are ~lr per weight regardless of gradient size. On the long narrow quadratic a single accumulated direction is exactly what momentum wants, so momentum wins there. On the final stretch, momentum (and SGD with a tuned schedule) can catch up or beat Adam on accuracy — which is why the “best” optimizer depends on what you are optimizing for.
Equalized steps. Suppose a weight’s gradient is steadily g. Then m̂ ≈ g and v̂ ≈ g², so
Bias correction. m and v both start at zero, so early on they are dragged toward zero. Unroll m with a steady gradient g:
In short: RMSProp rescales, momentum smooths, and bias correction makes the first steps honest. Without it, Adam’s early steps are mis-scaled for thousands of steps, because the second moment needs so long to warm up.
Why does Adam divide the update by √v̂ (the running average of squared gradients)?
In a one-dimensional curve, a local valley can trap you. In a million-dimensional loss, the traps are almost always saddle points — and the escape kit is already in your hands: momentum and noise.
A saddle point is a point where the gradient is zero but the loss curves upward in some directions and downward in others — a minimum along one axis, a maximum along another. For a random critical point in N dimensions to be a true local minimum, every one of the roughly N directional curvatures must be positive. Randomness makes that exponentially unlikely: flat, saddle-like plateaus dominate, and the local minima that do exist usually have loss values close to the global one.
The practical failure is not “stuck forever in a bad valley” but “stalled on a plateau where the gradient is nearly zero.” Momentum carries you across such regions using accumulated speed; mini-batch noise gives you a random push off them. That is why the default toolkit pairs the two.
There is one more empirical fact worth keeping: sharp minima — narrow, steep-walled dips — tend to generalize worse to new data than flat minima. The noise of SGD with momentum makes sharp minima harder to settle into, which is one reason it often edges out Adam on final test accuracy even when Adam trains faster.
The loss is x² − y², a saddle at the origin: a bowl along x, a hill along y. Start on the y = 0 ridge and watch a noiseless optimizer stall; start just off it and watch momentum and Adam pull away.
In a million-dimensional loss there are far more saddle directions than valleys. That is why the field stopped fearing local minima and started respecting saddles — and why momentum and mini-batch noise are standard equipment.
Answer before you look. The learning-rate, Adam and mini-batch questions are the ones that come up every time you tune a real training run.
0 / 5 answered · 0 correct
01What does “optimization” mean in the context of training a neural network?
02What happens if the learning rate is too large during gradient descent?
03How does Adam differ from vanilla gradient descent?
04Why is the noise in mini-batch SGD considered beneficial rather than just a nuisance?
05What does a cosine annealing learning-rate schedule do?
Click a card to swap the lazy description for what it actually means.
Four problems, all runnable in the labs above. Try first; a worked answer is one click away.
The numbers below are the final loss after 5000 steps: lr = 0.0001 → 2.10 (crawling: not enough step) lr = 0.0005 → 0.0408 (converging) lr = 0.001 → 0.00363 (converging, best of the list) lr = 0.005 → diverges (NaN: the steep walls win) lr = 0.01 → diverges (NaN) So 0.001 is the largest listed rate that converges. The steep direction of the valley has curvature ~200 (the y-term), and its safe step limit is roughly 2/curvature ≈ 0.01 — but the x-direction coupling pushes the practical limit lower, and 0.001 is where 5000 steps still lands cleanly. The lesson: the steepest direction, not the average one, sets the largest safe lr.
Loss sampled at steps 0 / 100 / 200 / 500 / 1000: β = 0 → 4.00 / 3.83 / 3.66 / 3.13 / 2.10 β = 0.5 → 4.00 / 3.67 / 3.32 / 2.10 / 0.406 β = 0.9 → 4.00 / 2.33 / 0.435 / 0.038 / 0.00324 β = 0.99 → 4.00 / 0.138 / 0.502 / 0.0306 / 0.000265 β = 0.99 is fastest early (loss 0.138 by step 100) but overshoots — the loss jumps back up to 0.502 by step 200 before settling to 0.000265. β = 0.9 is the reliable middle: it accelerates the long valley floor without throwing the run past the minimum. Higher β means longer memory: smoother and faster in consistent directions, but slower to react when the valley turns.
From (0.01, 0.01), after 20 steps: GD: y = 0.383, loss = −0.147 (slow: y grows by 1 + 2·lr = 1.2 per step) momentum: y = 0.162, loss = −0.026 (fast early, then oscillates) Adam: y = 1.025, loss = −1.051 (fastest: ~lr per step, immediately) GD moves along x toward 0 (that direction is a minimum) and along y away from 0 — but near the saddle the y-gradient is tiny, so it creep-crawls at first. Momentum builds speed in the consistent y direction, and Adam's normalised step (±lr every step) pulls away fastest of all. Now start at (0.01, 0). The y-gradient is exactly 0 and stays 0 in exact arithmetic: all three optimizers stroll along x into the saddle and stall there (x → 0, y = 0, loss ≈ 0). No deterministic method fixes this — the escape in real training comes from mini-batch noise or from initialising off the stable direction.
5000 steps from (−1, 1), final loss: lr₀ = 0.001 constant → 0.00363 decay 0.999 → 0.413 lr₀ = 0.003 constant → 0.304 decay 0.999 → 0.0176 decay 0.9995 → 0.00177 lr₀ = 0.004 constant → 0.572 decay 0.999 → 0.00574 decay 0.9995 → 0.000548 Two honest conclusions. First, if a small constant lr already works, decaying it too fast starves the run: at lr₀ = 0.001 the rate is down to ~0.0000067 by step 5000 and the loss stalls at 0.413, much worse than constant. Second, decay's real payoff is letting you start bigger: 0.004 constant just bounces around (0.572), while the same start with decay 0.9995 lands at 0.000548 — big steps early, careful steps late. That trade — start higher because the schedule will bring you down — is the modern warmup-plus-decay recipe in one experiment.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 01, Lesson 08) and the Math Foundations Notebook reference build. Interactive figures, recomputed numeric checks, worked exercise answers and the saddle-point lab are original to this page. Every optimizer trajectory is computed in your browser from the update rules.