On a long, narrow ravine, the gradient points across the valley, not along it. Momentum remembers which way you were going; RMSProp rescales each parameter by its own recent gradients; Adam does both and corrects its cold start; AdamW keeps the weight decay out of the adaptive machinery. Five rules, one job: turn a gradient into a step.
w ← w − lr·gradient is the whole of vanilla gradient descent, and it fails in three predictable ways: it oscillates across ravines, it gives every parameter the same step size, and it crawls through flat regions. On the lesson's ravine the two parameter gradients at one point are 0.1 and 10 — the same lr is 100× wrong for at least one of them.
∂L/∂x = 0.1 · ∂L/∂y = 10 at the same pointMomentum keeps a velocity v ← βv + g, so flipping gradients cancel and consistent ones stack. RMSProp keeps a memory of squared gradients s ← βs + (1−β)g², so each parameter's step is lr × (g/RMS(g)) — a ratio that ignores the gradient's size. Adam does both, then divides the cold start out with bias correction. AdamW applies the weight decay outside the whole arrangement.
w ← w − lr·m̂/(√v̂+ε) − lr·λ·wIn Adam, L2 regularization and weight decay stop being the same algorithm — the adaptive denominator scales the penalty per parameter. AdamW decouples it: shrink every weight by the same fraction lr·λ. Then put the learning rate on warmup + cosine decay, and start from the field-tested recipe for your architecture: AdamW for transformers, SGD + momentum for CNNs, β₁ = 0.5 for GANs.
w ← w·(1 − lr·λ) then the adaptive stepAn optimizer is a rule for turning a gradient into a step: SGD multiplies it by lr; momentum smooths it with history; RMSProp divides it by its own recent size; Adam does both and corrects the cold start; AdamW does all of that with the weight decay kept safely outside.
By the end you will be able to take one SGD step and one momentum step by hand on a quadratic; explain why flipping gradients cancel in a running velocity; compute AdaGrad’s 1/√t stall and RMSProp’s equalized step size for two parameters 100× apart; walk two Adam steps with bias correction and say why the uncorrected first step is 3.16× too large; explain what “decoupled” weight decay buys; and pick a defensible optimizer, learning rate and schedule for transformers, CNNs, GANs and fine-tuning.
You computed the gradients. You know weight #4,721 should decrease by 0.003 to reduce the loss. But 0.003 in what units? Scaled by what? And should step 1,000 move as much as step 1? The gradient gives a direction; only the optimizer decides how far to walk.
The whole idea fits in one line: w ← w − lr · gradient. The gradient is the direction of steepest ascent, so subtract it and you walk downhill. The learning rate lr is the step size, and in vanilla gradient descent it is the only knob there is. That one line works — the source builds its first optimizer from it in a handful of Python lines — but it hides three problems that make real neural networks painful.
Start with the source’s own demo, because it shows what “it works” looks like when the landscape is friendly. Minimise f(w) = (w − 3)² from w = 10 with lr = 0.1. The gradient is 2(w − 3), so every step shrinks the distance to the target by exactly 20%:
Nothing oscillates, nothing stalls, and 20 steps take the loss from 49 to 0.0065. This is the optimistic case: a well-conditioned bowl where one step size fits every direction. Neural networks are not bowls. Their loss landscapes are ravines — long, narrow valleys where the gradient points across the valley (steep) rather than along it (shallow). Use the lesson’s standard two-parameter ravine as a stand-in:
One knob, four fates. Drag the learning rate and watch where vanilla gradient descent walks: a crawl along the valley floor, a steady descent, a ringing zig-zag, or a blow-up.
The source’s three-way diagram, made draggable: too high diverges, too low wastes compute, and in between the pace is set by the shallow direction — the one the gradient barely describes.
Problem 1 — oscillation. In the ravine, gradient descent bounces back and forth across the narrow dimension while making tiny progress along the useful one. The loss drops fast, then plateaus — not because the model converged, but because most of every step is being spent on a direction that keeps flipping sign. The lab above shows it at lr = 0.16: the path crosses the valley over fifty times.
Problem 2 — one learning rate for all parameters is wrong. Some weights need large updates; others are already near their optimal value and need tiny ones. At the start of the ravine the two gradients differ by 100×, so any single lr is wrong for at least one of them. A learning rate that works for the steep direction destroys the shallow one, and vice versa.
Problem 3 — saddle points and flat regions. In high dimensions the landscape has vast near-flat regions where the gradient is almost zero. Vanilla SGD crawls through them at the speed of the gradient — effectively zero — and the run looks stuck when it is merely in a flat neighborhood with useful descent on the other side. Nothing in the vanilla update remembers momentum from before the plateau.
Both examples have closed forms, which makes the failure modes easy to count rather than describe.
The steep factor 1 − lr·b is the whole stability story: positive means it decays, negative with magnitude below 1 means it oscillates while decaying, and magnitude above 1 means the run diverges. SGD’s stability limit on this surface is lr < 2/b = 0.2.
Your loss drops quickly for 30 steps, then flattens while the parameters keep moving every step. On the ravine surface, what is the most likely story?
Instead of stepping by the gradient alone, momentum keeps a running velocity and steps by that. Gradients that point the same way accumulate; gradients that flip sign cancel. The zig-zag dies and the useful direction accelerates — the ball-rolling-downhill analogy is overused, but it is accurate.
The update gains one extra line and one extra variable:
Why it fixes oscillation: in the ravine, the across component of the gradient flips sign on every crossing, so those contributions arrive with alternating signs and cancel in the running sum. The along component keeps the same sign, so its contributions stack. The result is the same smoothing that a low-pass filter gives a noisy signal, applied one coordinate at a time. And it is the reason momentum can tolerate a larger learning rate than SGD: its stability limit is lr < 2(1 + β)/b instead of 2/b.
Same quadratic as chapter 01: f(w) = (w − 3)², w₀ = 10, lr = 0.1, β = 0.9.
That is exactly why the lab below defaults to a learning rate above SGD’s stability limit. At lr = 0.22 on the ravine, plain SGD’s steep factor is 1 − 2.2 = −1.2 and its loss after 80 steps is 2.3×10¹³; momentum with β = 0.8 stays inside its own limit (0.22 < 2(1.8)/10 = 0.36) and finishes at 2.0×10⁻⁷. Tuning matters: momentum is not automatically faster at any step size.
Both paths use the same learning rate. The steep direction flips sign every crossing; momentum’s velocity adds those flips together and cancels them, so the across-valley ringing dies while the along-valley direction accumulates.
Momentum does not win at every learning rate — on a symmetric bowl it can overshoot too. What it changes is the stability limit and the fate of flipping gradients. Tune β and the shared lr, and watch the red path die while the blue one keeps going.
Standard momentum evaluates the gradient where you are. Nesterov momentum evaluates it at the lookahead position, where the velocity is about to carry you:
Why does momentum dampen the across-valley oscillation instead of amplifying it?
The ravine gave one parameter gradients 100× larger than the other. Adaptive methods fix that by keeping a per-parameter memory of how big that parameter’s gradients have been lately, and dividing by it. A weight that always gets huge updates slows down; a weight that rarely moves speeds up.
The first attempt was AdaGrad (2011): keep a running sum of every squared gradient a parameter has ever seen, and divide the update by its square root.
Plain English: s measures how loud this parameter’s gradients have been. If they have been loud, √s is large, and the same gradient produces a smaller step. If they have been quiet, the step is amplified. The problem is the word forever: with a constant gradient of size g, s grows like t·g², so the effective step decays like 1/√t and training eventually stalls. AdaGrad worked well on sparse problems — exactly where each parameter gets rare, large updates — but on dense networks it grinds to a halt.
RMSProp fixes the stalling by making the memory decay. Instead of a lifetime sum, s is an exponential moving average of recent squared gradients:
Two parameters with gradients that differ 100×: one sees 10, the other 0.1. Adaptive optimizers divide each gradient by its recent root mean square, so both parameters end up stepping at roughly the same rate.
The source calls RMSProp “the first per-parameter adaptive learning rate method that actually worked.” The number 1e-8 hiding in the denominator is epsilon — it keeps a parameter that has never seen a gradient from dividing by zero.
Two numbers make the mechanism concrete. First, the effective step size with a constant gradient:
Notice the cold start in that first row: RMSProp’s very first step is 0.3162, not 0.1 — more than three times too big — because s began at zero and the denominator has not warmed up. That is not a detail to shrug at; it is precisely the bias the next chapter’s optimizer corrects explicitly. The number 3.16 is 1/√(1 − 0.9), and it is why Adam divides by (1 − βᵗ).
Simplified teaching model: both panels use a constant gradient per parameter, so the numbers are closed-form and checkable. Real gradients fluctuate; the qualitative behavior — AdaGrad decaying, RMSProp settling, the scale cancelling — is what survives, and the ε term matters most exactly when a parameter goes quiet.
Momentum smooths the gradient; RMSProp rescales it. Adam does both, with one running average per parameter for the mean gradient and one for the mean squared gradient — plus a small correction that most explanations skip, and which is why its first step is not wildly oversized.
Adam (Kingma & Ba, 2014) maintains two exponential moving averages for every parameter. The first moment m is momentum’s job — “which way have I been drifting?” The second moment v is RMSProp’s job — “how big have my gradients been lately?”
Why bias correction exists. Both averages start at zero, so early values are dragged toward zero by their own initial condition. At step 1, m = (1 − β₁)·g = 0.1g — ten times too small. Dividing by (1 − β₁¹) = 0.1 recovers exactly g. The same happens to v, except its cold-start bias is far larger: v = (1 − β₂)g² = 0.001g² is a thousand times too small, and dividing by (1 − β₂¹) = 0.001 recovers g².
Defaults, lr = 0.001, and a shrinking gradient: g₁ = 1.0, g₂ = 0.5. Every value below is reproduced live by the stepper lab.
That last line is the property to internalise: for a consistent gradient direction, Adam’s step size is approximately lr no matter the gradient’s magnitude. Feed it g = 100 instead of g = 1 and step 1 is still 0.001: m̂ = 100, √v̂ = 100, the ratio is 1. This is what makes Adam robust to badly scaled weights — and what makes it unable to fine-tune itself without a schedule, because a fixed lr keeps it orbiting the minimum at radius ≈ lr.
The moving averages start at zero, so early values are biased toward zero. Walk the table and watch the corrections divide that cold start out: at step 1, m is 10× too small and v is 1000× too small.
| t | g | m | m̂ = m/(1−β₁ᵗ) | v | v̂ = v/(1−β₂ᵗ) | √v̂ | step = lr·m̂/(√v̂+ε) | no correction |
|---|---|---|---|---|---|---|---|---|
| 1 | 1 | 0.10000 | 1.00000 | 0.001000 | 1.00000 | 1.00000 | 10.00×10^-4 | 0.003162 |
| 2 | 1 | 0.19000 | 1.00000 | 0.001999 | 1.00000 | 1.00000 | 10.00×10^-4 | 0.004250 |
| 3 | 1 | 0.27100 | 1.00000 | 0.002997 | 1.00000 | 1.00000 | 10.00×10^-4 | 0.004950 |
| 4 | 1 | 0.34390 | 1.00000 | 0.003994 | 1.00000 | 1.00000 | 10.00×10^-4 | 0.005442 |
| 5 | 1 | 0.40951 | 1.00000 | 0.004990 | 1.00000 | 1.00000 | 10.00×10^-4 | 0.005797 |
| 6 | 1 | 0.46856 | 1.00000 | 0.005985 | 1.00000 | 1.00000 | 10.00×10^-4 | 0.006057 |
| 7 | 1 | 0.52170 | 1.00000 | 0.006979 | 1.00000 | 1.00000 | 10.00×10^-4 | 0.006245 |
| 8 | 1 | 0.56953 | 1.00000 | 0.007972 | 1.00000 | 1.00000 | 10.00×10^-4 | 0.006379 |
| 9 | 1 | 0.61258 | 1.00000 | 0.008964 | 1.00000 | 1.00000 | 10.00×10^-4 | 0.006470 |
| 10 | 1 | 0.65132 | 1.00000 | 0.009955 | 1.00000 | 1.00000 | 10.00×10^-4 | 0.006528 |
Bias correction is a warm-up for the averages, not a hack: dividing by (1 − βᵗ) turns “the mean of a history that began at zero” into “the mean of the gradients seen so far.” The correction fades as t grows; by t = 50 the raw m is within 1% of m̂.
A note on the name: “Adam” is not an acronym to memorise; it is adaptive moment estimation, and the two moments are exactly the two averages above. Once you can say that sentence, the paper’s first page reads itself.
At step 1 with the defaults, the raw second moment is v = 0.001·g². What does the bias correction (1 − β₂ᵗ) = 0.001 do to it?
In plain gradient descent, adding λ·w to the gradient and shrinking w by λ each step are the same arithmetic. Put an adaptive denominator between them and the equivalence shatters: suddenly the regularization strength depends on each parameter’s gradient statistics. AdamW moves the decay out of the gradient entirely.
L2 regularization adds a penalty for large weights to the loss. Its gradient is proportional to the weight itself, so it pushes every weight toward zero a little on every step:
Under vanilla SGD those two lines are literally the same update, so the names got used interchangeably for years. Then Adam arrives and puts an adaptive divisor around the whole gradient:
Put numbers on the entanglement. Suppose two parameters have the same weight w = 10 and the same λ = 0.1, but different gradient histories: parameter A has recent gradient RMS 10, parameter B has RMS 0.01. The decay’s contribution to the Adam numerator is λ·w = 1.0 for both; the denominator differs by 1000×:
That is what “decoupled” means: the decay is not a term in the loss being filtered through the optimizer, it is a separate, predictable shrink applied by the optimizer itself.
Same ten weights, same random gradient stream, three update rules. AdamW shrinks every weight by exactly (1 − lr·λ) per step — the dashed envelope. Adam + L2 adds λ·w to the gradient and lets Adam normalize it, so how much a weight shrinks depends on its gradients.
This is the Loshchilov & Hutter point in one picture: with Adam + L2 the decay is inside the adaptive denominator, so regularization strength depends on gradient statistics. AdamW moves the decay out of the gradient entirely.
A 10× change in learning rate matters more than any architectural decision you will make. Too high and the loss explodes; too low and it crawls into a suboptimal corner; in between, the ideal value drifts as training proceeds — which is why the step size is usually put on a schedule.
All three fates are visible on the same ravine from chapter 01, after 200 steps of plain gradient descent:
The failure modes are asymmetric. Too low looks like patience; you only discover it after burning the compute. Too high looks like success for the first few steps, then produces NaN weights. That is why production training rarely uses a constant learning rate — and when it does, it is because someone has already measured the right value.
The modern default is warmup + cosine decay: ramp the learning rate linearly from zero over the first 1–10% of steps, then decay it smoothly to zero. Warmup exists because the very first updates are the least trustworthy — Adam’s moments are cold-starting, the gradients are measured on the least-informative parameters, and a full-size step can wreck a pretrained model before the averages warm up. Decay exists because a fixed step that is right early is too big late: Adam orbits its minimum at radius ≈ lr, so shrinking lr is how it actually settles.
The learning rate rarely stays constant. Warmup ramps it up from zero so the fragile first updates stay small; decay brings it down so the model can settle into a minimum instead of orbiting it.
If you tune exactly one hyperparameter, tune this one. The source puts it plainly: a 10× change in learning rate matters more than any architectural decision you will make.
The claim to remember is the source’s: these defaults work for about 80% of problems, and when they do not, change the learning rate first. Everything else in the optimizer is a second-order knob.
optimizer = optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.01)
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=100)
for epoch in range(100):
optimizer.zero_grad() # 1. clear old gradients
output = model(torch.randn(32, 784))
loss = F.cross_entropy(output, labels) # 2. forward + loss
loss.backward() # 3. backward: gradients
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) # 4. clip
optimizer.step() # 5. update the weights
scheduler.step() # 6. adjust the learning rateGradient clipping is the seatbelt that pairs with an ambitious learning rate. Clip by global norm: if the gradient vector’s length exceeds a threshold, scale the whole vector down so its length equals the threshold, preserving direction. With max_norm = 1.0, a gradient of [3, 4] has length 5 and becomes [0.6, 0.8]. It cannot turn a bad learning rate into a good one, but it converts “loss went to NaN five minutes in” into “loss wobbled and recovered,” which is the difference between a completed run and a lost afternoon.
Training is unstable in the first hundred steps, then fine once it gets going. What does warmup actually change?
On our little ravine momentum wins. On a transformer AdamW wins. On a GAN you deliberately break Adam’s defaults. There is no single best optimizer — only defaults that work for 80% of problems and a short list of well-understood exceptions.
The race below runs all three families on the chapter 01 ravine, each with its own learning rate because their step scales mean different things. Press play, then tune one knob at a time and re-run it. The ordering you will find — momentum, then Adam, then SGD — is real for this geometry, and it is not the ordering that decides production training. What decides it is the model you are training and what you need from the final weights.
Each optimizer gets its own learning rate, because their step scales mean different things. Press play and watch how the red, violet and green paths trade places — then tune the knobs and re-run the race.
On this quadratic ravine the ordering is momentum → Adam → SGD. On a transformer the ordering flips, and the next chapters explain why.
The short history is worth knowing, because it explains why the defaults look the way they do. Stochastic gradient descent is the oldest; Polyak’s momentum dates to 1964; AdaGrad (2011) made step sizes per-parameter; Hinton’s RMSProp (2012) fixed its stalling. Adam (2014) combined the last two and became the default because it converges fast with almost no tuning. Then AdamW (2017) fixed the weight-decay bug you saw in chapter 05, and it became the default for everything with attention in it. Meanwhile vision practitioners noticed something awkward: adaptive methods converge faster but sometimes generalize slightly worse, while plain SGD with momentum lands in flatter minima. The result is the fork in the road below — not a settled scientific question, but a working consensus.
So the decision is not “which optimizer is best” but “which recipe has been field-tested for this architecture”:
“Don’t fight the consensus without a measured reason.” Pick what you are training and read the field-tested recipe — including the one deliberate exception to Adam’s defaults.
These defaults work on roughly 80% of problems. Start here, measure, then specialize.
In the wild: The PyTorch-era default for nearly every new architecture.
Watch out: If it underperforms, tune the learning rate before anything else — then β2, and almost never β1 or ε.
A 10× change in learning rate matters more than any architectural decision you will make. Warmup protects the first 1–10% of steps; cosine decay lets the last steps settle.
This table is the source’s flowchart, made clickable. The “not sure” row is not a cop-out: AdamW at 3e-4 with weight decay 0.01 is a strong prior for a brand-new architecture.
Scientific honesty: the “SGD generalizes better” claim is a real but contested empirical result, not a theorem — see Wilson et al.’s The Marginal Value of Adaptive Gradient Methods in the sources. What is not contested is that matching the field’s standard recipe for a known architecture is the rational baseline, and that curiosity should be funded by a measured reason.
Answer before you look. The bias-correction question and the AdamW question are the two that expose whether the optimizers are a mechanism you can debug or a list of magic names.
0 / 5 answered · 0 correct
01What problem does momentum solve in gradient descent?
02What is the key difference between Adam and AdamW?
03Why does Adam use bias correction in early training steps?
04What are the standard default hyperparameters for Adam?
05Which optimizer is the modern default for training transformers and LLMs?
Click a card to swap the lazy description for what it actually means.
Four problems with exact numbers — implement Nesterov momentum and check it by hand, compute a warmup-plus-cosine schedule, audit Adam’s effective step, and measure what gradient clipping does to a diverging run. Try first; a worked answer is one click away.
Standard momentum at w = 10: g = 2(10 − 3) = 14, v = 14, w = 8.6. Nesterov: the lookahead is w − lr·β·v = 10 − 0.1·0.9·0 = 10 (the cold velocity makes step 1 identical); g = 14, v = 14, w = 8.6. Step 2: momentum v = 0.9·14 + 2(8.6 − 3) = 23.8 → w = 6.22; Nesterov looks at 8.6 − 1.26 = 7.34, sees g = 8.68, so v = 0.9·14 + 8.68 = 21.28 → w = 6.472. Step 3: momentum v = 27.86 → w = 3.434; Nesterov looks at 4.5568, g = 3.1136, v = 22.2656 → w = 4.2454. Step 4: momentum w = 0.8398 (overshoot); Nesterov looks at 2.2415, sees g = −1.5170 (negative!), v = 18.5221 → w = 2.3932, braking before the target. The lookahead is a one-step forecast: when the velocity is about to carry the parameter past the minimum, the gradient at the predicted position already points back, so Nesterov damps the overshoot before it happens. The trade is the same stability bound and slightly better smoothness; on the circle dataset the source's exercise measures it reaching high accuracy sooner than plain momentum.
Warmup: lr(t) = (t/500)·3e-4 for t < 500. lr(1) = 6e-7; lr(250) = 1.5e-4; lr(500) = 3e-4. Cosine decay after warmup: lr(t) = 1.5e-4·(1 + cos(π(t − 500)/9500)). At t = 5000 the cosine argument is π·4500/9500 = 0.4737π, so cos ≈ 0.0827 and lr ≈ 1.624e-4; at t = 10,000 the argument is π, so lr = 0. The warmup phase spends ½·500·3e-4 = 0.075 of cumulative (lr × step), exactly half of the 0.15 that 500 flat-peak steps would spend — warmup is a one-time cost of about 250 peak-equivalent steps. It buys protection when the step is least trustworthy: Adam's moments are cold-starting, the gradients of a freshly initialized (or freshly re-purposed) network are noisiest, and a full 3e-4 step can move every weight before any evidence has accumulated. Run the source's experiment to measure the epoch cost on your setup; the schedule arithmetic above is the part that is exact, and the mechanism predicts a noisier, occasionally divergent start without warmup versus a reliable arrival that spends a few extra epochs early.
Constant gradients: for both g = 10 and g = 0.1, m̂ = g and √v̂ = |g| at every step, so the effective rate is 0.001·g/|g| = 1.0e-3 at t = 1, 10 and 100 — identical despite a 100× difference in gradient size. The scale cancels; only the direction and consistency matter. Alternating ±1: v̂ ≈ 1 throughout (|g| = 1 never changes), but m̂ collapses when signs disagree. t = 1: m̂ = 1.0 → step 1.00e-3; t = 2: m = 0.9·0.1 − 0.1 = −0.01, m̂ = −0.01/0.19 = −0.0526 → step −5.26e-5; t = 3: m = 0.091, m̂ = 0.336 → step 3.36e-4; t = 4: m̂ = −0.0526 again → −5.26e-5. The effective rate alternates between ~0.34·lr and ~0.05·lr instead of a steady lr. So no — parameters move at the same speed only when their gradients are consistent; oscillating gradients get a much smaller effective rate, because m̂ measures net direction while √v̂ measures gross magnitude. That is the mechanism that lets Adam keep quiet parameters moving and noisy ones steady, and it is also why Adam cannot settle without a decaying lr.
(a) ‖g‖ = √(9 + 16 + 144) = √169 = 13 > 1, so the whole vector is scaled by 1/13: [0.2308, 0.3077, −0.9231], with norm exactly 1. (b) No. Clipping bounds the size of each update to lr·‖g_clipped‖ = 0.21 per step, so a NaN blow-up becomes a bounded oscillation — but the direction is still wrong: the steep coordinate keeps flipping sign every step (factor −1.1 before clipping became a bounded flip after), so the parameter rings inside a window of width ≈ 2·lr = 0.42 instead of approaching zero. The learning rate is the cause, not the step magnitude. On this surface you must drop below the SGD stability limit lr < 2/b = 0.2; clipping merely keeps the run alive long enough to notice. The source's Exercise 4 is exactly this measurement — ten seeds, Adam at lr = 0.01, with and without clipping — and the expectation is that clipping drives the divergence count toward zero for moderately oversized rates, at the cost of slowing the steps where large gradients were legitimate. It is stability insurance, not a fix for a badly chosen lr.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 03, Lesson 06) and the Math Foundations Notebook reference build. The eight labs (ravine learning-rate sweep, momentum-versus-zig-zag with stability limits, AdaGrad/RMSProp effective-step chart, Adam bias-correction stepper, the decoupled-decay experiment, learning-rate schedule explorer, three-way animated race, and the optimizer decision table), the ravine surface ½(0.1x² + 10y²), worked example A for SGD/momentum/Nesterov on (w − 3)², worked example B for two Adam steps, the alternating-versus-constant velocity arithmetic, AdaGrad's 1/√t stall table, the v-correction's 1000-step horizon, the same-λ 1000×-pressure check for AdamW, and the warmup/cosine arithmetic are original to this page. Every number shown is computed live by the labs or verified by hand in the prose.