Set it high and the loss rings or explodes. Set it low and the run crawls to a floor it never leaves. Every serious training recipe answers with a schedule: ramp the step size up while Adam’s statistics are cold, hold it through the middle, and decay it to let the model settle. Five shapes, one job — spend the step size where it pays.
On the lesson's one-parameter bowl, the step factor is 1 − lr·c: stability needs |1 − lr·c| < 1, the noise floor grows with lr², and the rate that reaches the floor fastest is not the rate that reaches it lowest. Ten steps in, lr = 0.1 has the loss at 0.12 but its floor is 0.05; lr = 0.005 is 15× behind at that point yet floors 40× lower; lr = 0.35 diverges, multiplying the loss by 6.25 every step. Training wants both ends of that trade — at different times.
stable iff lr < 2/c · floor ≈ lr²σ²/(2c)02 / WARM UP, THEN WIND DOWN
Start smaller than the peak. End smaller than the peak.
Warmup ramps the step size linearly from near-zero over the first 1–5% of steps, so Adam's zero-initialised moment estimates can absorb a few batches of real gradients before any full-size update is taken. Decay then lowers the rate smoothly, because the residual size of the steps sets the noise floor the model orbits at. On the source's circle network with Adam at peak 0.1, no warmup ends at 0.063 while a 10% warmup reaches 0.0032 — the warming run is behind early and better at the end.
lr(t) = peak·t/T_w → cosine or step decay03 / MEASURE THE PEAK, MATCH THE BUDGET
The shape is a decision; the peak is a measurement.
The LR range test finds the peak by growing the rate exponentially for a few hundred steps and reading where the smoothed loss stops falling — in the lesson's run, a minimum at 5.9×10⁻², so train around 6×10⁻³–2×10⁻². After that the flowchart is short: warmup + cosine is the transformer default; 1cycle wins short budgets; step decay is the legacy vision recipe; and when the batch size grows k×, the learning rate scales roughly k× and the warmup gets longer.
range test → peak · warmup+cosine → default
MENTAL MODEL IN ONE SENTENCE
A schedule is a plan for the step size over time: tiny while the optimizer’s statistics are cold, large while there is ground to cover, small again while the model settles — and the plan is worth more than any single rate inside it.
By the end you will be able to compute a step-decay staircase, a cosine value and a warmup ramp by hand; explain why Adam needs warmup more than SGD does; run and read an LR range test; write the warmup + cosine formula from memory and check its endpoints with one line of arithmetic; build a schedule curve from sliders and say what it spends; and choose a defensible schedule — with the right peak, floor, warmup and batch-size coupling — for transformers, CNNs, fine-tunes and short experiments.
01
ONE NUMBER, THREE FATES
The learning rate is the one knob that must change.
Set it to 0.1 and the run diverges in three steps. Set it to 0.0001 and after a hundred epochs the model has barely moved from random. The values in between work for a while, then the loss oscillates around a minimum it can never reach. It is not that the number is wrong — it is that no single number can be right for the whole run.
That opening belongs to the source, and it names the entire problem. The source puts it even more bluntly than most courses would: the learning rate is the single most important hyperparameter — not the architecture, not the dataset size, not the activation function. Every major model published in the last few years uses a schedule: Llama 3 peaked at 3e-4 after 2,000 warmup steps and cosine-decayed to 3e-5; GPT-3 warmed up over 375 million tokens and decayed to zero. Those are not arbitrary numbers — they are the output of million-dollar hyperparameter sweeps.
Before the formulas, make the failure modes exact. The smallest model that shows all of them is a one-parameter quadratic with noisy gradients — a simplified teaching model, but one whose arithmetic you can check with a calculator:
f(w) = ½ · c · w² gradient g = c·w + σ·ξ, ξ ~ N(0, 1)
w ← w − lr·g = (1 − lr·c)·w − lr·σ·ξ
deterministic part: multiply the parameter by (1 − lr·c) each step
stable iff: |1 − lr·c| < 1 i.e. 0 < lr < 2/c
noise floor: Var(w) = lr²σ² / (1 − (1 − lr·c)²)
E[loss] = ½ · c · Var(w)
With c = 10, the stability limit is lr < 2/c = 0.2. Everything else is a consequence of that one factor. At lr = 0.35 the factor is 1 − 3.5 = −2.5: the parameter flips sign every step and grows 2.5×, so the loss is multiplied by 2.5² = 6.25 per step — five steps in it has grown 9,500×, and by step 10 it passes 10⁸. At lr = 0.19 the factor is −0.9: stable, but ringing, and the noise floor is high because lr is large. At lr = 0.005 the factor is 0.95: no ringing, but the parameter shrinks only 5% per step, so the run looks frozen for hundreds of steps.
One learning rate, five fates
A one-parameter bowl with noisy gradients — the smallest model where the three failure modes are exact arithmetic. Drag the rate and watch the loss trace: it crawls, descends to a floor, rings, or explodes, and the step factor 1 − lr·c tells you which before you run it.
lr 0.19000
step factor 1 − lr·c -0.9000
|factor| 0.9000 < 1 — stable
verdict RINGS WHILE DECAYING
stationary floor formula lr²σ²/(1 − (1 − lr·c)²) · ½c = 0.9500
measured tail mean 1.118 (last third of the run)
stability limit lr < 2/c = 0.2
at lr = 0.35 the factor is −2.5 → loss ×6.25 per step → 10⁸ by step 10.
the factor is negative, so the parameter flips sign every step: the loss swings up and down while shrinking toward the noise floor.
The trade nothing constant can escape: lr = 0.10 is fast but its floor is
0.05000; lr = 0.005 floors at 0.001282 but
needs 200+ steps to get there. That gap is what a schedule exploits.
A simplified teaching model — one parameter, independent noise — but its arithmetic is exact: the factor, the floor and the divergence threshold are the same quantities a real network’s loss surface supplies in every direction.
Worked arithmetic — why the floor is a floor
The noise term never goes away, so the parameter never sits at zero. It settles into a stationary distribution whose variance you can compute exactly: repeatedly apply w ← (1 − lr·c)w − lr·σ·ξ with independent unit noise and the variance converges to lr²σ² / (1 − (1 − lr·c)²). Three values of lr, checked against the lesson’s measured tail means with c = 10, σ = 1:
lr = 0.19 factor −0.90 Var = 0.0361 / (1 − 0.81) = 0.190
E[loss] = 5 × 0.190 = 0.950 measured tail mean 1.141
lr = 0.10 factor 0.00 Var = 0.0100 / (1 − 0) = 0.010
E[loss] = 5 × 0.010 = 0.050 measured tail mean 0.0498
lr = 0.05 factor 0.50 Var = 0.0025 / (1 − 0.25) = 0.00333
E[loss] = 5 × 0.00333 = 0.0167 measured tail mean 0.0158
and the trap: lr = 0.35 → 1 − lr·c = −2.5 → 1 − 2.5² is negative →
the stationary variance does not exist. The run has no floor to reach.
Read the middle table slowly, because it is the whole lesson in miniature. Ten steps in, lr = 0.1 has cut the loss from 5.0 to 0.12 — but its floor is 0.05. The same ten steps at lr = 0.005 leave the loss at 1.79, 15× behind; yet that rate floors at 0.0013, about 40× lower. The fastest rate and the best rate are different rates. A schedule is how a run gets both: fast while it can afford noise, slow when it cannot.
Quick check
Training loss falls for the first fifty epochs, then stops improving and oscillates in a narrow band. On the noisy-quadratic model, which explanation fits best?
02
THE SCHEDULE GALLERY
Five shapes. Each spends the budget differently.
A schedule is just a function from step to learning rate. Everything in the field is a variation of five of them: keep it constant, drop it in steps, multiply it down smoothly, follow a cosine, or ramp up first. Compose any of them with a warmup and you have every modern recipe. Each formula is one line — the rest of the lesson is choosing among them.
Constant.lr(t) = lr₀. Pick a number, use it for every step. Rarely optimal — it is either too high for the end of training (the oscillation from the last chapter) or too low for the beginning (wasted compute on tiny steps). It is fine for small models and for debugging, because nothing is hidden; it is a poor choice for anything that trains for more than an hour.
Step decay. The old-school approach from the ResNet era: cut the learning rate by a factor at fixed milestones.
lr(t) = lr₀ · γ^⌊t / step_size⌋ γ = 0.1, milestones every 30 epochs
peak 0.1, ×0.1 every 300 steps:
steps 0–299 0.100000
steps 300–599 0.010000 one drop = ÷10
steps 600–899 0.001000 two drops = ÷100
steps 900–999 0.000100 three drops = ÷1000
ResNet-50 used exactly this: lr = 0.1, dropped by 10× at epochs 30, 60 and 90. The problem is that the right milestones depend on the dataset and the architecture — move to a different problem and you re-tune when to drop. The transitions are also abrupt: the loss can bump for a few epochs right after a cliff, because the step size changed under the optimizer’s feet.
Exponential decay. Everything step decay does, without the cliffs: multiply the rate by a fixed factor every step.
lr(t) = lr₀ · γᵗ γ = 0.999 per step
peak 0.05: t = 100 → 0.045240 t = 1000 → 0.018385
t = 500 → 0.030319 t = 2000 → 0.006760
per epoch (200 steps/epoch): 0.999²⁰⁰ = 0.819
to reach 1e-5: ln(1e-5 / 0.05) / ln(0.999) ≈ 8,513 steps
Smooth, one hyperparameter, and it decays faster early than late — which is backwards from what most schedules want. It also never actually finishes: after 2,000 steps 13.5% of the peak is still there.
Cosine annealing. The modern default, and the shape the rest of the lesson builds on — a smooth half-cosine from the peak to a floor:
lr(t) = lr_min + ½ · (lr_max − lr_min) · (1 + cos(π · t / T))
peak 3e-4, T = 100, lr_min = 0:
t = 25 lr = 2.5607e-4 cos(π/4) = 0.7071 → ½ · 1.7071 = 0.8536
t = 50 lr = 1.5000e-4 half the distance: exactly lr_max / 2
t = 75 lr = 4.3934e-5 cos(3π/4) = −0.7071 → ½ · 0.2929 = 0.1464
t = 100 lr = 0 cos(π) = −1 → the floor
The decay is gentle at first, steepens through the middle, then flattens near the floor — which matches the empirical observation that most learning happens in the middle of a run, where the step size is still substantial. There are no milestones to tune: the only choices are the peak, the floor lr_min and the total steps. That is why it became the default for most modern training.
Every schedule on one chart
Toggle the six shapes, then move the peak, the total steps, the warmup length and the floor. Two schedules that look nothing alike can spend almost the same learning-rate budget — with a 10% floor, cosine and warmup + cosine both average 0.55 of the peak over a full run — while the step function spends 0.28.
The budget line is the honest comparison: cosine and warmup + cosine both average half the peak over the run, so a schedule is not just a shape — it is a way of spending the same total step size differently across time.
Worked arithmetic — what each shape spends
Plotting shapes at the same peak is misleading: they do not spend the same total step size. The honest comparison is the mean learning rate over the run, which is the area under the curve divided by its width.
mean lr ÷ peak, over a 1,000-step run (floors negligible):
constant 1.000 every step at full size
step decay 0.278 ×0.1 at 25%, 50%, 75% of the run
exponential 0.432 γ = 0.999 per step, over 2,000 steps
cosine 0.501 half the peak, by symmetry
warmup+cosine 0.500 the ramp adds back almost nothing
1cycle 0.510 the up-ramp roughly replaces what the tail loses
In lr·steps: cosine spends ≈ 500 peak-steps; a constant rate spends 1000.
Two schedules with the same peak can differ 3.6× in budget.
This is why “just lower the constant rate” is not the same as adding a schedule: dropping to half the peak for the whole run spends the same 500 peak-steps as cosine, but it moves slowly in the early phase where progress is cheap and still moves at half speed in the late phase where precision matters. Cosine spends the same currency — just earlier.
03
WARMUP
The first steps are the least trustworthy ones.
Adam and every other adaptive optimizer keeps running estimates of the gradients. At step 0 those estimates are zero — and the first batches are the least representative data the model will ever see. Warmup is the ramp that keeps the early steps small until the statistics, and the model, have something to stand on.
The source states the mechanism in one paragraph, and it is worth internalising exactly: an adaptive optimizer maintains running estimates of gradient mean and variance, initialised to zero; the first few updates are therefore based on unreliable statistics; if the learning rate is large during that window, the model takes huge, poorly-directed steps. Warmup replaces the full rate with a linear ramp:
lr(t) = lr_max · (t / warmup_steps) for t < warmup_steps
peak 3e-4, warmup = 50 steps (5% of a 1,000-step run):
step 10 lr = 6.0000e-5 one fifth of the way up
step 25 lr = 1.5000e-4 half way
step 50 lr = 3.0000e-4 the peak — warmup ends
step 200 lr = 2.8192e-4 first cosine value (see chapter 04)
same peak, warmup = 200 steps (20%): step 200 is exactly the peak.
Bias correction from the optimizers lesson fixes the scale of Adam’s estimates — at step 1 it turns m = 0.1g into m̂ = g and v = 0.001g² into v̂ = g² — but it cannot fix the variance: v̂ at step 1 is built from a single squared gradient. One sample of a noisy quantity is not a statistic. Until a few batches have been seen, the per-coordinate step size lr·m̂/(√v̂ + ε) is at the mercy of whichever gradient happened to arrive first, and a full-size step in that direction can move the whole network before any evidence has accumulated. Warmup simply refuses to take full-size steps until the evidence exists.
How long is enough? The field’s answer is small: 1–5% of total steps. Llama 3 warmed up for 2,000 steps on a ~1.8-trillion-token run — a rounding error in run length — and GPT-3 warmed up over 375 million tokens. The reason such a short ramp suffices is the arithmetic in the margin: the ramp is a triangle, so it spends only half of the steps it occupies at partial size.
Warmup, measured on a real training loop
The source’s circle network, trained with Adam at the same peak, the same initial weights and the same data — the only difference is how long the ramp lasts. No warmup ends 19.0× higher than the best warmup run here.
peak 0.10 · seed 0 · 20000 steps · selected: 0% · none
warmup steps end loss best loss accuracy
0% · none 0 0.06506 0.04723 93.0%
1% 200 0.009136 0.009136 99.5%
5% 1000 0.003426 0.003426 99.5%
10% 2000 0.003653 0.003653 99.5%
20% 4000 0.01267 0.01267 99.5%
no-warmup gap: 19.0× above the best of the ramped runs
warmup costs half a triangle: ½ · T_w · peak = 0 lr·steps.
The whole 100-epoch run spends about 1000 lr·steps, so this warmup
costs 0.0% of the budget.
Honest scale caveat: this is a 2-8-1 network on 200 points. On such an easy task the source’s own Python sweep showed all warmup lengths within 11% of each other. The trend still appears once Adam takes full-size steps into cold statistics — and grows with model size. Treat warmup as cheap insurance, not a speed trick.
The cost of warmup is half a triangle
Because the ramp is linear from 0 to the peak, its cumulative learning rate is the area of a triangle, not a rectangle. That is the entire cost calculation.
cost of warmup = ½ · T_w · peak (learning-rate × steps)
cost of the whole warmup+cosine run ≈ ½ · peak · T
warmup as a fraction of the run's budget = T_w / T
T_w = 1% of T costs 1% of the budget
T_w = 5% of T costs 5% of the budget
T_w = 20% of T costs 20% of the budget
concrete: peak 0.1, T = 20,000 steps, T_w = 1,000 (5%):
warmup = ½ · 1000 · 0.1 = 50 lr·steps
the run ≈ ½ · 0.1 · 20,000 = 1,000 lr·steps → 5% premium
measured end losses (Adam, source circle net, seed 0):
0% 0.06317 93.0% accuracy ← the cold start wins early, loses late
1% 0.02487 97.5%
5% 0.01132 99.0%
10% 0.00322 100.0%
20% 0.01042 99.0%
across weight seeds 0–3: 0% ends 0.059–0.093; every warmed run 0.002–0.025.
Two honest observations. First, the no-warmup run is ahead for roughly the first ten epochs — it is not slower because it is confused, it is faster because it takes bigger steps — and then it stalls at a much worse loss. Warmup is insurance against a bad early trajectory, not a speed trick. Second, the exact best warmup length moves with the seed: past 5% the ordering in the table above is noise on a network this small. What is robust is the gap between some warmup and none.
Quick check
Bias correction already fixes Adam's zero-initialised moments at step 1. Why does warmup still help?
04
THE MODERN DEFAULT
Warm up, then glide. Two lines, every transformer.
Linear warmup followed by cosine decay is what Llama, GPT, PaLM and virtually every modern transformer use. Warmup protects the fragile first steps; the cosine spends the largest rates through the middle where most learning happens and lands softly at the floor. No milestones, no cliffs — just three numbers.
The whole schedule is one if-statement:
if t < T_w: lr(t) = lr_max · (t / T_w)
else: p = (t − T_w) / (T − T_w)
lr(t) = lr_min + ½ · (lr_max − lr_min) · (1 + cos(π · p))
peak 3e-4, T = 1,000 steps, T_w = 50, lr_min = 0:
step 10 lr = 6.0000e-5 ramp: 10/50 of the peak
step 25 lr = 1.5000e-4 ramp: halfway up
step 50 lr = 3.0000e-4 the peak — warmup ends
step 200 lr = 2.8192e-4 p = 150/950 = 0.158 → cos = 0.8794
step 525 lr = 1.5000e-4 p = 475/950 = 0.5 → exactly half the peak
step 1000 lr = 0 p = 1 → cos π = −1 → the floor
The checkpoints are worth memorising because they are how you debug a schedule with a printout. At a quarter of the way through the run the cosine is still near the peak (gentle start); at half it is exactly halfway down; at three-quarters it is near the floor (gentle finish). The steepest descent happens in the middle — precisely where the source says most learning happens and where you want the step size to still be serious.
The schedule race on a real network
The source’s five-way comparison, recomputed live: the same 2-8-1 network, same 200 points, per-sample SGD, each schedule with the same peak. The source ran this at peak 0.05, where the task is so easy every schedule ties. We include 0.5 so the constant rate’s plateau is visible.
at epoch 149
constant 0.01353
step decay 0.01164
cosine 0.007791
warmup + cosine 0.007998
1cycle 0.005518
final (epoch 149) best accuracy
constant 0.01353 0.01353 98.0%
step decay 0.01164 0.01164 100.0%
cosine 0.007791 0.007791 100.0%
warmup + cosine 0.007998 0.007998 100.0%
1cycle 0.005518 0.005518 100.0%
constant ends 2.5× above 1cycle at this peak.
step decay: ×0.1 at 25%, 50% and 75% of training. warmup + cosine ramps over the first 5% of steps.
The scale lesson is the one the source’s own run makes: at peak 0.05 its constant schedule actually finished lowest (0.0054 vs 0.0082–0.0145 for the decayed runs) — on a tiny, easy task a well-chosen constant rate is hard to beat. At peak 0.5 on the same task the ordering flips and the decayed schedules end several times lower. The schedule’s value grows with the difficulty and length of the run.
One alternative deserves a line because of how many models used it: linear decay swaps the cosine for a straight line from the peak to the floor. BERT trained that way. It is the same idea with a different interpolation, and it is slightly cheaper to reason about — but the cosine’s gentle start buys back early progress that a straight line gives away, which is why cosine won.
Worked arithmetic — the endpoints, and the source's own run
The cosine branch has three values you can verify without a calculator, because cos(0) = 1, cos(π/2) = 0 and cos(π) = −1:
t = T (p = 1): 1 + cos(π) = 0 → lr = lr_min
t = T_w (p = 0): 1 + cos(0) = 2 → lr = lr_max
p = 0.5: 1 + cos(π/2)= 1 → lr = (lr_max + lr_min) / 2
p = 0.25: 1 + 0.7071 = 1.7071 → lr = lr_min + 0.8536·(lr_max − lr_min)
p = 0.75: 1 − 0.7071 = 0.2929 → lr = lr_min + 0.1464·(lr_max − lr_min)
so a warmup+cosine schedule at p = 0.25, peak 3e-4, lr_min 3e-5:
lr = 3e-5 + 0.8536 · 2.7e-4 = 2.6046e-4
And now the honest part, because it is more interesting than a victory lap. The source’s own Python run on its tiny 2-8-1 network ends like this — constant first, decayed schedules worse:
Both results are real, and the difference is the lesson: the schedule’s value scales with the difficulty of the run. On 200 easy points with 683 parameters, a well-chosen constant rate is hard to beat — the network converges long before any decay phase matters. On a run that is longer, larger, or more fragile (which is every transformer), the late phase is exactly where the difference between 90% and 95% lives, and the constant rate cannot reach it. Read the two tables as one sentence: schedules are insurance whose premium grows with the size of what you’re insuring.
Scientific honesty: the tiny-network comparison above is a teaching model, not evidence about schedules at scale. The claim that matters is structural: any run whose optimal step size shrinks over time — which is any run approaching a minimum — gains from a decay, and any run with a fragile start gains from a warmup. The published-model configurations in chapter 07 are the field’s empirical answer at scale.
05
1CYCLE & RESTARTS
Sometimes the schedule goes up first.
Leslie Smith’s discovery from 2018: ramp the learning rate from low to high over the first half of training, then back down. A high rate mid-run sounds like sabotage — until you see it as noise that helps the model explore, followed by a long, careful descent into whichever basin it found.
The 1cycle recipe is a triangle with curved edges, in two phases:
phase 1 (0 → T/2): lr ramps from lr_max/25 up to lr_max
phase 2 (T/2 → T): lr ramps from lr_max down to lr_max/10000
peak 0.05, T = 1,000 steps:
step 0 lr = 2.0000e-3 start at peak/25 — deliberately low
step 250 lr = 2.6000e-2 over half the peak on the way up
step 500 lr = 5.0000e-2 the peak, at halfway
step 750 lr = 2.5003e-2 halfway down
step 999 lr = 1.0499e-4 nearly the floor
step 1000 lr = 5.0000e-6 peak/10,000
If you recognise the up-ramp: yes, the first half is a warmup that never
stops warming up.
The theory in the source’s words: a high learning rate acts as regularization by adding noise to the optimization trajectory. The model explores more of the loss landscape during the ramp-up phase and finds better basins; the ramp-down then refines within the best basin found. In the lesson’s race at peak 0.5, 1cycle finishes at 0.0055 against the constant run’s 0.0135 — half the loss — and it gets there by being behind for the first fifty epochs. The trade is stated bluntly in the source: 1cycle often trains faster for a fixed compute budget, but you must know the total number of steps in advance.
Cosine annealing with warm restarts (SGDR) applies the same “explore, then settle” idea repeatedly: run a cosine to the floor, jump back to the peak, and run a longer cosine, doubling the cycle each time.
cycle i has length Tᵢ = T₀ · multⁱ, with mult = 2, T₀ = 100:
cycle 0: steps 0–99 (100 steps)
cycle 1: steps 100–299 (200 steps)
cycle 2: steps 300–699 (400 steps)
cycle 3: steps 700–1499 (800 steps)
peak 0.05, lr_min 1e-5:
t = 50 2.5005e-2 mid first cycle
t = 99 2.2334e-5 about to reset
t = 100 5.0000e-2 reset — the full peak again
t = 700 5.0000e-2 another reset
t = 1000 3.4570e-2 300 steps into the 800-step cycle
Every restart is a deliberate destabilisation: the model climbs out of the basin it settled into and explores again, with longer cycles late so it has time to reconverge. The cost is modest — the table in the derivation shows a 1.24× learning-rate budget premium over a single cosine — but it adds two hyperparameters, T₀ and T_mult, and it is the wrong tool for a short run.
Both of these schedules need a peak, and “find the best learning rate” is itself a measurement. Smith’s learning-rate range test: train for a few hundred steps while the rate grows exponentially from about 1e-7 to 1, record the smoothed loss, and read where it stops falling.
Find the peak with a range test
Leslie Smith’s measurement: train while the rate grows exponentially, and watch the smoothed loss. It falls, bottoms out, and climbs. The bottom is the largest rate the run tolerates; practice then backs off by three to ten times for the schedule’s actual peak.
minimum of smoothed loss 0.1494 at lr 5.929e-2 (step 494)
conservative peak (÷10) 5.929e-3
band shown on the chart 5.93e-3 – 1.98e-2
what happens next:
below the band the loss falls, slowly — you are paying epochs for safety
inside the band the usual choice for a warmup + cosine schedule's peak
above the minimum the smoothed loss climbs: the rate is too large even briefly
Adam keeps taking ≈ lr-sized steps even when the gradients saturate, so the rise after the minimum is clean and easy to read.
The ramp itself is the source’s Exercise 2: lr(t) = 1e-7 × 10^(7t/T). In our 600-step run it passes 1e-3 at step ≈343 and 1e-2 at step ≈429 — most of the sweep is spent in the useful range, which is why a few hundred steps are enough.
Worked arithmetic — the restart budget
Sum the schedule and compare it to a single cosine of the same length. With T₀ = 100, mult = 2, peak 0.05 and lr_min = 1e-5 over 1,000 steps:
full cycles inside 1,000 steps: 100 + 200 + 400 = 700 steps
cycle 3 contributes its first 300 of 800 steps
each complete cycle's mean ≈ (peak + lr_min)/2 ≈ 0.025
full cycles: 700 × 0.025 = 17.50
partial cycle 3: the cosine from u = 0 to 0.375 averages
≈ 0.05 × 0.892 = 0.0446 → 300 × 0.0446 = 13.38
total ≈ 30.9 lr·steps
single cosine of 1,000 steps: 0.5 × 0.05 × 1,000 ≈ 25.0 lr·steps
restart premium = 30.9 / 25.0 = 1.24×
The exact sum from the lesson’s implementation is 30.968 against 25.030 — the same 1.24×. A quarter more budget buys several fresh explorations, which is why the source’s Exercise 4 asks you to run the head-to-head on a long training run rather than trust the theory either way.
Quick check
A colleague argues: “raising the learning rate halfway through training must be wrong — by then the model is close to a minimum.” What is the defense of 1cycle?
06
SCHEDULES IN PRACTICE
Three couplings decide whether the plan survives contact.
A schedule does not exist alone. It is coupled to the optimizer it feeds, to the batch size that shapes every gradient, and to the question of whether you are training from scratch or nudging a pretrained model. Get the couplings wrong and the prettiest curve still fails.
The loop order. In PyTorch every schedule is an object that reads the optimizer’s step count. The source’s warning is the one bug everyone writes once: scheduler.step() must come afteroptimizer.step() in the same iteration, so the schedule adjusts the rate for the next step using a count that already includes the step just taken.
What precision looks like in PyTorch and Hugging Facepython
import torch.optim as optim
from torch.optim.lr_scheduler import CosineAnnealingLR
optimizer = optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.01)
scheduler = CosineAnnealingLR(optimizer, T_max=1000, eta_min=1e-5)
for step in range(1000):
optimizer.zero_grad()
loss = criterion(model(batch), targets)
loss.backward()
optimizer.step() # 1. move the weights
scheduler.step() # 2. then adjust the rate for the next step# the modern default, from Hugging Face:from transformers import get_cosine_schedule_with_warmup
scheduler = get_cosine_schedule_with_warmup(
optimizer,
num_warmup_steps=2000, # Llama 3's warmup, in steps
num_training_steps=100000,
)
The Hugging Face helper is literally the warmup + cosine formula from chapter 04 with the two parameters this lesson names: how long the ramp is, and how long the run is. When in doubt, use it with warmup = 3–5% of total steps.
The batch-size coupling. The source gestures at it — “when you increase batch size, the warmup period needs to change” — and the result behind that sentence is Goyal et al.’s linear scaling rule from the one-hour ImageNet run. A gradient from a batch of size k·B has roughly 1/k the noise of one from size B, so the step size can grow roughly k× to keep the same progress per weight-space unit. But bigger steps at the start are exactly what warmup exists to prevent, so the rule comes with a companion: scale the rate up, and make the warmup longer.
batch × k → lr ≈ × k (linear scaling rule)
→ warmup longer (a gradual ramp, ~5 epochs at large batch)
→ re-verify with a range test if k is large
optimizer coupling — the peak bands the field converged on:
SGD + momentum 0.01 – 0.1 needs a schedule most of all
Adam / AdamW 1e-4 – 3e-4 the modern default
fine-tuning 1e-5 – 5e-5 ~10–15× below the fresh-training band
from scratch vs fine-tune: the same architecture wants a 10–15× smaller
peak when the weights already encode a task.
The optimizer coupling. The same number means different things to different optimizers. Adam’s step is approximately lr in every coordinate once its statistics warm up, regardless of the gradient’s size; SGD’s step islr·gradient and therefore scales with the gradient. That is why the field’s peak band for AdamW is three orders of magnitude below the band for SGD with momentum — and why you should never port a learning rate between them, and why a schedule’s floor matters more under Adam: it is the only way anlr-sized orbit ever shrinks.
Build your own schedule
Choose a peak, a warmup length, a decay shape and a floor — the four decisions every schedule in this lesson makes. Watch the checkpoints and the budget line: the exponential curve is still at 37% of peak after a thousand steps, while cosine has already spent its budget halfway through.
warmup 2%, cosine to peak/10 — the modern transformer default
learning rate at checkpoints:
start step 0 0.000e+0
5% step 50 2.994e-4
10% step 100 2.956e-4
25% step 250 2.649e-4
50% step 500 1.693e-4
75% step 750 7.108e-5
end step 1000 3.000e-5
budget: mean lr 1.64×10^-4 = 0.548 × peak
0.1645 lr·steps over the run
a constant rate would spend 0.3000 lr·steps (1.000 × peak)
warmup cost: ½ · T_w · peak = 0.003000 lr·steps (1.8% of this plan)
The shape decides when the budget is spent, not just how much. Keep the peak honest with a range test, keep the warmup at 1–5% of steps unless the run is fragile, and set the floor from what the loss is still doing at the end — if it is still falling, lower lr_min; if it is bouncing, raise it.
Quick check
You quadruple the batch size and keep every other knob fixed. What does the linear scaling rule suggest, and what happens to warmup?
07
CHOOSING A SCHEDULE
Ask two questions. Then take the default.
The source’s flowchart reduces to: do you know the total number of steps, and how big is the compute budget? Everything else is the field’s accumulated experience about which shape survives which situation — and the experience is public.
The decision tree, exactly as the source draws it:
Choosing an LR schedule
│
├─ Do you know the total training steps?
│ ├─ no → constant LR with manual decay (and log it, chapter 06)
│ └─ yes → what is the compute budget?
│ ├─ large (days/weeks) → warmup + cosine decay ← Llama/GPT default
│ ├─ small (hours) → 1cycle policy ← fastest convergence
│ └─ moderate → cosine annealing ← safe default
│
final knobs:
warmup = 1–5% of steps
lr_max = from an LR range test
lr_min = lr_max / 10 for cosine (0 only if you truly finish at the floor)
The published configurations are worth reading as a table, because they show both the consensus and its exceptions. Llama 3, GPT-3 and BERT all warm up and decay — but with three different decay shapes and three different warmup lengths. ResNet-50, the vision workhorse, uses no warmup at all and a step decay. The lesson to take is match the recipe to the architecture and the scale, not that one shape rules them all.
The source’s flowchart, made clickable: what are you training, and do you know the total number of steps? Each answer lands on a field-tested recipe — with the failure it is guarding against.
WHAT ARE YOU TRAINING?
linear warmup + cosine decay
peak 3e-4 (6e-4 for GPT-3) · warmup 1–5% of steps · lr_min ≈ peak/10
The field's default because it needs no milestone tuning: warmup survives the cold start, cosine spends the largest rates mid-run where most learning happens, and the floor keeps a little movement while the loss settles. Llama 3 used 2,000 warmup steps and cosine from 3e-4 to 3e-5.
In the wild: Llama 3 (3e-4 → 3e-5, cosine), GPT-3 (6e-4 → 0, 375M-token warmup), PaLM.
Watch out: Do not copy the peak into a fine-tune — 3e-4 is 15× above the fine-tuning band and will spike a pretrained model.
scenario Pretraining a transformer / LLM
schedule linear warmup + cosine decay
numbers peak 3e-4 (6e-4 for GPT-3) · warmup 1–5% of steps · lr_min ≈ peak/10
why
The field's default because it needs no milestone tuning: warmup survives the cold start, cosine spends the largest rates mid-run where most learning happens, and the floor keeps a little movement while the loss settles. Llama 3 used 2,000 warmup steps and cosine from 3e-4 to 3e-5.
the checklist before you launch (in order)
1. range-test the peak: 600 steps, read the smoothed minimum, ÷3–10
2. set warmup to 1–5% of steps (longer if the run is fragile or batches are huge)
3. set lr_min to peak/10 for cosine; 0 only if you truly finish at the floor
4. scale the peak with batch size: k× batch → roughly k× the rate, longer warmup
5. run the loop in order: zero, forward, loss, backward, clip, optimizer.step,
scheduler.step — never schedule before step
the source's warning applies to schedules too: one change at a time, with a
baseline number in hand, or the next 10× win becomes an unattributable mystery.
The published table is the fastest way to sanity-check a peak: if your number is 100× away from the field’s value for the same architecture, the schedule is not the question — the units are.
One closing claim, straight from the source and worth carrying out of the lesson: if you tune nothing else, tune the learning rate. The schedule is the highest-leverage part of that knob, because it gets to choose the rate at every step instead of one rate for all of them. And the discipline is the same as everywhere else: change one thing — peak, warmup, floor, shape — measure against a baseline, and keep the run that wins. The next lesson wires all of these pieces into a small framework; this one is the piece that decides whether any of them converge.
08
CHECK YOURSELF
Five questions. Then the terms worth keeping.
Answer before you look. The warmup question and the 1cycle question are the two that separate memorised names from a mechanism you can debug when a run breaks at step 10,000.
0 / 5 answered · 0 correct
01Why is a constant learning rate usually suboptimal for training neural networks?
02What is the purpose of learning rate warmup?
03What learning rate schedule do Llama 3, GPT-3, and most modern LLMs use?
04What makes the 1cycle policy different from other schedules?
05If training loss suddenly spikes and diverges, what is the most likely learning rate issue?
Key terms, demystified
Click a card to swap the lazy description for what it actually means.
Exercises from the lesson
Four problems with exact numbers — compare exponential decay with cosine, run and read a range test, sweep the warmup length with its budget arithmetic, and implement SGDR warm restarts with their 1.24× premium. Try first; a worked answer is one click away.
Implement exponential decay lr(t) = lr₀·γᵗ with γ = 0.999 per step. Compare it to cosine annealing on the circle dataset at peak 0.05 over 2,000 steps: compute both learning rates at t = 100, 500, 1,000 and 2,000, and say when each schedule spends its learning-rate budget.Show one worked answer
The two curves, computed from the formulas: t = 100 → exp 4.524×10⁻², cosine 4.969×10⁻²; t = 500 → 3.032×10⁻² vs 4.268×10⁻²; t = 1,000 → 1.839×10⁻² vs 2.500×10⁻²; t = 2,000 → 6.760×10⁻³ vs the floor 1×10⁻⁵. Exponential decay is modestly ahead early (×0.905 per 100 steps), falls behind around t ≈ 1,150, and never finishes: reaching 1×10⁻⁵ takes ln(1e-5/0.05)/ln(0.999) ≈ 8,513 steps. Budget: the mean of γᵗ over 2,000 steps is (1 − 0.999²⁰⁰⁰)/(2000 × 0.001) = (1 − 0.1351)/2 = 0.4324 → 865 peak-equivalent steps; cosine's mean is 0.5005 → 1,001 peak-equivalent steps. So exponential spends 13.5% less learning-rate budget and leaves 13.5% of the peak on the table at the end. On the tiny circle network both cross 99% accuracy; the difference shows up on long runs, where the residual rate is what keeps the loss from settling into the lowest basin. One trap worth internalising: γ is per step, not per epoch. With 200 steps per epoch, 0.999 per step is ×0.819 per epoch; if you meant a 0.999 per-epoch decay, the per-step value is 0.999^(1/200) ≈ 0.999995.
Implement Leslie Smith's learning-rate range test: train for a few hundred steps while the learning rate grows exponentially from 1e-7 to 1, and plot the smoothed loss against the LR. Where does the loss stop falling, and what peak rate would you pick? (b) At which step does the ramp pass through 1e-3 and 1e-2?Show one worked answer
The ramp is lr(t) = 1e-7 × 10^(7t/T), so the step that hits a value v is t = T·log₁₀(v/1e-7)/7. For T = 600 steps: 1e-3 at t = 600 × 4/7 ≈ 343, and 1e-2 at t = 600 × 5/7 ≈ 429. The lesson's port — Adam on the circle network, 600 steps, loss measured on the full dataset every 2 steps and averaged over 25 samples — falls from 0.317 to a minimum of 0.1494 at lr = 5.93×10⁻² (step 494), then climbs: 0.1564 at 9.1×10⁻², 0.2097 at 1.6×10⁻¹, 0.3000 at 3.5×10⁻¹. So the direct reading of 'just before the loss increases' is ≈ 6×10⁻², and the conservative rule of thumb — divide by ten — gives ≈ 6×10⁻³. Both are usable: 6×10⁻² works as the *peak* of a warmup + cosine schedule (you only sit there briefly), while 6×10⁻³ is safe as a constant rate. Two caveats. First, smoothing matters: the per-step loss is far too noisy to read directly. Second, use the optimizer you will actually train with — in this port the same ramp under plain SGD never showed a clean rise, because the tiny network's saturated sigmoid froze its gradients at a decent state; the test is a measurement of *your* setup, not a universal recipe.
Train with warmup + cosine at peak 0.1 for 100 epochs (20,000 steps), varying the warmup length over 0%, 1%, 5%, 10% and 20% of training. Compute each warmup's cost in learning-rate steps, predict the trend, then compare with the measured end losses from the lesson's Adam experiment (seed 0).Show one worked answer
Cost arithmetic first. The warmup's cumulative learning rate is the area of a triangle: ½·T_w·peak. For T_w = 0, 200, 1,000, 2,000 and 4,000 steps and peak 0.1 that is 0, 10, 50, 100 and 200 lr·steps. The whole warmup + cosine run spends about ½·peak·T = 1,000 lr·steps, so each warmup costs its length as a fraction of the run — 1%, 5%, 10% and 20% of the budget, exactly. Measured end losses (Adam, the source's circle network, seed 0): 0% → 0.0632 (93.0% accuracy); 1% → 0.0249 (97.5%); 5% → 0.0113 (99.0%); 10% → 0.00322 (100%); 20% → 0.0104 (99.0%). Every warmed run ends 2–20× lower than no warmup while staying within a few accuracy points of the others — the claim to trust is 'some warmup', not 'this exact length'. Re-running across weight seeds 0–3 widens the gap for 0% (0.059–0.093) and leaves the warmup runs in the 0.002–0.025 band with a noisy ordering past 5%. For comparison, the source's own Python sweep on its tiny network showed essentially no difference between any warmup length (0.00836–0.00881) — the honest boundary condition: warmup is insurance whose premium only pays when early training is actually fragile, and it is cheap enough that the field buys it by default.
Implement cosine annealing with warm restarts (SGDR): peak 0.05, lr_min 1e-5, first cycle T₀ = 100 steps, doubling each cycle (T_mult = 2). Compute the LR at steps 50, 99, 100, 299, 300, 700 and 1,000, and the total learning-rate budget over 1,000 steps; compare with a single cosine over the same 1,000 steps.Show one worked answer
The cycle boundaries are [0, 100), [100, 300), [300, 700), [700, 1,500): each cycle has length Tᵢ = 100·2ⁱ, and inside a cycle lr(t) = lr_min + ½(peak − lr_min)(1 + cos(π·local/Tᵢ)). Values: step 50 (mid first cycle) → 0.025005; step 99 (almost over) → 2.2334×10⁻⁵; step 100 (reset) → 0.05; step 299 → 1.3084×10⁻⁵; step 300 (reset) → 0.05; step 700 (reset) → 0.05; step 1,000 is 300 steps into the 800-step third cycle, so local/T = 0.375 and lr = 0.03457. Budget: summing the schedule over 1,000 steps gives 30.97 lr·steps against 25.03 for the single cosine — a 1.24× premium. What the premium buys is escape velocity: every restart pushes the LR back to the peak for a while, so the model can climb out of the basin it settled into and explore again before settling lower. The doubling keeps the late cycles long enough that the loss has time to reconverge. The costs are two extra hyperparameters, T₀ and T_mult, and a schedule that is deliberately destabilising — evidence of benefit is strongest in long runs, and the source's Exercise 4 recommends exactly this head-to-head with standard cosine.
Terms this lesson borrows from later lessons (or outside)
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
gradient descent — The loop that nudges parameters downhill. This lesson only changes one term inside it — the scalar multiplying the gradient — but that scalar is the difference between a run that settles and a run that explodes. (Phase 1, Lesson 08)
mini-batch — A random subset of the data used to estimate the gradient. Its size changes the noise in every step — and therefore which learning rate is safe, and how much warmup a large-batch run needs. (Phase 1, Lesson 08)
bias correction — Adam's division of its zero-initialized moments by (1 − β^t). It is warmup's mathematical sibling: both exist because the first steps are taken with statistics that have not been earned yet. (Phase 3, Lesson 06)
loss surface / ravine — The landscape the optimizer descends. Its curvature sets the stability limit lr < 2/c and its noise sets the floor a constant rate can reach — the two facts that make schedules necessary. (Phase 3, Lesson 06)
weight initialization — The choice of starting weights, and the source of the very first gradients. A run with badly scaled initialization produces the large, unrepresentative early updates that warmup exists to survive. (Phase 3, Lesson 08)
KEEP GOING
A picture is a start. Practice is the rest.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 03, Lesson 09) and the Math Foundations Notebook reference build. The seven labs (noisy-quadratic fates on a constant rate, the six-shape schedule gallery with budgets, the measured Adam warmup experiment, the five-schedule race at two peaks, the learning-rate range test with smoothing, the schedule composer, and the schedule decision board) are original to this page, as are the stability and noise-floor derivation with its three numeric checks, the cosine checkpoint table at 25/50/75 of 100, the step-decay staircase, the exponential-versus-cosine comparison and its per-step vs per-epoch trap, the warmup half-triangle cost arithmetic, the SGDR cycle table and 1.24× budget check, the warmup-length sweep, and the note that the source's own Python run has a constant rate winning on its tiny task. Every number shown is computed live by the labs or verified by hand in the prose.