That gap is not bad luck — it is memorisation, and a big enough network will happily do it on data with no pattern at all. This lesson is the tax office: weight decay, dropout, normalization, augmentation and early stopping, each one a different way to buy generalization by giving up a little training accuracy.
A big enough network can drive training loss to zero on completely random labels — Zhang et al. showed it on ImageNet in 2017. So a perfect training score proves nothing except that the model had the capacity to memorise. The number that matters is the gap: 99.9% train against 65% test is a 34.9-point alarm.
99.9 − 65 = a 34.9-point overfitting gapWeight decay keeps any single weight from growing large. Dropout stops units from co-adapting. Normalization makes the loss landscape smoother so the optimizer finds flatter minima. Augmentation makes memorising one view useless. Early stopping ships the checkpoint at the validation minimum instead of the last one.
magnitude · redundancy · landscape · views · timeEvery technique trades capacity for generalisation: push it too far and the model underfits, which is just as wrong as overfitting. The source's rule of thumb — gap over 10% heavy, 5–10% moderate, under 5% leave it alone — is a starting dose. Then you test one change at a time against the held-out score, because that is the only referee that is not fooled.
gap > 10% · 5–10% · < 5%Regularization is a tax on complexity: you surrender a little training accuracy so the model stops paying attention to noise — and the held-out score, never the training score, is the ledger you balance.
By the end you will be able to read a train–validation gap and say which side of the bias–variance dial you are on; compute one weight-decay step and the half-life of a shrinking weight; explain why L1 deletes weights while L2 only shrinks them; run dropout with inverted scaling and prove its expected value is unchanged; compare BatchNorm, LayerNorm and RMSNorm with actual numbers; pick an early-stopping patience and count the epochs it saves; and choose a defensible regularization budget instead of sprinkling techniques you cannot justify.
A model that scores 99% on the data it trained on has told you almost nothing. The number that matters is the other one — the score on data it has never seen — and the distance between them is the overfitting gap that every technique in this lesson exists to close.
Every model sits somewhere on a spectrum. Too simple and it cannot capture the pattern at all; that is underfitting, and its signature is a small gap with a low score on both sides. Too complex and it starts fitting the particular noise of the training set; that is overfitting, and its signature is a big gap. The sweet spot is in between, and regularization is how you push a model toward it from the overfit side.
Read the middle row again: the good model is worse on the training set than the overfit one, and that is the point. Training accuracy is a promise the model makes to data it has already seen. Held-out accuracy is the evidence about data it has not. Optimizing the promise instead of the evidence is the single most common way a beginner ships a model that fails in production.
Slide the capacity dial right and watch the two curves split. Then raise λ — the tax on complexity — and watch the validation curve flatten while the training curve pays a little. The random-label switch is the Zhang et al. experiment in miniature: capacity still drives training loss down, and validation never learns anything.
Curves are a simplified teaching model, not a trained network — the shape is what matters: validation loss is U-shaped, training loss is not. Regularization raises the left end of the U and lowers the right end, which is why it works best when you are on the overfit side.
The extreme case is not a thought experiment. Zhang et al. (2017) took standard image classifiers, replaced every label in ImageNet with a random one, and trained. The networks drove training loss to near zero — they memorized a million random input-output pairs — while held-out accuracy never rose above chance (0.1% for 1,000 classes). There is no pattern in random labels, so the only way to fit them is to memorize. With enough parameters, that is exactly what a network does.
This is why “my training accuracy is great” is not evidence of anything except capacity. GPT-3 has 175 billion parameters trained on roughly 500 billion tokens of text, which is plenty of capacity to regurgitate passages verbatim if nothing stops it. Scale makes the gap problem more pressing, not less: the bigger the model, the more it can memorize, and the more deliberately you have to make memorization unprofitable.
Suppose validation night gives you these three runs and nothing else. The gap column is the diagnosis; the held-out column is the verdict.
Model C is the trap. Its 99.9% training score is the highest of the three and the least useful: 34.9 points of that skill exist only on training data. The diagnosis is mechanical — a gap over 10% means heavy regularization, 5–10% moderate, under 5% leave the model alone — and the practice is a loop: measure the gap, change one thing, measure again.
A run reports 99.4% training accuracy and 61.2% validation accuracy. What is the most accurate diagnosis?
The takeaway: the gap is the symptom, and it can be large in either direction of the complexity dial. Everything from here on is a different way to tax complexity — and each one pays for generalisation with a little training accuracy.
Weight decay adds the squared size of every weight to the loss. A weight that is not earning its keep now costs something, every step, and the optimizer will shrink it toward zero. That is the whole mechanism — and it is the scalar one-step arithmetic you can do in your head.
The source’s two functions are the entire idea: add a penalty equal to half the squared magnitude of all the weights, and take its gradient. The task loss still does its job; the penalty adds a second force that pulls every weight toward zero, proportional to how large that weight currently is.
The second line is the one to internalise: with no task gradient at all, weight decay multiplies every weight by the same factor (1 − lr·λ) each step. A weight of 4.0 with lr = 0.1 and λ = 0.01 becomes 4.0 × 0.999 = 3.996 after one step. Nothing dramatic happens in one step — which is exactly why it works: a small, relentless pull toward small weights, applied thousands of times.
Why would small weights generalise better? An overfit model is one that has amplified directions in the data that happen to separate the training examples — including the noise. Large weights are the amplifier: a small change in an input produces a big swing in the output. Keeping weights modest limits the model’s effective capacity and forces it to rely on robust features that move the output together, instead of memorized quirks that each need one big weight to fire.
Ten noisy samples, six held out, one smooth curve underneath. Turn the degree up and the fit chases noise — training error falls toward zero while held-out error climbs. Then raise λ and watch the side panel bend back: the same model, told to keep its weights small.
Same sixteen points every run (fixed seed). The side panel is the whole argument for weight decay in one picture: for a fixed degree, held-out error is U-shaped in λ, and the minimum moves as you change capacity. The data and curve are a simplified teaching model.
First the ledger, on the source’s five weights with λ = 0.01. The penalty is (λ/2)·Σw², and each weight’s gradient contribution is λ·w:
Now one real training step, on a single weight w = 4.0 with a task gradient of 0.50, learning rate 0.1, λ = 0.01:
The last two lines are the intuition for λ. The decay removes a fixed fraction of each weight per step, so a weight left to the penalty alone halves every 693 steps — but 1,000 steps of pure decay has only brought 4.0 down to 1.47, because 0.1% per step compounds slowly. Real training runs have thousands to millions of steps, which is when that gentle pull becomes decisive. Raise λ and the half-life shortens in proportion.
| Setting | Typical λ | Why |
|---|---|---|
| AdamW · transformers | 0.01 | decoupled decay; the modern LLM default |
| SGD · CNNs | 1e-4 | smaller, because SGD has no adaptive rescaling |
| Heavily overfit runs | 0.1 | a big tax for a big gap — watch for underfitting |
A weight has no task gradient at all on this step. With w = 2.0, λ = 0.01 and lr = 0.1, what does one step of weight decay do to it?
The takeaway: weight decay is a contract — every weight pays lr·λ·w per step for the right to exist, large weights pay more, and the model ends up with only the weights it actually needs. The polynomial playground above is the same trade in one picture: crank λ and the wild fit straightens out.
Weight decay shrinks weights smoothly and never quite reaches zero. Swap the square for an absolute value and the picture changes: the penalty becomes a flat tax, and small weights run out of money. This is the difference between shrinking a model and pruning it.
L1 regularization adds λ · Σ|wᵢ| to the loss instead of (λ/2) · Σwᵢ². The derivative is not λw this time — it is λ · sign(w), which is constant in magnitude: every nonzero weight is pushed toward zero by the same amount, lr·λ per step, no matter how large or small it is. Large weights barely notice; small weights get wiped out. That is why L1 produces exact zeros while L2 only approaches them.
In two dimensions the geometry makes it obvious. The set of weights with a fixed L2 budget is a circle; with a fixed L1 budget it is a diamond with corners sitting on the axes. The regularized solution is where the smallest task-loss contour touches the budget shape — and the diamond’s corners are always waiting there. Touch a corner and a whole weight is zero. The circle has no corners, so the contact point almost never lands on an axis; L2 shrinks, L1 selects.
The ellipses are the task loss; the dashed shape is the penalty budget. The solution is where the smallest untouchable ellipse touches the budget. Squares have corners; circles do not — that single geometric fact is the difference between shrinking and deleting.
A simplified teaching loss (a tilted quadratic) and a two-weight model, chosen so the geometry is visible. The trajectories start at (0, 0): L1 uses the exact soft-threshold step, L2 folds the shrinkage into the gradient. The source’s exercise connects this picture to feature deletion — L1 is why Lasso selects.
The constant step is not an approximation. Differentiate the penalty term directly and plug in two weights of different sizes:
The practical reading: L1 is a sparsity regularizer, useful when you want the model to declare that a feature or a unit is unnecessary. L2 is a smoothness regularizer, useful when you want every weight a little smaller but none gone. (At w = 0 the L2 update is exactly 0 too, so a zero would stay zero — but in real training a task gradient is always pulling, and L2 only approaches the origin asymptotically while L1 lands on it.) In practice most deep networks use L2/weight decay, because a zero inside a layer can be undone by the next layer’s weights — but L1 is the reason Lasso can delete features, and the pairing of the two (elastic net) is a standard toolkit entry.
The takeaway: the shape of the penalty is the shape of the solution. Squares give you smooth shrinkage; absolute values give you corners, and corners give you zeros. You met this geometry first as Lasso feature selection (Phase 2, Lesson 18) — inside a deep network, it is the same picture with millions of weights.
On every training pass, delete each unit with probability p — then multiply the survivors by 1/(1 − p). That second half is what makes the trick work: the network trains under random damage, and inference needs no change at all.
The source’s Dropout class does exactly three things. In training mode it rolls a Bernoulli mask: each output survives with probability 1 − p, dies with probability p. Dead units become 0.0. Living units are divided by 1 − p. In eval mode it returns the inputs untouched. The backward pass applies the same mask, scaled the same way, so a dropped unit receives exactly zero gradient for that step.
Why random deletion regularizes anything? Because it makes co-adaptation unprofitable. A unit that learns to fire only when a specific partner unit is present is betting on a partner that disappears on half the passes; the loss punishes that bet. The network is pushed toward redundant representations — several units that can each do the job — which is exactly the kind of robustness that transfers to unseen data.
There is a bigger way to see it. A layer with N units has 2^N possible on/off patterns, so dropout is training a different subnetwork on every pass, all sharing the same weights. 10 units give 2¹⁰ = 1,024 subnetworks; 16 units give 65,536; 100 units give about 1.27 × 10³⁰. At test time you use every unit, and the scaling makes the full network’s output approximately the average of all those subnetworks — a gigantic ensemble for the price of one model.
Every training pass rolls a new mask: each unit is zeroed with probability p, and the survivors are multiplied by 1/(1 − p). The dashed lines are the expected values — and they land exactly on the eval bars. That is the whole trick: the network sees a different subnetwork each step, and inference needs no change at all.
The mask is re-rolled every 1.25 s from a deterministic sequence, so every reader sees the same passes. In the source’s Python, the backward pass multiplies by the same mask and the same 1/(1 − p) — dropped units receive exactly zero gradient for that step.
Take one unit whose no-dropout activation at test time is 3.0. Under a mask, it either survives (and gets scaled) or dies (and outputs 0). Watch the expected value:
The same algebra runs backward. The gradient arriving at the unit is multiplied by mask / (1 − p): a surviving unit contributes its usual gradient amplified by 1/(1 − p), a dead unit contributes 0. Averaged over masks, the expected gradient matches the no-dropout gradient — so the updates, like the outputs, are unbiased.
A unit's activation at test time should be 4.0. During one training pass with p = 0.25 the unit survives the mask. What does inverted dropout output, and why?
The takeaway: dropout is noise that cannot be memorized. The mask changes every step, the expectation is preserved exactly, and the network ends up redundant on purpose — a hallmark of models that generalize.
BatchNorm, LayerNorm and RMSNorm all rescale activations to a predictable size — they just disagree about which numbers count as “the group”. That disagreement decides which one can run at batch size 1, which one transformers use, and which one is 10% cheaper.
Batch normalization looks down each feature column of a mini-batch: it computes the mean and variance of that feature across the batch, normalizes every value with them, then applies two learnable parameters — a scale γ and a shift β — so the layer can undo the normalization if that is what the network wants. During training those statistics come from the current batch; during inference they come from running averages accumulated with momentum 0.1 (90% old value, 10% new batch), because inference batches may not resemble training batches at all.
The original paper credited “internal covariate shift” — the idea that each layer keeps chasing a moving input distribution. Santurkar et al. (2018) showed that explanation is wrong: the benefit is that the loss landscape gets smoother. Gradients become more predictive and the optimizer can take larger steps safely, which is why BatchNorm lets you raise the learning rate instead of babysitting it. The cost is the batch dependency: with batch size 1 the mean is the sample itself and the variance is zero, so x̂ = (x − x)/σ = 0 and the layer can only output β. Small batches (< 32) give noisy, unhelpful statistics.
Layer normalization fixes that by switching the axis. It normalizes across the features of one sample: mean and variance come from that sample alone, no other row is involved. The computation is now identical in training and inference, independent of batch size, and safe for variable-length sequences — which is precisely why transformers use it. RMSNorm takes the next step and removes the mean subtraction entirely: divide by the root mean square, multiply by γ, no β. The centering step turns out to contribute very little to accuracy while costing computation, so RMSNorm matches LayerNorm quality with roughly 10% less overhead — enough, at LLM scale, that LLaMA, LLaMA 2/3 and Mistral all use it.
Edit sample 1 and watch which axis each method normalizes. BatchNorm reads down the column and depends on the batch; LayerNorm reads across the row and does not. RMSNorm is LayerNorm without the centering — its output has rms 1 but not mean 0. Switch the batch to one sample and BatchNorm has nothing left to measure.
RAW BATCH · rows = samples, columns = features
| f1 | f2 | f3 | f4 | |
|---|---|---|---|---|
| s1 | 1.000 | 3.000 | 5.000 | 7.000 |
| s2 | 3.000 | -1.000 | 0.000 | 2.000 |
| s3 | 5.000 | 0.000 | 4.000 | 1.000 |
| s4 | 7.000 | 3.000 | 2.000 | 3.000 |
BATCHNORM OUT · γ 1, β 0
| f1 | f2 | f3 | f4 | |
|---|---|---|---|---|
| s1 | -1.342 | 0.980 | 1.172 | 1.646 |
| s2 | -0.447 | -1.260 | -1.432 | -0.549 |
| s3 | 0.447 | -0.700 | 0.651 | -0.988 |
| s4 | 1.342 | 0.980 | -0.391 | -0.110 |
column stats f1 · μ 4.000 · σ² 5.000
LAYERNORM OUT · sample 1 · γ 1, β 0
| f1 | f2 | f3 | f4 | |
|---|---|---|---|---|
| s1 | -1.342 | -0.447 | 0.447 | 1.342 |
row stats · μ 4.000 · σ² 5.000 · mean(out) 0.000
RMSNORM OUT · sample 1 · γ 1
| f1 | f2 | f3 | f4 | |
|---|---|---|---|---|
| s1 | 0.218 | 0.655 | 1.091 | 1.528 |
rms(in) 4.583 · mean(out) 0.873 · rms(out) 1.000
BatchNorm is the odd one out: its numbers change when you change the other samples, and with a batch of one it can only output β. That is why transformers — which often run with batch size 1 during generation and variable-length sequences — normalize across features instead, and why modern LLMs drop LayerNorm’s mean subtraction for RMSNorm’s ~10% savings.
Take the row [1, 3, 5, 7] and run it through all three. The numbers are small enough to check by hand; they are also exactly what the lab above displays.
Two details worth noticing. LayerNorm output really is zero-mean unit-variance before γ and β — the lab’s readout shows mean 0.0000 and rms 1.0000 for it. RMSNorm is not zero-mean: on this row the output mean is 0.8729. What it keeps is the scale guarantee, and the source’s measurements found that is most of the benefit. The running-statistics trace is the other half of the design: inference does not use today’s batch at all, which is why forgetting model.eval() makes a BatchNorm model misbehave.
| Method | Normalizes across | Batch size 1? | Train vs eval | Typical home |
|---|---|---|---|---|
| BatchNorm | the batch, per feature | meaningless — output is β | different (batch vs running stats) | CNNs, large batches |
| LayerNorm | the features, per sample | fine | identical | transformers |
| RMSNorm | the features, per sample, scale only | fine | identical | modern LLMs (LLaMA, Mistral) |
The takeaway: normalization regularizes by making the landscape smoother and the activations predictable, not by shrinking anything. Choose the axis that survives your deployment conditions: batch for CNNs with big batches, features for sequences, and features without the mean for the largest models.
Training loss falls forever. Validation loss falls, bottoms out, and climbs. Early stopping just refuses to keep training past the bottom — and ships the checkpoint from the minimum instead of the end.
The procedure is four lines of bookkeeping. After every epoch, evaluate on the validation set. If the validation loss is the best so far, save a copy of the weights and reset a counter. If it is not, increment the counter. When the counter reaches the patience — typically 5 to 20 epochs — stop training and load the saved copy. The model you ship is the one from the bottom of the validation curve, not the one from the end.
It is “free” in the sense that it adds no hyperparameter to the loss and no change to the model — but it is not free of judgment. Patience is the one dial: too small and noise in the validation curve stops you before a real improvement; too large and you pay for epochs you know are useless. And the validation set must stay honest: stopping on the test set turns the test set into a training signal, and your final number becomes fiction.
Training loss only ever falls. Validation loss falls, bottoms out, then climbs as the model starts memorising noise. Pick the patience — how many epochs you are willing to wait for a new best — and watch the green zone: epochs you never have to run, with the best checkpoint already saved from the bottom of the curve.
The curve is generated from a fixed formula with per-run noise (seeded), so every reader gets the same five runs. Switch seeds to see the honest part: the best epoch is never known in advance — 48, 48, 57, 57, 55 for seeds 1–5 with patience 20. Early stopping ships the checkpoint, not the last state.
A synthetic run, built to show the arithmetic (not a measured log): the budget was 1,000 epochs, patience 20, and the validation curve looked like this at a few checkpoints.
Two things to notice. First, the saved compute is not the point — the epoch-340 weights score better on held-out data than any later checkpoint, and shipping the epoch-360 weights would hand over 0.0140 more validation loss for nothing. Second, patience costs epochs on purpose: training continues for 20 epochs after the best one precisely so that a later, genuine improvement is not missed. That is the entire meaning of the dial.
Validation loss is lowest at epoch 12 (0.284). Training continues to epoch 20, where patience 8 triggers a stop and validation loss is 0.351. The team runs to epoch 40 anyway and finishes at 0.402. Which weights do you ship?
The takeaway: the best epoch is unknowable in advance, which is why you save the checkpoint and let patience decide. It pairs with every other technique in this lesson and belongs in every training script by default.
Data augmentation is regularization without touching the model: transform the inputs while preserving the label, and memorising any single view stops paying. It is the largest free lunch in vision — and it comes with the same tax as everything else.
The source’s lists are the standard catalogue. For images: random crop, flip, rotation, colour jitter, cutout. For text: synonym replacement, back-translation, random deletion. For audio: time stretch, pitch shift, added noise. The mechanism is identical in each case — produce a new input that still deserves the same label, so the model is forced to learn the structure that survives the transform instead of the particular pixels, words or samples in front of it.
A model that sees each image once can memorize it. A model that sees fifty augmented versions of each image cannot — the only thing common to all fifty is the invariant content, and that is what gets learned. The effect is exactly a regularization effect: the effective size of the training set grows without collecting a single new example, which is why augmentation is the first thing to reach for on a vision task with a large gap. The subtlety is that the transform must preserve the label. Flipping a 6 to make a 9, or rotating a digit past legibility, teaches the model an invariance that is not true; that is worse than no augmentation at all.
Data augmentation is regularization without touching the model: every epoch shows a different view of the same image, so memorising the original is useless. Each transform teaches one invariance — flip, tilt, position, noise, scale. None of them changes the label.
The digit is a stylised illustration drawn in code, not a real dataset image, and the transforms are the source lesson’s list applied to it. The same idea covers text (synonym replacement, back-translation) and audio (time stretch, pitch shift) — the mechanism is always “same label, different input”.
Label smoothing is the fourth kind of augmentation: instead of transforming inputs, it transforms targets. A one-hot label claims 100% certainty; soften it with ε = 0.1 across 10 classes and the target becomes 0.9 for the correct class and 0.1/9 = 0.0111 for each of the other nine. What that does to the loss is measurable:
That inversion is the entire regularizing effect: with hard targets, more confidence is always rewarded, so logits can sprawl; with smoothed targets, confidence beyond the target is penalized, so the model settles at a calibrated 0.9. GPT and most modern models use label smoothing or an equivalent (Phase 3, Lesson 05), and it is the natural companion to dropout — dropout fights variance in the features, smoothing fights overconfidence in the output.
None of these tools should be used blindly. The source’s prescription flow keys the dose to the gap you measured, and every row is a budget: you are trading some training accuracy for held-out accuracy, and past a point the trade turns negative (underfitting).
| Measured gap | Level | Budget |
|---|---|---|
| over 10% | Heavy | dropout p 0.3–0.5 · weight decay 0.01–0.1 · aggressive augmentation · early stopping |
| 5–10% | Moderate | dropout p 0.1–0.2 · weight decay 0.001–0.01 · BatchNorm or LayerNorm |
| under 5% | Light | dropout p 0.05–0.1 · weight decay 1e-4 · early stopping only if the curve turns |
The source lesson’s prescription flow, turned into a board. Dial in the train–validation gap you actually measured, and read the budget: what to switch on, at what strength, and what each technique costs you if you overshoot. The gap decides the dose.
| Technique | Constrains | Main risk |
|---|---|---|
| Dropout | co-adaptation: no unit can rely on another | too high and the model underfits; transformers use p 0.1, MLPs p 0.5, CNNs p 0.2–0.3 |
| Weight decay | weight magnitude: no single weight dominates | decays biases and norm gains if applied blindly; use AdamW for Adam runs |
| Augmentation | effective dataset size: views, not memorisation | a transform that changes the label teaches the wrong invariance |
| Normalization | activation scale: smoother loss landscape | BatchNorm breaks at batch size 1 and with variable-length sequences |
| Label smoothing | confidence: targets stop being 100% certain | too much smoothing caps confidence the model legitimately needs |
| Early stopping | training time: ship the best checkpoint, not the last | free; the only cost is keeping a copy of the best weights |
The board is a teaching summary of the source’s “when to apply what” flow — the numbers are typical ranges, not laws. The only reliable loop is: measure the gap, add one technique, measure the gap again.
The takeaway: augmentation and label smoothing both change the data rather than the model, which is why they can be added on top of weight decay and dropout without fighting them. In practice they are the first two levers a vision team touches, and the budget board above is how the source decides when that team should reach further.
Answer before you look. The dropout-scaling question and the LayerNorm-versus-BatchNorm question are the two that separate “I read about regularization” from “I can debug a training run”.
0 / 5 answered · 0 correct
01What is overfitting in neural networks?
02How does dropout regularize a neural network?
03Why do transformers use LayerNorm instead of BatchNorm?
04What is the key difference between RMSNorm and LayerNorm?
05Why is it critical to call model.eval() before running inference in PyTorch?
Click a card to swap the lazy description for what it actually means.
Four problems with exact numbers — spatial dropout arithmetic, a label-smoothing loss comparison, an early-stopping checkpoint patrol, and LayerNorm versus RMSNorm by hand. Try first; a worked answer is one click away.
Expected active units are the same either way: 8 × (1 − 0.5) = 4.0. The structure is what differs. Standard dropout gives each channel the probability 0.5⁴ = 0.0625 that all four of its units survive (and 0.5⁴ = 6.25% that all four are zeroed), so channels are chopped into random fragments — the network is pushed to make every individual unit redundant. Spatial dropout rolls one mask per channel: each channel survives intact with probability 0.5, and when it survives all four units pass (scaled by 1/(1 − 0.5) = 2), which is 8× more likely to keep a channel whole. The redundancy pressure moves from units to channels: the network must not rely on any single feature map, but inside a live map the spatial structure (edges, textures) stays coherent. That is why spatial dropout is the standard choice in convolutional networks — unit-level masking destroys the spatial correlations convolutions exist to exploit. In the backward pass, spatial dropout multiplies by the shared per-channel mask (and the same factor 2), so whole channels receive zero gradient together.
Softmax: e⁴ = 54.598, e² = 7.389, e⁰·⁵ = 1.6487, total 63.636 → p = [0.8580, 0.1161, 0.0259]. Hard target [1, 0, 0]: loss = −ln 0.8580 = 0.1532. Smoothed target [0.9, 0.05, 0.05]: loss = −(0.9·ln 0.8580 + 0.05·ln 0.1161 + 0.05·ln 0.0259) = 0.9(0.1532)+0.05(2.1531)+0.05(3.6533) = 0.1378+0.1077+0.1827 = 0.4282. The smoothed loss is 2.8× larger here, so the gradient step on these logits is larger — but it is pushing toward a different goal. The hard target's minimum is p = [1, 0, 0], which needs logits pushed to infinity; the smoothed target's minimum is p = [0.9, 0.05, 0.05], a finite target (its loss floor is H(t) = 0.9·ln(1/0.9) + 0.1·ln(1/0.05) = 0.0948 + 0.2996 = 0.3944). Train longer under smoothing and the model converges to calibrated-but-not-certain logits rather than racing away; combine it with dropout, which is what suppresses the variance that made the model overconfident in the first place.
The minimum is epoch 7 (1-based) at 0.436 — epochs 8 and 9 both fail to beat it (0.441, 0.439 are higher), and epoch 6's 0.44 is also higher. Patience 3: the window expires after three consecutive non-improving epochs, so training stops at epoch 10 (0.447) and you ship the epoch-7 checkpoint; the run consumed 10 of 40 epochs, saving 30 epochs, 75% of the budget. Patience 2: stop at epoch 9 instead, saving 31 epochs (77.5%) but accepting a stop that lands closer to the minimum — with noisier curves a short patience risks stopping before a later improvement, which is exactly the trade the patience dial sets. Note both answers ship epoch 7, not the last epoch: the final checkpoint had already overfit by 0.011 loss units, and on a real curve that drift is the thing that costs test accuracy.
LayerNorm: mean = (2−1+4+3)/4 = 2; variance = ((0)² + (−3)² + (2)² + (1)²)/4 = (0+9+4+1)/4 = 3.5, σ = 1.8708. x̂ = [(2−2)/1.8708, (−1−2)/1.8708, (4−2)/1.8708, (3−2)/1.8708] = [0.0000, −1.6036, 1.0690, 0.5345]; mean 0.0000, rms 1.0000 (the output is exactly zero-mean unit-variance before γ and β act). RMSNorm: rms = √((4+1+16+9)/4) = √7.5 = 2.7386, output = x/2.7386 = [0.7303, −0.3651, 1.4606, 1.0954]; mean 0.7303, rms 1.0000 (check: (0.7303² + 0.3651² + 1.4606² + 1.0954²)/4 = (0.5333+0.1333+2.1333+1.2000)/4 = 1.0). So RMSNorm keeps the scale normalization — the output has rms exactly 1 whatever the input scale — and drops the centering: a positive input vector stays positive, mean 0.7303 instead of 0. This is the change Zhang & Sennrich measured as ~10% faster with equal accuracy, and why LLaMA and Mistral can drop both the mean subtraction and the β parameter.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 03, Lesson 07) and the Math Foundations Notebook reference build. The eight labs (gap-spectrum explorer with a random-label control, polynomial fit playground with a live validation-versus-λ panel, L1/L2 penalty-contour lab, animated dropout mask with train/eval scaling, three-normalization numeric board, early-stopping patience explorer, augmentation gallery, and the regularization budget board), the three-model gap ledger, the weight-decay ledger with the pure-decay half-life, the L1-versus-L2 one-step race, the inverted-dropout expectation check at p = 0.5/0.2/0.1, the [1, 3, 5, 7] normalization row with the running-mean EMA trace, the synthetic early-stopping checkpoint ledger, the label-smoothing loss floor, and the L1/L2 memory hook are original to this page. Every analytic curve is labelled a simplified teaching model; every number shown is computed live by the labs or verified by hand in the prose.