E[(y − ŷ)²] = bias² + variance + noise splits every wrong prediction into a systematic miss, a jumpy reaction to the data, and randomness no model can touch.
Bias is the systematic miss of the average model, variance is how much the model jumps when the data changes, and noise is the randomness no model can remove. The three always add up, exactly.
E[(y − ŷ)²] = bias² + variance + σ²Both high with a small gap is underfitting: the model is too rigid. Train low and test high with a wide gap is overfitting: the model memorized the noise in its training rows.
gap = test − trainMore data, bagging, regularization and early stopping cut variance. More features, more capacity, and less regularization cut bias. The complexity dial decides which problem you have.
underfit ← complexity → overfitBias is being wrong the same way every time, variance is being wrong differently every time, and noise is being wrong no matter what — so read the train–test gap first, then spend your effort on the term that owns it.
By the end you will be able to decompose an expected error into its three terms, name the culprit from a pair of error numbers, read a learning curve to choose between more data and a better model, explain why averaging and regularization work, and recognize a double-descent curve when the classical U-curve is not the whole story.
A model is trained on one dataset, but the world keeps dealing new ones. Average its squared error over every dataset it could have seen, and the number splits cleanly into bias, variance and noise.
Write the data-generating process as y = f(x) + ε. The true pattern is f(x); ε is the noise in every measurement, with mean zero and variance σ². Your model, ŷ = f̂(x; D), is trained on a particular dataset D — a different sample would have produced a different model. The expectation E[…] below is an average over all of those possible datasets and all of that noise.
The decomposition is E[(y − ŷ)²] = bias² + variance + σ². In plain English: the average squared mistake equals the square of how far the average model sits from the truth (bias), plus how much individual models jump around that average (variance), plus the randomness in the labels that no model can predict (noise). The three terms are squares of errors, so they are never negative, and they add exactly.
| Model at x = 1 (truth f(1) ≈ 1.50) | Predictions on 3 datasets | Average | bias² | variance | noise | total |
|---|---|---|---|---|---|---|
| Rigid (degree 1) | 0.8, 1.0, 1.2 | 1.00 | 0.25 | 0.0267 | 0.25 | 0.5267 |
| Flexible (degree 12) | 0.6, 1.2, 2.7 | 1.50 | 0.00 | 0.7800 | 0.25 | 1.0300 |
Read the two rows carefully. The rigid model is wrong the same way in every world: bias² = 0.25, variance only 0.027, total ≈ 0.53. The flexible model averages out to the truth — bias² = 0.00 — yet its total is almost double, 1.03, because the three predictions disagree wildly (variance 0.78). Fixing bias did not fix error; it moved the error to the other term.
Let f(x) be the truth, ŷ = f̂(x; D) the model trained on dataset D, and f̄(x) = E_D[f̂(x; D)] the average model over all datasets. Two facts make the algebra collapse: the noise ε is independent of the model and has mean 0; and the gap (f̄ − f) does not depend on D.
You train the same model on fifty different resamples and it predicts almost the same wrong value every time. Which term dominates?
The two failure modes look nothing alike once you can resample the data. One model keeps missing the same curve; the other redraws itself every time the sample changes.
High bias means the model is too rigid for the pattern. A straight line fit to a curve is wrong on every dataset in the same direction: its bias² is large, its predictions barely move between resamples, and more data will not help because the limitation is the model’s shape, not the sample size.
High variance means the model is reading the noise. A degree-12 polynomial threads through fourteen noisy points and its shape is different on every resample: each individual fit can look excellent on its training rows while the fan of fits is enormous. Here more data does help, because averaging more noise cancels more of it.
Each click resamples the 14 training points from the same truth and refits. Every thin curve is a different dataset coming from the same process — their spread at the probe is the variance, computed from exactly the curves you see.
Every number above is recomputed from the fan the canvas draws. Push the degree up and watch the strip widen: same truth, same sample size, wildly different predictions — that is variance.
| Symptom | Diagnosis | First thing to try |
|---|---|---|
| High train error, high test error, small gap | Bias — the model is too rigid | More features, more capacity, less regularization |
| Low train error, high test error, wide gap | Variance — the model memorized noise | More data, bagging, regularization, simpler model |
| Low train error, low test error | Good fit | Ship it, then monitor for drift |
| Train error falling, test error rising | Overfitting in progress | Early stopping, more data, stronger regularization |
A model reports 2% error on its training set and 25% on unseen data. What is the most likely diagnosis?
Model complexity is a single knob — polynomial degree, tree depth, layer count. Turn it up and bias falls while variance rises. Turn it down and the exchange reverses.
In the lesson’s experiment the model is a polynomial of degree d, fitted to 30 noisy samples of f(x) = sin(1.5x) + 0.5x. For each degree, many fresh datasets are drawn and the decomposition is computed from the resulting fits. The dial does not reduce total error monotonically: it slides error back and forth between bias² and variance, so the total traces a U. The bottom of the U is the sweet spot; both ends are failures with different names.
Drag the polynomial degree. The fan is 24 models fitted on independent noisy datasets; the bars are computed from those exact fits.
The noise floor is 0.5² = 0.25, printed as the gray segment. No degree can push the total below it — the lever only moves error between the two colored segments.
One run of the experiment — seed 1000, 150 resampled datasets of 30 rows, noise σ = 0.5 on a grid of 41 test points — gives this ledger. Every row, the three components add to the total:
| Degree | bias² | variance | noise σ² | total |
|---|---|---|---|---|
| 1 (too simple) | 0.4350 | 0.0365 | 0.2500 | 0.7215 |
| 5 (sweet spot) | 0.0017 | 0.0559 | 0.2500 | 0.3076 |
| 9 (wiggly) | 0.0035 | 0.3451 | 0.2500 | 0.5986 |
| 12 (wild) | 0.2457 | 19.4848 | 0.2500 | 19.9805 |
Notice that at degree 12 even bias² climbs again: with so much freedom, the average model starts to wobble. High variance eventually contaminates everything.
One train/test pair is a snapshot. Sweep the training-set size and you get a trajectory: whether more data will help, whether the model has run out of capacity, or whether both are fine.
A learning curve holds the model fixed and grows the training set. A validation curve holds the data fixed and grows the model (that was the complexity dial). Run both before deciding how to spend the next week: the first answers “do I need more data?”, the second answers “do I need a different model?”
Read the gap between the curves, then read the slope of the validation curve. A large gap with validation still falling means more data buys accuracy. A small gap with both curves plateaued high means the model itself is the ceiling — more rows change nothing.
Pick a model and slide the training-set size. Train error is measured on the rows the model saw; validation error on a fresh set it did not. The shaded band is the gap — each curve averages 40 random splits.
A validation set is noisy too, so the curves bottom out near the noise floor (0.5² = 0.25 here), not at zero. The shape above the floor is the part the model controls. Degree 9 validation error at n = 12 is in the hundreds — small data plus high capacity is the worst corner of the whole lesson.
| Scenario | Train error | Validation error | Gap | What to do |
|---|---|---|---|---|
| High bias | High and flat | High and flat | Small | More data will not help — change the model |
| High variance | Low | High, still falling | Large | More data will help — or regularize now |
| Good fit | Moderate | Moderate, flat | Small | Ship it |
| Variance, improving | Low | Falling with more data | Shrinking | Keep collecting data |
The lab’s experiment (40 repeated 60/40 splits, noise σ = 0.5, errors in MSE against noisy labels) produces three unmistakable shapes at n = 12 rows:
Both error numbers bottom out near the noise floor σ² = 0.25 when the model is good, because the labels themselves carry that much randomness. A learning curve that flattens just above 0.25 is the best news this diagnostic gives.
Small print worth knowing: a single split gives a noisy curve, which is why the lab averages over 40 repeated splits. The crossing points and the exact floor move run to run; the shape does not.
Both curves have plateaued with a small gap, but both errors are high. What is the right next move?
If each model catches a different piece of the noise, averaging their predictions cancels a lot of it. This one idea — bagging — powers random forests, deep ensembles and half of applied ML.
Bagging (bootstrap aggregating) trains many copies of a high-variance model on different resamples of the data and averages their predictions. Each copy still overfits its own sample, but the independent parts of their mistakes cancel. The result is a smoother prediction with much lower variance and roughly the same bias.
The ideal math is simple: averaging N independent predictions of variance σ² gives variance σ²/N. The catch is the word independent. Every model saw data drawn from the same world — often the same 200 rows — so their errors move together. With an average pairwise correlation ρ, the variance of the average is σ²·(1 + (N−1)ρ)/N. When ρ is close to 1, averaging more models changes almost nothing.
The pool is 48 high-variance degree-12 fits. Average the first N and the result calms down — but not as fast as the ideal 1/N, because every fit learned from the same noisy world and they move together. ρ is their measured pairwise correlation.
Doubling N from 8 to 16 barely changes the average because ρ ≈ 0.5: with shared data, models are not independent witnesses. Bagging still helps — it just cannot beat (1 + (N−1)ρ) / N.
Let each model’s prediction be a random variable with variance σ², and let every pair share correlation ρ. The average is (1/N)·Σ f̂ᵢ. The variance of a sum has one term per pair, so:
| Method | What it does | Effect |
|---|---|---|
| Bagging | Averages many high-variance models | Variance falls, bias unchanged |
| Boosting | Builds models sequentially on the last errors | Bias falls, variance can rise |
| Stacking | Learns how to combine different models | Depends on the base models |
| Dropout | Trains with random units off (implicit bagging) | Variance falls, bias rises slightly |
Regularization deliberately makes the model less free. The penalty forbids the wild fits that chase noise, and the same λ dial slides you along the bias-variance trade-off.
Ridge regression minimizes MSE + λ·Σwᵢ². The first term wants to fit the training data; the second charges rent for every unit of weight. Big λ means a flatter, calmer model — higher bias, lower variance. The intercept is usually left unpenalized, because shrinking it would just drag every prediction toward zero regardless of the data.
The same trade appears under other names: dropout randomly removes units so no single one can memorize a sample; early stopping halts training while the model still generalizes; L1 goes further than L2 and drives useless weights to exactly zero. Different mechanisms, one ledger entry: more constraint, more bias, less variance.
The model is fixed at degree 12 — only λ changes. The penalty shrinks coefficients toward zero, trading a little bias for a lot less variance. Watch the fan collapse as λ grows.
The optimal λ is the bottom of the dashed total curve, and you find it with cross-validation — never by reading the test set. The gray noise row is untouchable at λ = 100 just as it was at λ = 0.
In the lab, degree 12 with 24 resampled datasets and σ = 0.5 gives this path. Each row’s components still sum to its total:
The optimal λ depends on the data, the noise level and the model size, so there is no universal default. The lab’s plateau is flat between roughly 0.0003 and 0.1, which is typical: near the bottom, small changes in λ barely move the total.
| Technique | Mechanism | Bias–variance effect |
|---|---|---|
| L2 / Ridge | Adds λ·Σwᵢ² to the loss | Shrinks every weight toward zero; keeps all features |
| L1 / Lasso | Adds λ·Σ|wᵢ| to the loss | Drives some weights exactly to zero — feature selection |
| Dropout | Randomly disables units during training | Forces redundancy; variance falls, bias rises a little |
| Early stopping | Stops before training error bottoms out | Freezes the model before it chases noise |
Classical theory says more capacity always hurts after the sweet spot. Modern models live far to the right of the U — and the curve turns back down, because they are not at the worst place anymore.
Push capacity far past the number of training examples and the classical U-curve breaks. The worst point is the interpolation threshold, where the model has just enough parameters to fit every training point. There, it is forced through every noisy label and its variance peaks. Add more parameters and the model can fit the data many different ways; the learning algorithm, left to itself, prefers the solution with the smallest weights — the smoothest way through the points. That implicit regularization is why hugely overparameterized networks generalize at all.
Practical rule: stay comfortably below the threshold with explicit regularization, or go well past it. The one place not to linger is p ≈ n, parameters roughly equal to rows. The same shape appears when you sweep training epochs (epoch-wise double descent) or even dataset size (sample-wise), so “more data can hurt” is not as absurd as it sounds in the narrow threshold band.
A simplified linear experiment: 15 noisy examples, each with p measurements; the label depends on the first three, the rest are pure noise. The learner keeps the smallest-weight fit. Sweep p past the number of examples and watch error dive past the peak.
The peak sits at p = n because that is where the model can just interpolate the training rows and is forced through every noisy point. With σ = 0 the bump disappears — variance comes from fitting noise. Labeled a simplified experiment: real networks differ, but the shape is the lesson.
Two numbers, five verdicts. Come back to this after every training run.
| Symptom | Diagnosis | What to do |
|---|---|---|
| High train error, high test error | Bias (underfitting) | More features, more capacity, less regularization |
| Low train error, high test error | Variance (overfitting) | More data, bagging, regularization, early stopping |
| Low train error, low test error | Good fit | Deploy and monitor |
| Train error falling, test error rising | Overfitting in progress | Early stopping now |
| Both errors stuck at the noise floor | At the limit of this data | Better features, not more rows |
In practice the diagnostics run in one sequence. This loop takes minutes on tabular data and saves days of guessing:
Where does a model's variance peak, and what should you do about it?
Answer before you look. The diagnosis questions are the ones that come back every week of a working ML career.
0 / 6 answered · 0 correct
01A straight line is fitted to a clearly curved (quadratic) relationship. Which error component dominates?
02The bias-variance decomposition has three terms. Which one cannot be reduced by any model?
03Adding L2 regularization increases bias and decreases variance. Why is that useful?
04A model reports 2% training error and 25% test error. What is the most likely diagnosis?
05You train the same architecture on 50 random training subsets and the predictions vary wildly between them. What does this indicate?
06In the double-descent picture, what happens just past the interpolation threshold (parameters ≈ rows)?
Click a card to swap the lazy description for what it actually means.
Four problems with numbers — including a bagging experiment. Try first; a worked answer is one click away.
The noise term becomes exactly 0.00, and variance collapses too: with no noise to chase, every resample is fitted at the same true curve. In the lesson's setup at n = 30, degree 1 still carries bias² ≈ 0.43, but from degree 5 upward the total is ≈ 0.00 (degree 5 total 0.0013; degrees 9–15 even smaller). So the optimal complexity moves right: any flexible-enough model is fine, and extra capacity is no longer punished. The catch is that σ = 0 never holds in practice — the hardware of the experiment changed, not the principle. Zero noise removes the variance term, so the U-curve flattens into an L.
For a fixed degree-7 model the variance falls from 0.1145 at n = 30 to 0.0166 at n = 100 and 0.0053 at n = 300 — roughly a factor of 20 overall — while bias² goes from 0.0015 to 0.0000. In this experiment the optimal degree stays at 5, but the penalty for going higher all but disappears: degree 15's total error drops from 201.8 at n = 30 (variance dominates catastrophically) to 0.26 at n = 300. More data does not make the model smarter; it makes extra capacity safe by shrinking the variance term.
The path is monotone in each component and U-shaped in the total. The lab runs the sweep at degree 12 for speed; from it, λ = 0 gives bias² 0.17 / variance 2.78 / total 3.20; λ = 0.01 gives bias² 0.016 / variance 0.050 / total 0.32; λ = 1 gives bias² 0.30 / variance 0.049 / total 0.60; λ = 100 gives bias² 1.24 / variance 0.062 / total 1.55. Degree 15 is the same curve shifted: the unregularized variance is even larger, so the optimal λ is larger too. Variance falls by orders of magnitude as λ grows and then stops changing, because the model has already been squeezed toward a nearly flat fit. Bias² first falls too (the unregularized average is itself unstable) and then climbs steeply. The total bottoms out near λ = 0.01 — pick λ by cross-validation, not by the test set, because the exact minimum depends on the data and noise.
The lab measures the pairwise correlation of 48 degree-12 fits at ρ ≈ 0.51. Averaging N = 10 correlated models gives variance σ²(1 + 9·0.51)/10 = 0.56·σ² — a 44% cut, where the independent ideal 1/10 = 0.10 would promise a 90% cut. Doubling to N = 20 only reaches (1 + 19·0.51)/20 = 0.53·σ². The ensemble still helps — the prediction is visibly calmer than any single fit — but shared training data sets a floor: with perfect correlation ρ = 1 the average is just one model, and no number of copies moves the variance. Practical reading: bagging is worth it; expecting 1/N is not.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 02, Lesson 10) and the Math Foundations Notebook reference build. Interactive figures, the six labs, the hand-worked numeric examples and worked exercise answers are original to this page. Every decomposition number a lab prints is computed live from the fits it displays.