0.87 versus 0.89 on 500 test examples is about one standard error — the kind of gap pure luck produces a third of the time. This lesson is the toolkit that tells the difference: standard error, confidence intervals, p-values, and effect size.
The mean is the balance point, the median the halfway mark when sorted, the mode the most common value. Variance is the average squared distance from the mean; its square root, the standard deviation, is back in the data's units. When mean and median diverge, the data is skewed.
mean 28.6 · median 23 · s ≈ 24.2Every test-set number is one sample from a noisy process. Its standard error is σ/√n: quadruple the data and the error halves. The central limit theorem says sample means become bell-shaped as n grows, which is why we can put an error bar on almost anything.
SE = σ / √n · 4× data → ½ errorThe p-value is P(data this extreme | the null is true) — it is not the probability the null is true. With enough data, a 0.03% gain becomes significant. Cohen's d measures the size of the difference independently of n, and Bonferroni keeps multiple tests honest.
p < 0.05 but d = 0.015 → ship nothingEvery number you measure on a test set is a sample from a noisy process, so statistics asks one question over and over: could this difference have happened by chance? Standard errors, confidence intervals, p-values and the bootstrap are all different ways of measuring how big “chance” is.
By the end you will be able to describe a dataset with the right summary, read a p-value and a confidence interval without misquoting them, correct for multiple comparisons, and decide whether a model difference is real, large, or neither.
Before you model anything, compress the data into a few numbers that capture its shape. The two questions are always the same: where is the centre, and how far does the data wander from it?
The mean adds every value and divides by the count; it is the point where the data balances. The median sorts the data and takes the middle value. The mode is the most frequent value, which is useful for categories and rarely informative for continuous measurements. Watch them disagree on [1, 2, 3, 4, 1000]: the mean is (1+2+3+4+1000)/5 = 202, but the median is 3. The one rich value dragged the mean; the median did not move.
Spread answers the second question. Variance is the average squared distance from the mean, s² = Σ(xᵢ − x̄)² / (n − 1); the standard deviation is its square root, which returns the number to the units of the data. Percentiles divide sorted data into 100 parts: P50 is the median, P95 and P99 are the tail. For a training loss, the mean is dragged around by a few hard examples, while percentile monitoring shows whether the tail is growing.
| Statistic | Formula | Plain English | Robust to outliers? |
|---|---|---|---|
| Mean | x̄ = (1/n) · Σ xᵢ | The balance point of the data. | No — one huge value drags it |
| Median | middle value when sorted | The 50th percentile: half the data on each side. | Yes — it only counts positions |
| Variance | s² = (1/(n−1)) · Σ (xᵢ − x̄)² | The average squared distance from the mean. | No — the square magnifies extremes |
| Standard deviation | s = √(s²) | The typical distance from the mean, in the data's units. | No — same reason as variance |
| Quartiles | Q1, Q3 | The 25th and 75th percentiles: the middle half lives between them. | Mostly — quartiles move slowly |
| IQR | Q3 − Q1 | The width of the middle 50%, used by box plots and outlier rules. | Yes — built for it |
Second worked example, a left-skewed loss. Five training examples lose [0.2, 3.0, 3.0, 3.2, 3.4]. The mean is 12.8 / 5 = 2.56, below the median 3.0, because one unusually easy example pulls the mean down. Incomes do the opposite: a few billionaires push the mean above the median. The direction of the gap tells you which way the data is skewed.
Drag a dot to the right and watch the mean chase it while the median barely moves. The box is the middle 50% (Q1 to Q3), and the 1.5 × IQR rule marks outliers.
The mean is the balance point, so one far-away weight tips it. The median only counts positions, so it stays where the middle is. That is why latency dashboards quote P50 and P99, not the mean.
Population variance uses the true mean μ. You only have the sample mean x̄, which sits closer to your particular samples than μ does, so the deviations from x̄ come out systematically a little too small. How much too small?
For [1, 2, 3, 4, 1000], which pair of statistics best tells you what a typical value looks like?
Covariance says two variables rise and fall together; correlation rescales it so that every pair is comparable on a −1 to +1 ruler. Both are about linear co-movement — and neither one is causation.
The covariance between X and Y is Cov(X,Y) = (1/(n−1)) Σ (xᵢ − x̄)(yᵢ − ȳ): multiply how far each point sits from its own mean, and average. Positive means the variables rise together, negative means one rises as the other falls, zero means no linear co-movement. For features x = [1, 2, 3] and y = [2, 4, 6], the centred products are 2, 0, 2, so the sample covariance is 4/2 = 2. The covariance matrix of the two features is [[1, 2], [2, 4]] — variances on the diagonal, covariance off it. Its determinant is 4 − 4 = 0, exactly because the two features are the same information rescaled.
Covariance has units, so it cannot tell you whether 2 is strong or weak. Dividing by both standard deviations gives Pearson’s r = Cov(X,Y) / (sₓ·s_y), which lives in [−1, 1]: +1 is a perfect increasing line, −1 a perfect decreasing line, 0 no line. Spearman’s ρ is the same computation after replacing every value with its rank, so it catches any monotonic relationship and shrugs off outliers. Use Pearson for continuous, roughly normal data with no extreme points; use Spearman for rankings, skew, or outliers.
Centre both variables: uᵢ = xᵢ − x̄ and vᵢ = yᵢ − ȳ. Then the correlation is a dot product divided by two lengths:
One extreme point can swing the direction of u or v and therefore r, which is exactly why Pearson is fragile and Spearman replaces values with ranks first.
Model A scores 0.87 and Model B scores 0.89 — but on a different test set the order could flip. The sampling distribution tells you how much a statistic would move if you ran the whole experiment again.
Your test set is one sample from a noisy process. If you had drawn a different 500 examples, you would have measured a different accuracy. The sampling distribution of a statistic is the distribution of the values it takes across all those hypothetical repetitions. Its standard deviation has a special name: the standard error. For a sample mean, SE = σ/√n — the population spread divided by the square root of the sample size.
The square root is the part everyone underestimates. If σ = 15 and you test 100 examples, SE = 1.5. Test 400 examples and SE = 0.75. To get SE down to 0.3 you need n = (15/0.3)² = 2,500 examples. Ten times the precision costs a hundred times the data.
The central limit theorem (CLT) is why this works even when the data is not bell-shaped: as n grows, the distribution of the sample mean approaches a normal distribution centered on the true mean, whatever shape the population had. A rule of thumb is n ≥ 30 for mildly skewed data, n ≥ 100 for heavily skewed data. What the CLT does not promise: it makes the mean normal, not the data; it fails for infinite-variance tails (Cauchy) and for dependent observations such as sentences from one document or images from one patient.
Each of the 500 experiments draws n values from the same skewed population and records its mean. The histogram is the sampling distribution; the orange curve is the normal prediction centered on the true mean with width σ/√n.
The individual values are skewed, but the means are not — that is the central limit theorem. Watch the blue histogram: as n grows it becomes a narrower, taller bell around the true mean.
“Accuracy 0.88” hides how much data it came from. A confidence interval turns the guess into a range, and the width of that range is the honesty — or the lack of it — in the row.
For a mean, the 95% confidence interval is x̄ ± 1.96 · s/√n. The piece s/√n is the standard error; the 1.96 is how many standard errors you must step out to cover the middle 95% of a normal distribution. For the opening story — accuracy 0.88 on 500 examples — SE = √(0.88 × 0.12 / 500) ≈ 0.0145, so the interval is about 0.88 ± 0.028, or [0.852, 0.908].
Compare two models on the same 500 examples. Each estimate has SE ≈ 0.0145, and the difference has SE_diff = √(0.0145² + 0.0145²) ≈ 0.021. A 0.02 gap is about one standard error, so the 95% interval for the difference runs from roughly 0.02 − 0.04 = −0.02 to 0.02 + 0.04 = +0.06 — it contains zero. “One standard error” happens by luck about a third of the time, which is exactly why the less accurate model can win a small test set.
What the 95% means. If you repeated the experiment forever, 95% of the intervals you built this way would contain the true value. It is a property of the recipe, not of any single interval. The width measures precision; it says nothing about accuracy. A biased sample gives a tight interval around the wrong number — precise and wrong.
Each row is one whole experiment: draw n samples, compute the confidence interval. Green bars contain the true mean, red bars miss. The guarantee is about the procedure — not about any single bar.
A common misreading: “this specific interval has a 95% chance of containing μ”. Wrong — this interval either does or does not. The 95% describes the recipe: repeat it forever, and 95% of the intervals come out right.
The t-test needs a formula for the standard error and a normality assumption. The bootstrap needs neither. Your sample is your best estimate of the population, so resampling from it with replacement imitates drawing fresh data from the world.
The price: the bootstrap trusts your sample. If the sample is biased, every resample inherits the bias — a narrow interval around the wrong answer, with no warning.
A model’s 95% confidence interval for accuracy is [0.83, 0.89]. Which statement is correct?
A hypothesis test is a what-if machine: assume there is no effect, then ask how weird your measurement would look. The p-value is the answer to exactly that question — and to no other.
Start with the null hypothesis H₀ — the boring default, usually “no difference”. For two models that is “A and B have the same accuracy”. The alternative H₁ is what you are trying to find evidence for, such as “B is better”. Then measure how far the data landed from the null world.
If p < α (usually 0.05), the data is unusual enough under H₀ that we reject it and call the result statistically significant. Otherwise we fail to reject H₀ — which is not the same as proving it; it often means the experiment was too small. The test statistic turns the comparison into a ruler: t = (x̄ − μ₀)/(s/√n) asks “how many standard errors away from the null is this?”, and Welch’s two-sample version t = (x̄₁ − x̄₂)/√(s₁²/n₁ + s₂²/n₂) does the same for two groups without assuming equal variances. When both models are evaluated on the same folds, subtract fold by fold and run a one-sample test on the differences — pairing removes the fold-to-fold difficulty and buys a lot of power. With only 5–10 folds the normality assumption is shaky, so the rank-based Wilcoxon signed-rank test is often the safer paired choice (and the bootstrap works there too).
Group B is shifted by the effect you choose. The null distribution is built by shuffling the 16 labels 2,000 times — that is what the world looks like when H₀ is true and there is no difference. The p-value is the share of shuffles at least as extreme as what we observed.
Slide the effect to 0: both groups come from the same world, and the observed difference lands in the middle of the null distribution, so p is large. That is the whole logic of a significance test.
Check 1 — not significant. A sensor claims mean 100. You measure x̄ = 103 on n = 16 readings with s = 8:
Check 2 — same gap, different noise. Two models on 100 examples each, both with s = 0.05, means 0.87 and 0.89:
Counts, for completeness. A model produces 120 positive and 80 negative outputs where 100/100 was expected: χ² = 20²/100 + 20²/100 = 8 with 1 degree of freedom, so p < 0.005 — the output distribution is not what was expected.
The permutation flavour (what the lab runs) skips the formula: pool the two groups, shuffle the labels 2,000 times, and count how often the difference is at least as extreme.
A model comparison reports p = 0.2. What does that mean?
A 5% false-positive rate sounds safe until you run twenty experiments. This is the trap behind hyperparameter sweeps, multi-metric evaluations and leaderboard wins.
Under H₀, each test falsely rejects with probability α = 0.05. Run m = 20 independent tests and the chance that at least one of them fires is 1 − (1 − 0.05)²⁰ = 1 − 0.358 ≈ 0.64 — a 64% chance of a bogus “significant” result, and an expected one false positive per batch even though nothing is real. The more configurations you sweep, the more of your “wins” are arithmetic rather than signal.
The Bonferroni correction is the blunt fix: test each hypothesis at α/m instead of α. For 20 tests that is 0.05/20 = 0.0025, and the family-wise false-positive rate returns to 1 − (1 − 0.0025)²⁰ ≈ 0.049. It is conservative — real effects also have to clear the higher bar — but it is simple and safe when tests are independent. With hundreds of tests, the Benjamini–Hochberg false-discovery-rate procedure is a less punishing alternative. What no correction can fix: leakage, a biased split, or an experiment designed to find something.
Each card is one independent A/B test of a change that does not work in the real world; its p-value is uniform noise. At α = 0.05 you expect one “winner” per batch. Bonferroni demands p < 0.05/20 = 0.0025, and the winners almost vanish.
orange = p < 0.05 · green = p < 0.0025 (survives Bonferroni) · ★ = the one test with a real effect (when enabled)
Turn on the real effect: it survives Bonferroni because its p-value is genuinely tiny. Bonferroni does not kill real effects — it kills small ones that cannot clear the higher bar. Run batches until a false positive shows up under the correction, and notice how rare that is.
The p-value answers “is it real?”. The effect size answers “is it big enough to matter?”. A model comparison needs both answers, plus a test set neither model has ever seen.
With a million test samples, a 0.03% accuracy gap gets p = 0.001: statistically significant and rarely worth the engineering cost of a deployment. Effect size measures the difference in units of spread, independent of sample size. Cohen’s d = (mean₁ − mean₂)/pooled standard deviation, with familiar landmarks: d = 0.2 small, 0.5 medium, 0.8 large. Two worked examples show why the units matter:
Model A/B testing is not web A/B testing. Both models must be scored on the same test set; accuracy alone is never enough (add precision, recall, F1, latency, fairness); variance comes from cross-validation or bootstrap rather than a single number; and any test set used during model selection is contaminated, so keep a final holdout. The procedure: pick the metric and α; run both models on the same k folds; collect paired differences; run a paired t-test (or bootstrap the difference); report the confidence interval and the effect size; then decide against a minimum useful gap that you wrote down beforehand.
Underpowered experiments are their own trap. To detect a gap δ with 80% power at α = 0.05 you need roughly n ≈ 15.68 · σ²/δ² examples per group. With σ = 0.05 and δ = 0.02 that is 15.68 × 0.0025/0.0004 ≈ 98 per group — unpowered comparisons produce “no significant difference” that only means “not enough data”.
Two models, one shared test set. The p-value asks whether the gap is real; Cohen’s d and the interval ask whether it is big enough to care about. Slide the test set down and watch a “winner” evaporate.
Statistical significance is about whether the interval crosses zero. Practical significance is about whether the whole interval clears the minimum useful gap. With 1,000,000 samples a 0.03% gain passes the first test easily — and fails the second. This is a design calculator, so it treats the true gap as the observed gap: the interval shows the luck of sampling around a known effect.
| Mistake | Why it bites |
|---|---|
| Testing on the training set | Guarantees an optimistic number. |
| No confidence intervals | A single accuracy is unverifiable. |
| Ignoring multiple comparisons | The best of 50 configs is noise-inflated. |
| Accuracy on imbalanced data | 99% negatives means a do-nothing model scores 99%. |
| Leakage across splits | Normalizing before splitting, or using the future to predict the past. |
| Assuming independence | Correlated observations shrink the effective n. |
| p-hacking | Trying subsets and tests until p < 0.05. |
Model B beats Model A by 0.01% accuracy with p = 0.001 on five million examples. What is the right conclusion?
Answer before you look. The p-value and multiple-comparison questions are exactly the ones that decide real model comparisons.
0 / 5 answered · 0 correct
01What does a p-value of 0.03 mean in a hypothesis test?
02Why do you divide by (n − 1) instead of n when computing sample variance?
03You test 20 different model configurations at alpha = 0.05. What is the approximate probability of at least one false positive?
04Model A scores 0.9234 and Model B scores 0.9237 on 1 million test samples with p-value = 0.001. What should you conclude?
05What advantage does bootstrap have over the paired t-test for comparing two ML models?
Click a card to swap the lazy description for what it actually means.
Four short problems with fully worked answers. Try first; the solution is one click away.
Sum = 286, so mean = 286/10 = 28.6. Sorted, the middle two are 22 and 24, so median = 23. Squared deviations sum to 5260.4; sample variance = 5260.4/9 ≈ 584.5, so s ≈ 24.2. Q1 = 15 (middle of the lower half), Q3 = 30 (middle of the upper half), IQR = 15. The fences are 15 − 22.5 = −7.5 and 30 + 22.5 = 52.5, so 95 is an outlier. Remove it: the mean becomes 191/9 ≈ 21.2, a drop of 7.4, while the median becomes 22, a move of just 1.
Means: x̄ = 2.5, ȳ = 25. Centred vectors u = [−1.5, −0.5, 0.5, 1.5] and v = [−24, −17, 2, 39]. Σuv = 36 + 8.5 + 1 + 58.5 = 104, Σu² = 5, Σv² = 2390, so r = 104 / √(5 × 2390) = 104 / 109.3 ≈ 0.951. The ranks of x are 1, 2, 3, 4 and the ranks of y are also 1, 2, 3, 4, so Spearman is Pearson on identical rank lists: rho = 1 exactly. The curve is perfectly monotonic; Pearson only credits the linear part, which is why it reads 0.95 instead of 1.
P(at least one false positive) = 1 − (1 − 0.05)^20 = 1 − 0.95^20 ≈ 0.64, so about a 64% chance — and an expected one false positive per batch of 20. Bonferroni tests each hypothesis at 0.05/20 = 0.0025, which brings the family-wise rate back to 1 − 0.9975^20 ≈ 0.049 ≈ 5%. The cost: a real effect needs p < 0.0025 to survive, so small true effects are missed. A p-value of 0.03 means that if H₀ were true, data at least this extreme would appear 3% of the time. It is NOT the probability that H₀ is true (3%), NOT the probability the result is a fluke, and NOT evidence that the effect is large or worth deploying.
There are 3³ = 27 ordered resamples, which collapse into 10 multisets. Counting weights: median 2 in 1 + 3 + 3 = 7 samples (25.9%), median 4 in 1 + 3 + 3 + 6 = 13 samples (48.1%), median 9 in 1 + 3 + 3 = 7 samples (25.9%). The 2.5th and 97.5th percentiles both fall inside the extreme categories, so the 95% interval is [2, 9] — nearly the whole range of the data. The bootstrap is not broken; with n = 3 the sample genuinely carries almost no information about the median, and the interval says so honestly instead of pretending otherwise.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 01, Lesson 15) and the Math Foundations Notebook reference build. Interactive figures, the second worked examples, the correlation gallery, the coverage and permutation labs, and the multiple-comparison and A/B calculators are original to this page. Every lab runs in your browser.