precision = TP / (TP + FP) is one counting rule among many. Watch a threshold sweep rewrite the confusion matrix — then trace the ROC curve those counts draw.
Precision punishes false alarms; recall punishes misses; specificity clears the negatives; F1 balances precision and recall and refuses to be fooled by the strong one. Print all of them, plus the majority-class baseline, before believing any single score.
precision = TP / (TP + FP)A classifier outputs a score; the threshold turns it into a decision. Slide left to catch more positives and collect false alarms; slide right to be certain of every flag and miss more. ROC shows the whole sweep, PR focuses on the class that matters.
P ↑ ⇒ R ↓ along one sweepCross-validation gives every row a turn in validation, stratified folds keep the class ratio honest, and the fold scores become a mean ± spread. Finish with a bootstrap interval so a 0.002 difference cannot masquerade as a result.
accuracy 0.750 ± 0.186A model is a claim; evaluation is the evidence: count the four cells, pick the metric that prices your two mistakes, cross-validate the choice, and put an interval on the number before anyone ships it.
By the end you will be able to compute precision, recall, specificity and F1 from any confusion matrix, read ROC and PR curves at the operating threshold, explain what AUC does and does not promise, check calibration with a reliability diagram and Brier score, score regressions with MSE, RMSE, MAE, R² and MAPE, run K-fold and stratified cross-validation, and report a metric with a bootstrap confidence interval.
You trained a model and it scores 95%. Whether that is excellent or worthless depends on what a do-nothing model would have scored.
Suppose 95% of your rows are negative. A model that answers “negative” forever is right 950 times out of 1,000 and scores 95% accuracy while catching nothing. Accuracy answers “how often was the model right?” — but when one class dominates, being right is easy and being useful is not the same thing.
The first number to compute, always, is the majority-class baseline: the score of predicting the most common label everywhere. A model is only interesting if it beats that. The second habit is to split the data before touching it. Training rows fit the weights, validation rows choose hyperparameters and compare models, and the test set is opened exactly once at the end to report. Evaluate on rows the model trained on and it will happily recite the answers — memorization is not learning.
Concretely: a screening model tests 200 patients, 20 of whom have the disease. It flags 40 people, catching 15 of the 20; 25 healthy patients get a false alarm. Its accuracy is (15 + 155) / 200 = 0.85. The do-nothing baseline — calling everyone healthy — scores 180 / 200 = 0.90. The trained model is worse than doing nothing, and its headline 85% would never have told you.
Same story in both: a high accuracy, a higher baseline, and two class metrics that expose the failure immediately. This is why the next chapter stops looking for one number and starts counting correctly.
A dataset is 95% negative and 5% positive. A model predicts “negative” for every sample. What is its accuracy?
Split the predictions into the four ways a binary decision can go, and precision, recall, specificity and F1 stop being abbreviations to memorize — they become arithmetic you can see.
A binary classifier makes one of four mistakes or successes. True positive (TP): actually positive, predicted positive. True negative (TN): actually negative, predicted negative. False positive (FP): actually negative but flagged — a false alarm. False negative (FN): actually positive but missed. Every metric in this lesson is a ratio of these four counts.
Precision is about the predictions: of everything we flagged, what fraction was truly positive? Recall (sensitivity) is about the reality: of everything truly positive, what fraction did we flag? Specificity is recall for the other class: of the truly negative, how many did we correctly clear? A test that says “positive” to everyone has perfect recall, zero specificity, and terrible precision.
| Metric | Formula | Plain English | 200 patients |
|---|---|---|---|
| Accuracy | (TP + TN) / n | How often is the model right overall? | 0.850 |
| Precision | TP / (TP + FP) | Of the positives we flagged, how many were right? | 0.375 |
| Recall | TP / (TP + FN) | Of the real positives, how many did we catch? | 0.750 |
| Specificity | TN / (TN + FP) | Of the real negatives, how many did we clear? | 0.861 |
| F1 | 2PR / (P + R) | One number balancing precision and recall. | 0.500 |
The 200-patient matrix: TP = 15, FN = 5, FP = 25, TN = 155. Accuracy is (15 + 155) / 200 = 0.850; the always-negative baseline is 0.900. Read the F1 row backward and you see why it is popular: P = 0.375 and R = 0.750, but F1 = 0.500 — the harmonic mean refuses to reward the strong one.
Slide the cutoff over the same 20 test scores. Moving it left trades false alarms for misses; moving it right does the opposite. The counts and every metric below are computed from what you see.
Raise τ to 0.72 and precision climbs to 0.800 while recall falls to 0.500. Lower it to 0.31 and recall reaches 0.875 while precision drops to 0.500. F1 peaks in between, near 0.667 — that is the harmonic mean doing its job.
Type the four counts by hand — or load one of the presets — and every metric is computed on the spot. This is the whole of classification evaluation: four numbers, a handful of ratios.
Watch precision and recall move in opposite directions as you move a single count. That tug-of-war is the whole reason one number can never tell the story.
Precision and recall pull in opposite directions along the same threshold. F1 peaks in the middle, but the right operating point is decided by costs, not by F1: screening buys recall, spam filtering buys precision.
A model has TP = 8, FP = 2, FN = 4. What is its precision?
A classifier usually outputs a score, and you choose the cutoff. The ROC curve shows every cutoff at once; the precision–recall curve shows the same sweep through the lens of the class that matters.
Sweep the threshold from strict (almost nothing is positive) to loose (almost everything is). At each cutoff, two rates move: the true positive rate, TPR = TP / (TP + FN) — that is recall — and the false positive rate, FPR = FP / (FP + TN) = 1 − specificity. Plot TPR against FPR and the path is the ROC curve. Its area, AUC, is the probability that a randomly chosen positive gets a higher score than a randomly chosen negative: 0.5 is a coin flip, 1.0 is perfect ordering.
AUC’s selling point is that it is threshold-free, but that is also its blind spot. ROC normalizes each class by its own size, so it answers “how well are the classes separated?” and not “what happens when positives are rare?” The precision–recall (PR) curve plots precision against recall at every cutoff. Precision’s denominator is the number of positive predictions, which grows with prevalence, so PR notices a flood of negatives while ROC sails through unchanged.
One score list, two stories. The threshold slider moves the operating point on both curves. The imbalance control repeats every negative — the ranking never changes, so watch which summary notices.
ROC divides false alarms by all negatives, so duplicating negatives scales both axes and the curve stands still. Precision divides by predictions, so the extra negatives land in the denominator and AP falls: 0.743 (×1) → 0.624 (×2) → 0.506 (×4) → 0.408 (×8). When the positives are rare, PR is the honest curve.
The ranking never changed; only the number of negatives did. ROC rescales both axes by the class sizes, so duplicating negatives leaves the curve exactly where it was. Precision sees the new negatives as potential false alarms, so it drops. When the positive class is rare, AP is the number that tracks what production will feel.
You keep every positive and every score, but repeat each negative 8 times. What happens?
A model that ranks perfectly can still lie about its probabilities. Calibration measures whether the number that comes out of the model is the frequency you should expect — and the Brier score prices the lie.
AUC only cares about the order of scores. But in production the number is often what gets used: 0.9 drives one decision, 0.3 another. A model is calibrated when its predicted probabilities match reality — among all cases it scored 0.7, about 70% should be positive. The reliability diagram plots predicted probability on the x-axis against observed frequency on the y-axis; perfect calibration is the diagonal.
The single-number summary is the Brier score: the mean squared error on the probabilities themselves, mean((p − y)²), where y is 1 or 0. It is lower when better and is minimized by telling the truth: predicting the observed rate for every case. A useful comparison is the base-rate Brier, the score of always predicting the overall positive rate — for balanced data that is 0.5·0.5 = 0.25.
30 cases in three risk bands with observed positive rates 0.30, 0.50 and 0.70. The sharpen control exaggerates the same ranking toward 0 / 1. Perfect calibration sits on the dashed diagonal.
At k = 4 the extremes say 0.967 / 0.033 when the truth is 0.70 / 0.30: Brier climbs to 0.271, worse than the 0.250 of punting with the base rate. Ranking and calibration are two different questions, and a model needs both answered.
This is the trap the number exposes: confidence is not accuracy. At k = 4 the extremes claim 0.967 and 0.033, yet the truth in those bands is 0.70 and 0.30. Brier rises above even the do-nothing 0.25.
Model A and Model B have identical rankings (AUC 0.90), but A's Brier is 0.30 and B's is 0.12. Which is more useful as a risk score?
Accuracy has no meaning for a predicted price. Regression gets its own family of scores, and each one answers a different “how wrong?” question.
Start with the residual y − ŷ: the gap between the truth and the prediction. MSE squares every residual and averages; squaring makes big misses dominate and changes the units to dollars-squared, which nobody can feel. RMSE takes the square root, bringing the typical miss back into dollars. MAE never squares: it averages the absolute residuals, so a single catastrophic miss counts in proportion to its size, not its square.
R² compares the model with the laziest possible baseline, predicting the mean of y. If the model removes all of that baseline’s error, R² = 1. If it does exactly as well as the mean, R² = 0. If it does worse, R² goes negative — which is possible and worth checking, because “my model beats the mean” is the minimum bar in regression. MAPE divides each miss by the true value, so it reads as a percentage and is comparable across short and long predictions — with one sharp edge discussed below.
| Metric | Formula | What it answers | Units |
|---|---|---|---|
| MSE | mean((y − ŷ)²) | Average squared miss. Every error is amplified before it is averaged. | units² |
| RMSE | √MSE | Typical miss size in the target's own units. Outlier-sensitive. | same as y |
| MAE | mean(|y − ŷ|) | Average absolute miss. Every error counts the same. | same as y |
| R² | 1 − SS_res / SS_tot | Fraction of the target's variance the model explains. | none |
| MAPE | mean(|y − ŷ| / |y|) | Average miss as a percentage of the true value. | % |
That gap is the whole personality difference: RMSE is a big-miss alarm, MAE is the everyday average. Choose RMSE when large errors are disproportionately painful; choose MAE when every miss costs roughly the same.
A single train/validation split is a coin flip on which rows landed where. K-fold cross-validation gives every row a turn at being validated and turns one noisy number into a mean and a spread.
K-fold cross-validation cuts the data into K equal-sized folds. For each fold: train on the other K − 1, score the held-out one. Every row is validated exactly once and trains K − 1 times; the K scores are averaged. K = 5 or K = 10 is the standard trade: more folds means more training data per fold and a less biased estimate, but K times the compute. Stratified K-fold deals each class into the folds so every fold keeps the dataset’s class ratio — the fix for the unlucky fold that catches no positives.
At the extreme, leave-one-out (K = N) trains N models on N − 1 rows each. It uses almost all the data every time, but it costs N fits and the N training sets are nearly identical, so the estimates are highly correlated; for many models that makes the average a surprisingly unstable estimate. K = 5 or 10 is the practical default, and repeated K-fold (shuffle, re-deal, repeat) is the cheap way to buy more stability.
The K scores are a sample, not a truth: report the mean and the spread, and compare models fold by fold (chapter 07). One more diagnostic: the learning curve plots train and validation scores against training-set size, and the validation curve plots them against a hyperparameter. If both scores are low and close, the model underfits; if training is high and validation is far below, it overfits. No amount of folds fixes the wrong model.
24 rows, one threshold fit on the training folds, validated on the held-out fold. Click through shuffles and K, and toggle stratification to watch the per-fold scores (and the class ratios) settle down.
At seed 50 with K=4 the plain shuffle deals folds with 4, 3, 2 and 3 positives and scores [0.500, 0.667, 1.000, 0.833] — mean 0.750, std 0.186. Toggle stratification and every fold holds three of each class and scores 0.833. The class ratio was the difference.
Leave-one-out is the same arithmetic with K = 24: 24 models, each trained on 23 rows. It is far more compute for an estimate whose correlation makes the spread hard to read.
In 5-fold cross-validation, how many times is each data point used for validation?
The test set is finite, so every metric is an estimate. The bootstrap turns one measured number into an interval — and stops your team from shipping a 0.002 difference.
“Accuracy 0.85” comes from 200 specific patients. Draw 200 different patients and the number moves. The bootstrap imitates that draw: take the measured results, resample them with replacement to the same size, recompute the metric, and repeat a thousand times. The spread of those thousand values estimates how much the metric would wobble across reruns; the middle 95% of them is a percentile confidence interval.
The interval is widest when the metric rests on few cases. Accuracy uses all 200 rows; recall rests on the 20 positives; precision on the 40 positive predictions. Same dataset, very different confidence — and no formula fixes that except collecting more of the scarce class. When comparing two models, use the same folds for both and study the paired differences: some of the fold-to-fold noise cancels, and the question becomes “is the mean difference large relative to how much the difference itself varies?”
The 200-patient test set is resampled with replacement 1,000 times. Each resample recomputes the metric; the middle 95% of those values is the confidence interval. Resample again to see how much the interval itself wobbles.
The dataset never changed — only the question did. Accuracy is pinned down by 200 rows; recall rests on just 20 actual positives, so its interval spans almost 0.4. More data would narrow every interval, and no amount of resampling can widen or shrink the underlying sample.
All four intervals come from the same 200 rows. Accuracy is pinned to ±0.05 because it uses every row; recall and precision swing by ±0.15 and more because they depend on the tiny positive class. That is the honest error bar for the headline sentence.
Answer before you look. The imbalance question and the calibration question are the two that separate a memorized metric from a working instinct.
0 / 6 answered · 0 correct
01Why should you never tune hyperparameters based on test-set performance?
02A dataset is 95% negative and 5% positive. A model predicts “negative” for every single sample. What is its accuracy?
03In K-fold cross-validation with K = 5, how many times is each data point used for validation?
04A learning curve shows training score 0.95, validation score 0.60, and the gap does not close as more data is added. What should you try?
05AUC-ROC = 0.5 for a binary classifier. What does this indicate?
06Model A and Model B rank every case in the same order (identical AUC), but Model A outputs 0.99 / 0.01 while Model B's probabilities match the observed frequencies. Which statement is true?
Eight cards. Click one to swap the lazy description for what it actually means — and which mistake it prices.
Four problems with exact numbers — curves, nested folds, a permutation test, and a model comparison. Try first; a worked answer is one click away.
The code sorts unique scores descending, marks predictions positive when score ≥ threshold, and records (recall, precision) at each step. Average precision sums the precision at every recall increase: AP = Σ (Rₙ − Rₙ₋₁)·Pₙ. On the lesson's 20-row score set the AUC is 0.781, obtained by counting concordant positive–negative pairs: 75 / 96. The AP is 0.743. Now replicate the 12 negatives 8×: the ROC points all normalize to the same fractions, so AUC stays exactly 0.781, but AP falls to 0.408 and precision at the 0.5 cutoff collapses from 0.600 to 0.158 (recall holds at 0.750). When positives are rare, precision's denominator absorbs every false alarm while ROC's FPR divides by an ever-larger negative count. Use PR when the minority class is what you care about; use ROC when you need one threshold-free summary and prevalence is stable.
Outer split: 5 folds. For each outer fold, hold it out; inside the remaining 4 folds run a second 3-fold CV over each candidate hyperparameter; pick the value with the best mean inner score; retrain on all 4 outer-training folds; score once on the held-out fold. Total fits per model = 5 × (3 × candidates + 1). The trap it closes is subtle: if you tune on the same folds you report, the reported score is the maximum of many noisy estimates and reads high. A clean comparison reports the outer-fold mean ± spread via a paired test, and the hyperparameter chosen inside a fold may differ from fold to fold, which is honest — the selection itself is part of the model.
Shuffle the label column only, keeping the features and the split structure, then rerun the whole pipeline. Under a good model with no leakage the shuffled scores hover around the majority-class baseline (about 0.50 here), because the features now carry no signal. With 100 shuffles, p = (number of shuffled scores ≥ observed + 1) / (100 + 1). Suppose the real model scores 0.85 and 3 shuffles reach it: p = 4/101 = 0.0396, significant at 0.05. If the shuffled scores stay suspiciously high, some feature is leaking the answer — that is exactly the diagnostic this test is famous for.
Model A mean = 4.00 / 5 = 0.800 with standard deviation √(0.0058/5) = 0.0341. Model B mean = 4.02 / 5 = 0.804 with standard deviation √(0.00012/5) ≈ 0.0049. The means differ by less than one tenth of Model A's spread, so the headline numbers are a tie. Compare fold by fold instead: differences A − B are [0.01, −0.02, 0.04, 0.00, −0.05], mean −0.004, standard deviation 0.0301, so t = −0.004 / (0.0301/√5) = −0.30. With 4 degrees of freedom, |t| = 0.30 is nowhere near the 2.78 needed at p < 0.05 — no evidence that either model is better. Model B is more stable, so ship B for predictability, but report that the ranking difference is inside the noise.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 02, Lesson 09) and the Math Foundations Notebook reference build. Interactive figures, the animated threshold hero, the hand-counted AUC and AP checks, the calibration and bootstrap labs, the precision/recall memory hook and worked exercise answers are original to this page. Every score a lab prints is computed live from the rows it displays.