The do-nothing model scores 99% and catches nothing. Resampling, class weights and one moved threshold drag the boundary toward the rare class — and a live readout shows accuracy fall while AUPRC climbs.
With 1% positives, answering “no” forever is right 99% of the time. Accuracy counts the easy majority and ignores the reason the model exists. Always print the majority baseline beside it — and read recall, precision, F1, MCC or AUPRC instead.
baseline 0.990 · recall 0.000TP, FP, FN, TN are the whole scoreboard. Precision = TP/(TP+FP): of the rows we flagged, how many were real? Recall = TP/(TP+FN): of the real rows, how many did we catch? A low threshold buys recall with false alarms; a high threshold does the reverse.
precision = purity of flags · recall = reachRandom oversampling duplicates; SMOTE interpolates; class weights rescale the loss; the threshold moves the decision without touching the model. The first three need a retrain — the last one is a for-loop over cutoffs. Try the order: measure, then threshold, then reweight or resample.
data → loss → decisionImbalanced learning is deciding which mistake you can afford, then buying the other one cheaply: metrics that price both classes (recall, precision, F1, AUPRC, MCC), four levers that shift the boundary or the loss (resampling, SMOTE, weights, thresholds), and an evaluation protocol that keeps the rare class visible from validation all the way to production monitoring.
By the end you will be able to read a confusion matrix and compute every metric from it, say what each resampling strategy costs, interpolate a SMOTE point by hand, set a threshold from a cost ratio, and name the evaluation mistakes — leaks, unstratified folds, feedback loops — before they reach production.
One number can hide the entire reason a model exists. When 99% of the data is “normal”, predicting normal forever is the rational way to maximize accuracy — and a perfect way to catch nothing.
You build a fraud detector. It reports 99.9% accuracy. You celebrate. Then you look at the confusion matrix and find the model predicted “legitimate” for every single transaction. That is not a bug: with 0.1% fraud, guessing the majority class is the rational way to minimize total error. The model is technically correct and completely useless.
Accuracy treats every correct prediction as one point. Correctly clearing a legitimate transaction and correctly catching a fraudulent one both count as 1. But catching fraud is the entire reason the model exists. With 1,000 rows — 990 negative, 10 positive — a do-nothing classifier scores (0 + 990) / 1000 = 0.990, catches 0 / 10 = 0% of the positives, and beats almost any model that actually tries.
| 1,000 rows | Predicted positive | Predicted negative |
|---|---|---|
| Actually positive | 0 (TP) | 10 (FN) |
| Actually negative | 0 (FP) | 990 (TN) |
The always-negative model: accuracy = 990/1000 = 0.990, precision = 0, recall = 0, F1 = 0, MCC = 0. Every metric except accuracy agrees the model is worthless — which is exactly why this lesson stops using accuracy as the headline.
Raise the positive rate and watch the two headline numbers move in opposite directions. The classifier predicts “negative” for every row — and accuracy still loves it.
At 1 in 1,000, accuracy reads 0.999 while recall and F1 are 0.000, and the AUPRC any classifier must beat is 0.001. Accuracy measures the majority; AUPRC measures the rare class.
The base rate (prevalence) is the share of rows that are positive. It sets both the do-nothing baseline and how dramatic a failure looks.
The rarer the positive class, the higher the do-nothing baseline and the less accuracy has to say. At 0.1% prevalence, a model must exceed 99.9% accuracy just to be interesting — and the number still tells you nothing about how many frauds it caught.
A dataset is 95% negative and 5% positive. A model predicts “negative” for every row. What is its accuracy?
Every metric that survives imbalance is built from the same four counts. Each one answers a different question about the mistakes you are willing to make.
Take a screening set of 1,000 patients where 10 have the disease. The model flags 25 people: TP = 5, FP = 20, FN = 5, TN = 970. Accuracy is (5 + 970)/1000 = 0.975 — and it is the only number here that looks good.
Precision is about the predictions: of the 25 rows we flagged, 5 were real, so P = 5/25 = 0.200. High precision means few false alarms. Recall is about reality: of the 10 real cases, we found 5, so R = 5/10 = 0.500. High recall means few misses. They pull against each other along the same threshold, which is why one number is never the whole story.
| Metric | Formula | Plain English | Value / example |
|---|---|---|---|
| Precision | TP / (TP + FP) | Of everything we flagged, how many really are positive? | 5/25 = 0.200 |
| Recall | TP / (TP + FN) | Of everything truly positive, how many did we catch? | 5/10 = 0.500 |
| F1 | 2PR / (P + R) | The harmonic mean: one strong score cannot carry the other. | 0.286 |
| AUPRC | area under PR | Precision and recall at every threshold at once; baseline = positive rate. | 0.649 on the lab's 50-row list |
| MCC | (TP·TN − FP·FN)/√(…) | A correlation that stays high only when both classes are handled well. | 0.306 |
F1 = 2·0.200·0.500 / (0.200 + 0.500) = 0.286 — the harmonic mean refuses to reward the strong recall. MCC = (5·970 − 20·5) / √(25·10·990·975) = 0.306, a balanced score that stays low even though accuracy reads 0.975.
F-beta moves the balance: it is the same harmonic mean with recall weighted β² times as much as precision. With P = 0.200, R = 0.500, F2 = 5·0.200·0.500 / (4·0.200 + 0.500) = 0.385 when a miss is what hurts, and F0.5 = 1.25·0.200·0.500 / (0.25·0.200 + 0.500) = 0.227 when a false alarm is what hurts.
AUPRC puts precision on the y-axis and recall on the x-axis at every threshold, then measures the area. Its baseline is the positive rate: on this 10% problem a random ranking scores ~0.10, and on the lesson’s 50-row score list the model reaches 0.649. AUC-ROC starts at 0.5 no matter how rare the positives are — which is why it can look excellent while every practical threshold is a disaster.
Two degenerate models bracket the problem: catch nothing (perfect specificity, zero recall) or catch everything (perfect recall, terrible precision). Any honest report places the model between them with counts and at least one balanced metric.
When the two mistakes cost differently, write the costs down. The table below says a missed fraud costs 100× a false alarm; then the cheapest decision is to flag a row whenever p > C_FP / (C_FP + C_FN) — at 100:1 that is 1/101 = 0.0099, so a transaction scoring 0.15 is flagged. At 1:1 the same rule demands p > 0.5 and lets it through. The cost matrix, not a round number, sets the boundary.
| Cost-sensitive view | Predict positive | Predict negative |
|---|---|---|
| Actually positive | 0 (correct) | C_FN = 100 · C_FP |
| Actually negative | C_FP = 1 | 0 (correct) |
A model has TP = 5, FP = 20, FN = 5 — the screening set above. Which pair of numbers should its report lead with?
The first fix is the blunt one: make the classes look balanced before training. Oversampling duplicates the minority; undersampling discards the majority. Both buy attention for the rare class, and each pays for it differently.
Random oversampling repeats minority rows until the counts match. A 60/6 training set becomes 60/60 with 54 duplicate rows; the 950/50 set from the source becomes 950/950 with 900 duplicates. The model now sees the minority as often as the majority — but it sees the same six points over and over, so it can memorize them instead of learning the region they live in.
Random undersampling reaches balance from the other side: keep all 6 minority rows and randomly keep 6 of the 60 majority rows. Training is fast and the classes are balanced, but 54 majority rows — real information about what “normal” looks like — are simply gone. The variance of the fitted boundary goes up.
| Strategy | Data changed | Risk | When to use |
|---|---|---|---|
| Random oversample | Minority rows duplicated | Overfitting to exact copies, slower training | Small datasets, moderate imbalance |
| Random undersample | Majority rows removed | Information thrown away, higher variance | Large datasets, training time matters |
| SMOTE | Synthetic minority rows added | Boundary noise; needs enough minority rows | Moderate imbalance with a usable minority cluster |
Resampling changes the rows the model fits, not the rows it is judged on. Every boundary below is a simplified 600-step logistic fit — no L2, no early stopping — and the held-out test points are never resampled.
| strategy | train maj/min | R | P | F1 |
|---|---|---|---|---|
| Original 10:1 | 60/6 | 0.500 | 1.000 | 0.667 |
| Undersample majority | 6/6 | 0.750 | 0.429 | 0.545 |
| Oversample minority | 60/60 | 1.000 | 0.667 | 0.800 |
Oversampling raises recall to 1.000 by showing the minority rows many times; undersampling reaches the same balanced counts by throwing majority rows away. On this small, overlapping cloud the undersampled fit is the worst of the three — 6 majority rows were not enough to locate the boundary.
The lab trains the same 600-step logistic fit on three training sets and judges all of them on the same held-out 24 rows (20 majority + 4 minority). Only the training rows change.
Notice the undersampled fit is the worst of the three: with only 6 majority rows it misplaced the boundary it was supposed to learn, trading some false alarms for the wrong ones. Undersampling is not a free lunch — it is a bet that the majority rows you discard were redundant. On a large, genuinely redundant dataset that bet usually wins; here it did not.
Your training set has 4 million rows and 0.4% positives, and each epoch already takes 20 minutes. Which strategy does the lesson's table recommend first?
Random oversampling shows the model the same minority rows again and again. SMOTE builds new ones that are plausible but not copies: points on the line segments connecting minority neighbours.
SMOTE (Synthetic Minority Oversampling Technique) runs three steps:
x, find its k nearest neighbours among other minority rows.new = x + t · (neighbour − x) with t a random number in [0, 1].In plain English: take a real minority point, walk a random fraction of the way toward a nearby minority point, and drop a synthetic point there. Because t never leaves [0, 1], the new row lands inside the minority cluster’s region — never outside it. And because it is between two real points, no real row is ever repeated exactly.
Pick a minority row, pick one of its k nearest minority neighbours, and slide t along the line between them. The majority cloud is context — SMOTE never consults it.
t = 0.40 with the docs’ points (1.0, 2.0) → (1.5, 2.5) gives exactly (1.20, 2.20). Duplication would have produced no new point at all — interpolation is the whole idea.
The lab’s six minority rows include the source’s three points. For x = (1.0, 2.0) the nearest minority neighbours, sorted, are (0.8, 1.4), (1.7, 2.1), (1.5, 2.5) — with distances 0.632, 0.707 and 0.707.
Compare with duplication: random oversampling of x would produce (1.0, 2.0) again, ten times. SMOTE produces a family of distinct points along the segment, which gives the model a region rather than a set of repeated coordinates.
The data can stay untouched. Multiply each row’s contribution to the loss by a class weight and the same optimizer, on the same rows, leans toward the minority.
Class weights make a mistake on a rare row count for more than a mistake on a common one. The standard balanced choice for a class with n_c rows in a dataset of n rows with two classes is:
In plain English: a positive row with weight 10 counts as ten positive rows in the gradient. Misclassifying one positive costs as much as misclassifying about nineteen negatives. The weighted loss is the old cross-entropy with a per-row multiplier:
The slider scales how much each positive training row contributes to the loss. The dashed boundary is the unweighted fit; watch the solid one rotate toward the minority as w+ grows.
Recall goes 0.500 → 1.000 while precision goes 1.000 → 0.667: the weighted loss buys rare-class catches with false alarms, which is the trade the next chapter prices with a threshold.
Take the lesson lab’s weights (0.55 for majority, 5.5 for the minority) and one row of each class.
That ratio is why class weighting is described as oversampling in expectation: duplicating every minority row ten times changes the average gradient in exactly the same direction, without storing or training on a single duplicate.
Most classifiers output a probability. Predicting “positive” at 0.5 is a convention, not a law — and under imbalance it is usually the wrong one. This is the cheapest fix in the lesson: no retraining, no new data.
A fraud model outputs P(fraud) = 0.15 for a fraudulent transaction. At the default threshold 0.5 it is classified as not fraud — missed. Lower the threshold to 0.10 and the same model, with the same weights, catches it. Threshold tuning changes the decision, not the model.
The process is a loop, not a formula: train once, get predicted probabilities on the validation set, sweep the cutoff from 0 to 1, compute the metric you care about at each cutoff, and pick the winner. The model’s ranking is what it contributes; the cutoff turns that ranking into the confusion matrix the business lives with. Calibration still matters for interpreting probabilities as odds, but ranking is all that threshold separation requires. If frauds reliably score above non-frauds, some cutoff separates them.
Fifty scored rows, five positives. Move τ, and the operating point slides along the PR curve; change the cost ratio, and the cheapest point moves. All five numbers come from the displayed rows.
With equal costs the cheapest answer is precise (τ ≈ 0.43); at 10:1 or 100:1 it slides down to τ ≈ 0.09 and trades 9 extra false alarms for the last missed positive. The model never changed — only the price of being wrong did.
Five positives, 45 negatives, average precision 0.649. The table walks the cutoff down and recounts the confusion matrix from the displayed rows.
Notice what did not change: the scores, the ranking, the model. AUPRC stayed 0.649 no matter where the cutoff sat. Choosing a threshold is choosing a point on the curve — the curve itself is the model’s quality, and improving it needs chapters 3–5.
Your operating point has precision 0.90 and recall 0.40 at τ = 0.80. The fraud team says too many frauds are slipping through. What is the direct fix?
Imbalance makes good evaluation fragile and drift easy to miss. The protocol — stratified folds, grouped splits, resampling inside the fold — is what keeps the numbers honest; the monitoring is what keeps them true tomorrow.
Stratified cross-validation deals each class across the folds in proportion before shuffling inside each fold. With 100 rows and 10 positives, every fold gets 2 positives; without it, the shuffle can hand one fold zero positives, and recall on that fold is undefined — you cannot catch what was never there. The estimate you average over folds then wildly overstates or understates the model.
Two more splits matter when rows are not independent. Grouped splits keep every row of the same patient, user or device in one fold: otherwise a model can memorize a patient in training and recognize them in validation, scoring well without generalizing. Time-ordered splits train on the past and validate on the future, because the deployment question is always “what happens next?”, never “what would have happened on a random row from the same week?”.
One hundred rows with ten positives. Toggle stratified splitting and reshuffle; watch the per-fold positive rate. A fold that lands zero positives cannot measure recall at all.
Every fold is still 20 rows; the difference is how many of them are positive. With a rare class, a plain shuffle can silently create a validation fold with no examples of the class you care about most.
| fold | rows | positives | positive rate | share |
|---|---|---|---|---|
| fold 1 | 20 | 2 | 10.0% | |
| fold 2 | 20 | 2 | 10.0% | |
| fold 3 | 20 | 2 | 10.0% | |
| fold 4 | 20 | 2 | 10.0% | |
| fold 5 | 20 | 2 | 10.0% | |
| whole dataset | 100 | 10 | 10.0% |
The base rate also drifts after deployment. A model trained at 1% positives and run during a fraud wave at 3% sees the same ranking produce very different precision. That is why the monitoring target is precision at a fixed recall: it moves with the world even when nothing in the model changed.
Answer before you look. The threshold-direction question and the SMOTE question are the two that separate a memorized definition from a working instinct.
0 / 6 answered · 0 correct
01A fraud detection dataset is 99.9% legitimate and 0.1% fraud. A model predicts “legitimate” for every transaction. What is its accuracy?
02Which metric correctly identifies the always-predict-negative model as useless?
03How does SMOTE generate synthetic minority samples?
04You lower the classification threshold from 0.5 to 0.3 on an imbalanced dataset. What happens to precision and recall?
05Why is AUPRC more informative than AUC-ROC for highly imbalanced datasets?
06A training set has 950 negative and 50 positive samples. Using the balanced class-weight formula n / (2 · n_class), what weight does the positive class get?
Click a card to swap the lazy description for what it actually means.
Four problems with exact numbers — a five-ratio base-rate table, a cost matrix, SMOTE interpolated by hand, and the weighted-loss arithmetic. Try first; a worked answer is one click away.
Every row keeps recall = 0.600 by construction. 50/50 (500/500): baseline 0.500, accuracy 0.750, precision 0.857, F1 0.706. 70/30 (300/700): baseline 0.700, accuracy 0.810, precision 0.720, F1 0.655. 90/10 (100/900): baseline 0.900, accuracy 0.870, precision 0.400, F1 0.480. 95/5 (50/950): baseline 0.950, accuracy 0.885, precision 0.240, F1 0.343. 99/1 (10/990): baseline 0.990, accuracy 0.897, precision 0.057, F1 0.104. The last row's counts are TP 6, FN 4, FP 99, TN 891: accuracy still reads 0.897, but the do-nothing baseline is 0.990 — the model is now worse than useless by the metric everyone quotes. Precision collapses because the 99 false alarms dwarf the 6 real catches, and F1 follows precision down. The fix is never "more accuracy"; it is a strategy aimed at the minority class.
Derivation: flagging costs (1−p)·C_FP in expectation and not flagging costs p·C_FN, so flag when (1−p)·C_FP < p·C_FN, i.e. p > C_FP/(C_FP+C_FN). Numeric checks: 1:10 gives 1/11 = 0.0909 and 1:100 gives 1/101 = 0.0099 — the docs' fraud example (p = 0.15) is flagged in both, but not at 1:1, where the cutoff is 0.5. On the 50-row list (5 positives, 45 negatives): with C_FN = 1 the best total cost is 3 at τ = 0.425 (TP 4, FP 2, FN 1); with C_FN = 10 and with C_FN = 100 the best total cost is 11 at τ = 0.089 (TP 5, FP 11, FN 0). Why it moves: lowering τ from 0.425 to 0.089 adds 9 false alarms (2 → 11) and saves the last miss. At 1:10 that trade costs 9 and saves 10, so it is worth it; at 1:1 it costs 9 and saves 1, so the optimizer refuses. Write the cost ratio down before you tune.
Formula: new = x + t · (neighbour − x). First: (1.0, 2.0) + 0.4·((1.5, 2.5) − (1.0, 2.0)) = (1.0 + 0.2, 2.0 + 0.2) = (1.20, 2.20). The parent distance is √((1.5−1.0)² + (2.5−2.0)²) = √0.5 ≈ 0.707, so the new point sits 0.4 · 0.707 ≈ 0.283 from x — this is what "on the line segment" means. Second: (2.0, 1.5) + 0.75·((1.0, 2.0) − (2.0, 1.5)) = (2.0 − 0.75, 1.5 + 0.375) = (1.25, 1.875); the parent distance is √(1² + 0.5²) = √1.25 ≈ 1.118, so the point sits 0.75 · 1.118 ≈ 0.838 along the segment. Implementation detail: k is capped at n_minority − 1, so six minority rows allow k ≤ 5, and with fewer than two minority rows SMOTE cannot run at all. Because the synthetic point always lies between two real points, SMOTE never invents a value outside the minority region's hull — copies are replaced by plausible interior points, but boundary noise is inherited too.
Weights: w_neg = 1000/(2·950) = 0.5263 and w_pos = 1000/(2·50) = 10.0 — a ratio of exactly 19, matching 950/50. Weighted cross-entropy per row is −w · [y·log p + (1−y)·log(1−p)]. A positive row at p = 0.20 has unweighted loss −log(0.20) = 1.6094, weighted 10.0 · 1.6094 = 16.094; a negative row at p = 0.02 has −log(0.98) = 0.0202, weighted 0.5263 · 0.0202 = 0.0106. The misclassified positive contributes about 1,500× the loss of the barely-wrong negative until the model fixes it. Only the ratio matters because multiplying every weight by a constant c multiplies the total loss and its gradient by c; a fixed learning rate would then take c× larger steps, but once the gradient is normalized (here it is divided by the sum of weights) the update is unchanged. The model cares that positives count 19× negatives, not that the positive weight is literally 10.0 — which is exactly why class weights and oversampling produce the same decision boundary in expectation.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 02, Lesson 17) and the Math Foundations Notebook reference build. The hero's 99:1 margin animation, the 1,000-cell imbalance simulator, the three-fit resampling lab, the SMOTE interpolation lab, the class-weight lab, the cost-sensitive threshold lab and the stratified-CV console are original to this page. Every metric, boundary, weight and cost shown is computed live from the displayed rows with a deterministic teaching model.