A classifier is a function from 3,072 pixel values to a probability distribution over 10 classes. The model is the easy part. The dataset loop, the augmentation policy, the loss, the schedule and the evaluation are where the bugs live — and a broken pipeline quietly scores 70–75% where a correct one scores 93.45%.
CIFAR-10 is 60,000 images of 32×32 pixels in 10 classes: 50,000 for training, 10,000 for testing, 6,000 per class. Each image is nothing but 3,072 numbers between 0 and 255. Every vision task in the phase — detection, segmentation, retrieval — reduces to putting those numbers into a class. Getting the loop right is the skill that transfers.
32 × 32 × 3 = 3,072 values · 50,000 train / 10,000 testUnshuffled batches, ImageNet stats on CIFAR, labels misaligned, softmax before the loss, augmentation left on at eval — none of these raise an error, and the loss curve looks plausible the whole time. A ResNet setup that should hit 93% quietly lands in the 70s. The fix is not a better architecture; it is reading each stage's numbers.
simple CNN 80–85% · ResNet 93–95% · broken pipeline 70–75%Aggregate accuracy is one number; a confusion matrix is 100. Per-class precision asks how many flagged images were right; recall asks how many of the true images were found; F1 balances them. On the ResNet run, cat recall is 0.870 and ship recall is 0.965 — and the single worst pair, cat ↔ dog, costs 97 images in both directions.
P = TP/(TP+FP) · R = TP/(TP+FN) · F1 = 2PR/(P+R)A classifier is a function from 3,072 numbers to 10 probabilities: image [32,32,3] → logits [10] → softmax → loss. The architecture is a detail — the pipeline (dataset, augmentation, loss, optimizer, schedule, evaluation) is the deliverable, and each stage has one number that tells you whether it works.
By the end you will be able to name every stage of the CIFAR-10 pipeline and the bug it hides; predict a failure from a loss curve (flat at ln 10 = 2.3026 → labels, spike → learning rate, floor at 0.5448 → label smoothing is on, growing gap → overfitting); implement mixup, cutout and label smoothing from scratch and say why mixup raises the training floor to λ‑dependent values like 0.6109 at λ = 0.7; compute per-class precision, recall and F1 by hand from a confusion matrix; and recite the standard recipe — SGD momentum 0.9 or AdamW, cosine schedule, batch 128, dataset-specific normalization, and no augmentation at test time.
Detection classifies regions. Segmentation classifies pixels. Retrieval ranks by similarity to class centroids. Every vision task in this phase reduces to image classification — and most classification bugs are not in the network. They are one line of plumbing, and nothing raises an error.
A null pointer throws. A broken classifier runs to completion, prints a loss, draws a curve, and predicts. The source states the stakes in one sentence: a CNN that would hit 93% on CIFAR-10 with a correct setup commonly scores 70–75% with a broken one — and the loss curve looks plausible the whole time. Unshuffled batches, a swapped preprocessing stat, a softmax in the wrong place: each costs roughly the same as deleting a quarter of your architecture, and none of them crash.
Before wiring anything, memorize the dataset’s arithmetic, because every later number is a check against it. CIFAR-10 is 60,000 images of 32×32 pixels in three colour channels, split 50,000 train / 10,000 test across 10 classes. One image is 3 × 32 × 32 = 3,072 values; the whole set is 184,320,000 values — about 184 MB of raw bytes, and it was deliberately chosen to be small. Chance-level cross-entropy for ten classes is ln 10 = 2.3026: a loss parked there is not learning slowly, it is predicting the uniform distribution.
The lesson’s method is deliberately unglamorous: wire the whole pipeline by hand so every part is inspectable, and refuse to import anything from torchvision.datasets that could hide a bug. When the numbers are yours, a wrong one has an address. And when those labels are scarce, Lesson 05 (Transfer Learning & Fine-Tuning) starts from ImageNet weights instead of random ones.
| Component | Classic bug | What you see | First check |
|---|---|---|---|
| Dataset / labels | images and labels out of sync | held-out loss parked on ln 10 = 2.3026 from epoch 0 | print ten (image, label) pairs and look at them |
| Normalization | raw 0–255 pixels, or ImageNet stats on CIFAR | loss jumps to 12 → 41, then NaN by epoch 3 | recompute mean/std on the training split |
| DataLoader | shuffle=False in training | epoch-level sawtooth; converges late and noisily | shuffle=True for train, False for test |
| Model | output size ≠ classes; train()/eval() swapped | accuracy near chance, or dropout at test time | print the output shape; assert the mode |
| Loss | softmax applied before cross-entropy | loss stalls near 1.46 and never reaches the healthy band | pass raw logits to the loss |
| Optimizer | lr above the stability limit | spike to 9.4 by epoch 4, then permanent oscillation | lower lr 10×; check gradient norms |
| Scheduler | stepped per batch, or never stepped | never settles, or freezes after a few epochs | one scheduler.step() per epoch |
| Evaluation | augmentation left on at test time | held-out accuracy several points below its potential | eval transform = normalize only, model.eval() |
| Regularization | no augmentation, no weight decay | train → 0.03 while held-out bottoms at 0.80 and climbs | augmentation, weight decay, early stopping |
Every row in this table is a curve shape the loss-board lab draws in chapter 03. The point of the table is order: nine components, nine signatures, and the cheapest check first — data, shapes, one batch, gradients, rate.
A CIFAR-10 run finishes at 72% test accuracy and the loss curve looks plausible. Where is the bug most likely to be?
Every stage in this pipeline transforms one tensor into another. If you know the shape, the dtype and the value range at each boundary, you can find almost any bug by walking the batch — which is what this chapter does, once, slowly.
The source’s pipeline is a straight line: dataset → augment → normalize → batch and shuffle → model → logits → cross-entropy → backward → optimizer step → scheduler step, back to the model. The evaluation pass takes the same line but skips augmentation, flips model.eval(), turns off gradients, and accumulates a confusion matrix instead of an update.
Stage 1: the dataset. One image arrives as a 32×32×3 uint8 array with one integer label. The source’s ArrayDataset keeps exactly that contract: __getitem__ returns (img, label) and never touches anything else, so the transform is the only place pixels change.
def standardize(mean, std):
mean = np.array(mean, dtype=np.float32)
std = np.array(std, dtype=np.float32)
def _fn(img):
return (img - mean) / std
return _fn
def random_crop(pad=4):
def _fn(img):
h, w = img.shape[:2]
# reflect, not zeros: black borders are a signal the model
# would learn to ignore in a non-useful way
padded = np.pad(img, ((pad, pad), (pad, pad), (0, 0)), mode="reflect")
y = np.random.randint(0, 2 * pad + 1)
x = np.random.randint(0, 2 * pad + 1)
return padded[y:y + h, x:x + w, :]
return _fn
train_tf = compose(random_hflip(), random_crop(pad=4), standardize(mean, std))
eval_tf = standardize(mean, std) # NO augmentation at test timeStages 2 and 3: augment, then normalize. The order matters: pixel transforms run on 0–1 floats, and normalization comes last so the statistics it subtracts are not perturbed by the augmentation. For CIFAR-10 the stats are dataset-specific — mean (0.4914, 0.4822, 0.4465), std (0.2470, 0.2435, 0.2616). Three pixels on channel 0 tell you where the values land:
Stage 4: batch and shuffle. The DataLoader groups images into batches of 128 and reshuffles every epoch. A batch tensor is [128, 3, 32, 32]: 128 × 3,072 = 393,216 float32 values = 1.5 MiB, and one epoch over 50,000 images is 391 batches — 390 full ones plus a final batch of 80. shuffle=False looks innocent and produces an epoch-level sawtooth: with class-ordered data every batch is a single class, so the loss alternates between “already seen” and “brand new” while convergence crawls.
Stages 5–6: forward pass to loss. The model maps [128, 3, 32, 32] to [128, 10] logits — ten unbounded real numbers per image. Cross-entropy turns those into one scalar using the fused log-softmax; at initialization it reads ln 10 = 2.3026. Stage 7–9: backward, step, schedule. Autograd fills a gradient per parameter, the optimizer applies step = lr × |grad| shaped by momentum, and the scheduler decays the rate once per epoch.
def train_one_epoch(model, loader, optimizer, device, num_classes, use_mixup=True):
model.train() # 1. train mode: dropout + batch norm live
total, correct, loss_sum = 0, 0, 0.0
for x, y in loader:
x, y = x.to(device), y.to(device)
if use_mixup:
x_m, y_soft = mixup_batch(x, y, num_classes)
logits = model(x_m)
loss = soft_cross_entropy(logits, y_soft)
else:
logits = model(x)
loss = nn.functional.cross_entropy(logits, y, label_smoothing=0.1)
optimizer.zero_grad() # 2. before backward, once per step
loss.backward()
optimizer.step()
loss_sum += loss.item() * x.size(0) # 3. .item() so no graph survives
total += x.size(0)
with torch.no_grad(): # argmax against raw logits
pred = logits.argmax(dim=-1)
correct += (pred == y).sum().item()
return loss_sum / total, correct / totalTen stages, from the files on disk to the confusion matrix. At each step: the shape the tensors take, the values they hold, and the bug that hides there — because almost every classification bug is a pipeline bug.
The five invariants live across these stages: model.train() / model.eval(), zero_grad() before backward(), .item() on every accumulated metric, no_grad() at eval, and argmax over raw logits. Check them in this order and most failures name themselves.
You delete optimizer.zero_grad() from the loop, so gradients accumulate across steps. What does the run look like?
You cannot afford a debugging run per hypothesis, but you get a loss curve for free on every run. Nine components, nine characteristic shapes — and four exact numbers that anchor the y-axis.
The y-axis of a well-behaved CIFAR-10 run starts at ln 10 = 2.3026 and falls toward a floor set by your label recipe. That gives four reference values worth memorizing, because they are exact and they appear in the curve:
A flat line at chance is a data bug. If held-out loss stays on 2.3026 from epoch 0 while train loss falls, the model is fitting something the test set does not contain: misaligned labels, shuffled targets, or a train/validation split from two different worlds. The lab’s shuffled-label run keeps held-out exactly on ln 10 while train crawls from 2.303 to 2.157 — the model is memorizing noise. No amount of training fixes this.
A spike is a rate bug. A loss that jumps from 2.30 to 9.4 by epoch 4 and then oscillates above chance is a learning rate above the stability limit: oversized steps destroy what was learnt and the survivors never settle. The pathological version is raw pixels with no normalization — the input scale multiplies every gradient by about 255, the logits overflow exp(), and the curve reads 2.30 → 12.4 → 41.0 → NaN.
A growing gap is a regularization bug. Train loss going to 0.03 while held-out loss bottoms out near 0.80 at epoch 12 and climbs back to 1.07 is the textbook overfitting curve: the model has moved from learning the signal to memorizing the 50,000 training images. Augmentation, weight decay, and early stopping are the correct responses — and so is auditing labels, because noisy labels produce the same shape at lower epochs.
A crawl is also a rate bug. The lr-too-low run falls in a straight, almost horizontal line: 2.303 → 2.063 over twenty epochs, a 0.24 drop for an afternoon of GPU time. The curve is not wrong — the steps are just too small, which is why the learning-rate range test from Phase 3 exists.
Nine runs, one broken component each. The faint dashed line is the healthy reference — compare shapes, not just final numbers, because every one of these bugs raises no error.
The loss curve is the only free diagnostic you get on every run. Match its shape to a row above before changing code: a flat line at 2.303 is a data problem, a spike is a learning-rate problem, and a growing train/held-out gap is a regularization problem.
Held-out loss sits at exactly 2.3026 from epoch 0 while train loss slowly falls. What is the first thing to check?
A CNN inherits translation invariance from weight sharing and nothing else. Crops, flips, occlusion and lighting have to be taught — and the only way to teach them is to show the model pixels that exercise them. The contract is one sentence long: the label must survive the transform.
Every random transform is a statement: “these two images have the same label; learn the features that ignore the difference.” A flipped dog is still a dog; a dog with an 8×8 patch missing is still a dog; a crop of a dog is still a dog. That is why the source’s first two transforms are a 50% horizontal flip and a reflect-pad random crop. The rule breaks the moment the transform moves the label: 180° rotation turns a digit 6 into a 9, vertical flips turn “dog” poses into impossible anatomy, and a crop that is too large deletes the object. Augmentation helps only when the label survives.
The source’s crop uses mode="reflect" rather than zeros. A zero-padded crop carries a hard black frame, and the frame’s position leaks the crop offset — the network can learn to read the corners and partially undo the augmentation. Reflect padding mirrors real edge pixels, so every crop still looks like a photograph. It is a one-word change that shows up as roughly a point of test accuracy.
Mixup goes further: it interpolates both the image and the label. Sample λ ~ Beta(0.2, 0.2), mix two examples, and train against the mixed target. The model stops memorizing spiky one-hot answers and learns to behave smoothly between classes — which is the right prior for natural images and the reason the source calls it “the single cheapest robustness upgrade for any classifier”.
def mixup_batch(x, y, num_classes, alpha=0.2):
if alpha <= 0:
return x, torch.nn.functional.one_hot(y, num_classes).float()
lam = float(np.random.beta(alpha, alpha))
idx = torch.randperm(x.size(0), device=x.device)
x_mixed = lam * x + (1 - lam) * x[idx]
y_onehot = torch.nn.functional.one_hot(y, num_classes).float()
y_mixed = lam * y_onehot + (1 - lam) * y_onehot[idx]
return x_mixed, y_mixed
def soft_cross_entropy(logits, soft_targets):
# cross-entropy against a distribution; reduces to the usual loss
# when the target is exactly one-hot
log_probs = torch.log_softmax(logits, dim=-1)
return -(soft_targets * log_probs).sum(dim=-1).mean()Label smoothing is mixup’s cheaper cousin. Instead of a one-hot target, the true class gets 1 − ε and each of the other C − 1 classes gets ε/(C − 1). At ε = 0.1 over ten classes that is 0.9 and 0.0111 nine times. Two consequences follow, and both are measurable: the model can no longer drive logits to infinity, which improves calibration at essentially no accuracy cost, and the loss acquires a floor equal to the entropy of the target — 0.5448. If your training loss stops at 0.545 and refuses to move, nothing is broken: that is label smoothing doing its job. PyTorch has it built in since 1.10: nn.CrossEntropyLoss(label_smoothing=0.1).
Cutout is the third trick and the simplest: zero out a random square of the input. An 8×8 patch covers 64 of 1,024 pixels — 6.25% of a CIFAR image — which is small enough that the label survives and large enough that the model cannot rely on any single region. This is the same idea as random erasing, and it is four lines:
def cutout(img, size=8):
h, w = img.shape[:2]
y0 = np.random.randint(0, h - size)
x0 = np.random.randint(0, w - size)
img = img.copy()
img[y0:y0 + size, x0:x0 + size, :] = 0 # 64 / 1024 = 6.25% of pixels
return img
loss = nn.functional.cross_entropy(logits, y, label_smoothing=0.1)
# targets: 0.9 for the true class, 0.1/9 = 0.0111 for each of the other nine
# the target's entropy 0.5448 is the lowest loss the model can ever report| Trick | Add it when… | Skip it when… | Cost you can measure |
|---|---|---|---|
| Flip + crop | always — the first augmentation any vision pipeline gets | the label is mirror-sensitive (laterality, text, digits) | one word: padding_mode="reflect" |
| Cutout | the model overfits and objects are big enough to survive a 6.25% hole | objects are tiny, or train and val loss are already close | a regularizer: it can hurt when there is no overfitting to fix |
| Mixup | you want the cheapest robustness and calibration upgrade and can train longer | the run is already underfitting — harder targets do not help | train loss floor rises (0.6109 at λ = 0.7); expect slower early epochs |
| Label smoothing | almost always — ε = 0.1, one keyword argument | you are debugging the loss itself and want a clean zero floor | floors the loss at 0.5448, which makes “loss below x” checks meaningless |
One more number matters, because it explains mixup’s behaviour on short runs. β(0.2, 0.2) has mean 0.5 but is U-shaped: about ≈67% of draws land within 0.1 of 0 or 1 (barely mixed, almost the original example) and only about ≈6.5% land in the 0.4–0.6 middle (genuinely ambiguous blends). So most of a mixup batch is asked a question that has a precise answer, a small fraction is asked for an interpolation, and the epoch-average loss floor lands somewhere in the 0.35–0.5 range instead of 0. That is the entire mystery of “train loss went up and the model got better”.
Toggle the training-time transforms on a pair of toy images and watch what happens to the pixels — and to the target the loss is measured against. One rule survives every toggle: a pixel transform must leave the label alone.
Flip the pad mode and watch the corners: zero padding paints black bars the network can learn to detect, which is why the community default is reflect. Flip mixup on and the floor above rises — training loss should look worse, because the target now contains two answers.
Your training loss falls from 2.30 and then flattens at exactly 0.545 while held-out accuracy keeps improving. What is the most likely explanation?
A 10×10 confusion matrix counts what actually happened: entry (i, j) is the number of images of true class i that the model predicted as class j. The diagonal is right; everything else is a place to look.
Reading one class out of the matrix is a four-count exercise. For a given class, TP is the diagonal cell — true and predicted both that class; FP is everything else in its column — other classes that were predicted as it; FN is everything else in its row — its images predicted as something else. From those three counts:
That is the whole point of the chapter. The ResNet preset reports 93.45% accuracy — and cat recall is 0.870 while ship recall is 0.965. The aggregate number cannot tell you that, because it averages a class that is almost solved with one that is not. The source’s own warning is the same: aggregate accuracy hides imbalance, and per-class numbers are what surface underperforming categories.
Then look at where the misses go. The matrix’s off-diagonal mass is not spread evenly: on the ResNet run, cat ↔ dog costs 97 images in the two directions, deer ↔ horse costs 58, cat ↔ deer 50, automobile ↔ truck 45. On the simple-CNN preset the same pairs are worse — cat ↔ dog costs 260 and cat recall falls to 0.660 — but the ordering is identical. A concentration of errors on one off-diagonal pair is a localised problem with localised fixes: inspect those images. You will find genuinely ambiguous classes, mislabelled data, or a missing invariance, and the source’s advice is explicit — blanket changes like more data or a new optimizer rarely help when the failure is this specific.
def per_class_report(cm):
tp = cm.diag().float() # correct per class
fp = cm.sum(dim=0).float() - tp # column sum minus the diagonal
fn = cm.sum(dim=1).float() - tp # row sum minus the diagonal
prec = tp / (tp + fp).clamp_min(1)
rec = tp / (tp + fn).clamp_min(1)
f1 = 2 * prec * rec / (prec + rec).clamp_min(1e-9)
return prec, rec, f1
# rows are true classes, columns are predictions
for t, p in zip(y.cpu(), pred.cpu()):
cm[t, p] += 1Two runs on the same 10,000 test images: a simple CNN at ~80% and a ResNet at ~93%. Pick a true class, then compare its precision and recall — and look at which classes steal its predictions.
The ResNet still confuses cat with dog 97 times in both directions — more than any other pair. A single accuracy number would never tell you that; the matrix names the next dataset problem to fix.
The pipeline is plumbing, and plumbing has a recipe. These are the settings that get a ResNet into the 93–95% band on CIFAR-10 — and the wrong setting on any row is worth several accuracy points.
The source trains with an SGD optimizer, momentum 0.9, Nesterov, weight decay 5e-4, starting learning rate 0.1, and a cosine schedule that closes over T_max = 10 epochs. The learning rate is stepped by the scheduler once per epoch, after the optimizer step. That is the whole tuning stack — everything else in the pipeline is there to make those numbers behave.
| Knob | Default | Why this value |
|---|---|---|
| Batch size | 128 | 391 batches per epoch over 50,000 images; the gradient noise of a 128-batch is itself a regularizer |
| Optimizer | SGD momentum 0.9, nesterov, weight decay 5e-4 | the source's recipe; AdamW with a cosine schedule is the common modern substitute |
| Learning rate | 0.1 at the start, cosine to ~0 | SGD tolerates larger rates than Adam-style optimizers; the schedule closes the run |
| Schedule | CosineAnnealingLR(optimizer, T_max = epochs), stepped once per epoch | big steps early, tiny steps late — the model settles instead of bouncing |
| Normalization | CIFAR-10 mean/std, computed on the training split | dataset-specific: ImageNet stats here are a ~0.19-value error on every pixel |
| Augmentation | reflect-pad random crop + hflip (+ cutout, + mixup) | training only, label-preserving; the eval transform is normalize-only |
| Loss | cross-entropy on raw logits, label_smoothing = 0.1 | the fused log-softmax is the stable form; smoothing sets a 0.5448 floor |
| Evaluation | model.eval() · no_grad · no augmentation · argmax on logits | test-time augmentation is a different experiment — not the default pipeline |
from torchvision.datasets import CIFAR10
from torchvision.transforms import Compose, RandomCrop, RandomHorizontalFlip, ToTensor, Normalize
mean = (0.4914, 0.4822, 0.4465) # CIFAR-10's, not ImageNet's
std = (0.2470, 0.2435, 0.2616)
train_tf = Compose([
RandomCrop(32, padding=4, padding_mode="reflect"),
RandomHorizontalFlip(),
ToTensor(),
Normalize(mean, std),
])
eval_tf = Compose([ToTensor(), Normalize(mean, std)]) # no augmentation
train_ds = CIFAR10(root="./data", train=True, download=True, transform=train_tf)
val_ds = CIFAR10(root="./data", train=False, download=True, transform=eval_tf)device = "cuda" if torch.cuda.is_available() else "cpu"
model = TinyResNet(num_classes=10).to(device)
optimizer = SGD(model.parameters(), lr=0.1, momentum=0.9,
weight_decay=5e-4, nesterov=True)
scheduler = CosineAnnealingLR(optimizer, T_max=10)
for epoch in range(10):
tr_loss, tr_acc = train_one_epoch(model, train_loader, optimizer,
device, 10, use_mixup=True)
va_loss, va_acc, cm = evaluate(model, val_loader, device, 10)
scheduler.step() # once per epoch, after the update
print(f"epoch {epoch:2d} lr {scheduler.get_last_lr()[0]:.4f} "
f"train {tr_loss:.3f}/{tr_acc:.3f} val {va_loss:.3f}/{va_acc:.3f}")The schedule is lr(t) = 0.1 · (1 + cos(π·t/T)) / 2. Plug in real epochs with T = 10:
Two silent bugs live in that table. Stepping the scheduler per batch instead of per epoch finishes the entire cosine inside the first 10 batches of epoch 1 — 0.2% of the run gets the schedule, and the remaining 99.8% trains at lr ≈ 0, which the loss curve reports as a very early plateau. Never stepping it at all means the run oscillates at 0.1 forever and leaves accuracy on the table late in training. Both bugs produce a plausible curve, and both are a single line to fix.
The normalization row has the same character. With CIFAR’s own stats, pixel 255 maps to (1.0 − 0.4914)/0.2470 = 2.059; with ImageNet stats it maps to 2.249. That 0.190 error rides on every pixel of every image and every gradient that flows back, and the source’s estimate is roughly a point of accuracy that nobody catches until someone profiles the model.
Accuracy answers one question badly: how often is the top guess right? Two more numbers decide whether a classifier is safe to ship — whether the right answer is near the top, and whether the model’s confidence means anything.
Top-1 vs top-5. Top-1 accuracy is the fraction of images whose true class is the single highest-probability prediction; top-5 is the fraction whose true class is anywhere in the five highest. Chance levels are exact and easy to remember: k/C, so on CIFAR-10 chance is 10% top-1 and 50% top-5. Top-5 exists because image classes are genuinely ambiguous at the edges — ImageNet’s Norwich terrier and Norfolk terrier differ by ear shape — so top-5 separates “the model has never seen anything like this” from “the model is deciding between two nearly identical breeds”. A 93.45% top-1 model with top-5 near 100% makes near-misses; the same top-1 with a weak top-5 is making different kinds of errors. The ResNet preset has 655 errors (10,000 − 9,345 of them); top-5 asks how many kept the true class in ranks 2–5. Aggregate accuracy is silent on the difference.
Calibration. Accuracy asks whether the prediction is right; calibration asks whether the confidence is honest. Expected Calibration Error (ECE) bins predictions by confidence and compares the bin’s average confidence to the bin’s observed accuracy; the weighted average of those gaps is one number. Take three bins from a miscalibrated model:
The fix is almost free. Divide the logits by a single scalar T > 1 before the softmax, tune it on a validation set, and the confidences line up with the accuracies without changing a single prediction — the ranking is untouched. Label smoothing does something similar during training: it stops the model from driving logits to infinity, which is exactly what makes the confidences dishonest in the first place.
Type your own 3×3 confusion matrix — rows are true classes, columns are predictions — and read accuracy, per-class precision, recall and F1. Load the majority-trap preset to watch accuracy say 0.900 while two classes score zero.
Watch the majority-trap preset: accuracy 0.900 and the baseline are the same number, because predicting class 0 for every row is the baseline. Per-class recall collapses to 0.000 for both minority classes — that is the failure accuracy cannot see.
This is the last measurement you run before shipping, and it is worth running on your own numbers. Type a matrix you have seen — the clean 3-class one, the majority trap, or the CIFAR cat/dog/deer block — and read the per-class precision, recall and F1 next to the accuracy. The lesson’s deliverables follow directly: a pipeline auditor prompt that checks a training script against the five invariants and reports the first violation, and a classification diagnostics skill that takes a confusion matrix plus class names and proposes the single most impactful fix. Both are exercises in reading the numbers you just learned to read.
Answer before you look. The softmax-before-cross-entropy question and the confusion-matrix question are the two that separate a pipeline you have memorized from one you can debug at 2 a.m.
0 / 5 answered · 0 correct
01Your model outputs raw logits of shape (N, C). You write `loss = cross_entropy(softmax(logits), y)`. What goes wrong?
02You evaluate a 10-class classifier and get 92% accuracy. Class 0 has 9,000 examples; classes 1–9 share 1,000 (about 111 each). What does the headline number hide?
03Mixup replaces one-hot labels with interpolated soft targets like λ·y_i + (1−λ)·y_j. Why does this help generalisation?
04You swap `RandomCrop(32, padding=4, padding_mode='zeros')` for `padding_mode='reflect'` on CIFAR-10. Why is reflect better here?
05You train a classifier and the confusion matrix shows most errors are class 3 predicted as class 5 and class 5 predicted as class 3. What is the single most impactful next step?
Click a card to swap the lazy description for what it actually means — every definition carries the number that makes it checkable.
Three problems with exact numbers: run the mixup ablation and explain a higher training floor, implement cutout and compare four augmentation policies, and build the CIFAR-100 pipeline with a learning-rate sweep and a top-confusions table. Try first; a worked answer is one click away.
Run both arms with the same seed, the same optimizer and the same schedule; change only `use_mixup`. With mixup on, the target is a mixture and its entropy is the loss floor: at λ = 0.7 the floor is −(0.7 ln 0.7 + 0.3 ln 0.3) = 0.6109, at λ = 0.5 it is ln 2 = 0.6931, and a draw near λ = 0.9 has a floor of only 0.3251. Because λ ~ Beta(0.2, 0.2) is U-shaped, most batches are barely mixed and only ≈6.5% land in the 0.4–0.6 middle, so the epoch-average floor typically sits in the 0.35–0.5 range — visibly above the one-hot run's 0. That is the whole explanation: the mixup run is not training worse, it is being asked an unanswerable question for part of each batch ("this image is 70% cat and 30% dog") and answering it smoothly. Expected result on the synthetic set: train loss ~0.4–0.7 higher, held-out loss a little lower at epoch 5, accuracy within a point; on real CIFAR-10 with a small CNN, expect roughly +1 point of test accuracy and noticeably better calibration. Plot both train curves and both val curves on the same log-scale axes; the shape to look for is a higher train floor with a smaller train/val gap.
The implementation is three lines: pick x0, y0 uniformly in [0, 24), then `img[y0:y0+8, x0:x0+8] = 0` before normalizing. An 8×8 square covers 64 of 1,024 pixels — 6.25% of the image — which is the sweet spot the original paper found for CIFAR-style data; larger patches start deleting the object and the label stops surviving the transform. Keep the random seed fixed across arms and report test accuracy, not train. Expected ordering on CIFAR-10 with a small CNN: none (baseline, the 80.5%-preset ballpark) < hflip+crop (+1 to 2 points) < hflip+crop+cutout (another +0.5 to 1) ≈ hflip+crop+mixup (mixup is stronger on longer schedules and often looks worse at epoch 5 because its loss floor is high). Two honest caveats: cutout is a regularizer, so it helps most when the model overfits — if train and val are already close, it can hurt; and it interacts with mixup, so do not stack every trick at once. The deliverable is the four-row table plus one line of interpretation per row: augmentation helps only when the label survives.
Everything in the lesson transfers unchanged except the numbers. Chance loss becomes ln 100 = 4.6052 (versus 2.3026). The dataset is 50,000 train / 10,000 test with 500 train and 100 test images per class, so the same batch 128 gives 391 batches per epoch and the test pass is 79 batches. Read the ResNet-34 paper's CIFAR-100 top-1 number for your target and hold yourself to within 1 point. The sweep is the lesson's real content: baseline lr 0.1, cosine to 0, weight decay 5e-4, batch 128 — then lr ×3 and lr ÷3, and weight decay 5e-4 vs 5e-3, one variable per run. Log epoch, wall-clock seconds, learning rate, train loss, val loss and val accuracy to CSV, then plot loss against step (not epoch) for the six runs on one axis. Diagnostics to report: whether the best run was still improving when the cosine closed (if yes, train longer; if val loss bottomed early, regularize), and whether the top-confusions table shows the CIFAR-100 superclass structure (people, animals, vehicles) clustering errors — it usually does, which is the 100-class version of cat ↔ dog. If your best run misses the published number by more than 1 point, the culprit is a pipeline detail, not the architecture: normalization stats, the reflect-pad crop, or the schedule.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 04, Lesson 04) and the Math Foundations Notebook reference build. The five labs (the loss-curve diagnostic board, the augmentation playground, the 10×10 confusion explorer, the metric calculator and the pipeline stepper) are original to this page, as are the pipeline arithmetic (184,320,000 values, 391 and 79 batches, 1.5 MiB per batch), the normalization hand-checks and the ImageNet-stats gap, the label-smoothing floor 0.5448, the mixup floors with the Beta(0.2, 0.2) U-shape, the double-softmax floor 1.4612, the no-shuffle sawtooth and raw-pixel NaN paths, the two 10×10 CIFAR-10 confusion matrices with their per-class metrics and worst pairs, the top-k chance arithmetic, and the three-bin ECE example. The stylized teaching curves in chapter 03 are labelled as such; every other number is computed live by the labs or verified by hand in the prose.