A broken network does not crash. It prints a loss, draws a curve, and keeps going — while it memorizes noise or quietly predicts the uniform distribution. This is the debugger’s toolkit: read the curve, isolate one variable, prove the model can learn eight examples before it ever sees eight million.
Traditional bugs crash; neural bugs print a number. The loss decreases, the predictions look plausible, and the model is wrong anyway — memorizing noise or predicting the uniform distribution. The source cites an estimate that 60–70% of ML debugging time goes to these silent failures. Your first job is to make the failure visible.
2 classes: −log ½ = ln 2 = 0.6931 · 10 classes: ln 10 = 2.3026The golden rule: start simple, add complexity one piece at a time, verify each piece. Concretely: visualize the data, check shapes with asserts, overfit one tiny batch, watch gradient norms and activation statistics, sweep the learning rate — then scale. The overfit-one-batch test takes thirty seconds and catches broken losses, backward passes, optimizers and architectures.
DATA → SHAPES → ONE BATCH → GRADIENTS → RATEAdd forward and backward hooks that record each layer's activation mean/std/zero-fraction and gradient norm. Log loss, learning rate, gradient norms and weight histograms every N steps. Alert on NaN loss, on layers that are 100% zero across a batch, and on gradients that exceed their usual order of magnitude. Gradient clipping is the seatbelt; normalization is the fix.
grad norm: 0 dead · 1e-7 vanishing · 1 healthy · 1e3 explodingA network has no stack trace — it has numbers, so debugging is reading the numbers in the right order: data, shapes, one batch, gradients, learning rate. Isolate one variable per run and make every failure loud enough to see.
By the end you will be able to read a loss curve and name the likely cause from its shape; compute the chance level ln C for any number of classes; explain dead ReLUs, vanishing and exploding gradients with their arithmetic (σ′ ≤ 0.25, 0.25¹⁰ ≈ 9.5×10⁻⁷; ‖[3,4,−12]‖ = 13, clip → ×1/13); run the overfit-one-batch test and say what it can and cannot see; verify a backward pass with a central finite difference; sweep the learning rate; and carry a seven-point checklist that catches the classic PyTorch bugs before a full run burns an afternoon.
A null pointer throws. A type mismatch fails at compile time. A broken network runs to completion, prints a loss, draws a curve, and outputs predictions. Nothing tells you it is wrong — except the numbers, if you know which ones to read.
The source opens with an estimate that 60–70% of ML debugging time goes to “silent” failures: bugs that raise no error and degrade quality. Karpathy’s Recipe for Training Neural Networks (2019) starts from the same place — “the most common neural net mistakes are bugs that don’t crash”. The field’s most expensive bugs are differences of one line: a missing zero_grad(), a transposed dimension, a learning rate off by 10×.
The first skill is knowing what the loss should look like. Cross-entropy assigns a number to a probability distribution: if a model knows nothing and outputs the same probability 1/C for every one of C classes, the loss is −log(1/C) = ln C. That is the chance level, and it is worth memorizing:
The second skill is reading a loss value as confidence, not just correctness. Take logits (0.9, 0.1) for a two-class problem. Softmax exponentiates and normalizes:
This is why a model can be 99% accurate and nearly useless. A fraud detector that answers “legitimate” for every transaction is right 99 times in 100 when 1% of transactions are fraudulent, and its loss settles at the base-rate entropy −(0.99·ln 0.99 + 0.01·ln 0.01) = 0.056 — lower than where many healthy models start. Accuracy is the wrong dial; the loss, the confusion matrix and the held-out set are the right ones.
The mindset that follows has one golden rule, straight from the source: start simple, add complexity one piece at a time, and verify each piece independently. The feedback loop is slow — minutes to hours per training run — so each experiment must change exactly one variable. “It got worse” is only information if you know what changed.
Five loss curves from the same tiny network, each run with one thing changed. The faint dashed line is the healthy reference — compare shape, not just the final number.
The number to remember is ln 2 = 0.693: a two-class classifier that has learned nothing outputs 0.5 for both classes, and −log 0.5 = 0.693. A loss parked there is not converging slowly — it has found the uniform distribution and stopped moving.
Your two-class classifier's loss has been sitting at 0.693 for many epochs. What is the model actually outputting?
One flat curve can mean a learning rate 1000× too small, labels that never reached the model, or ReLUs that all died on step one. The way through is a taxonomy: each symptom has a shape, each shape narrows the cause, and each cause has one measurement that separates it from its neighbours.
Symptom 1 — the loss will not decrease. Three sub-cases dominate. The learning rate is too high: the loss oscillates, spikes, or jumps to NaN — the update keeps overshooting the valley. The learning rate is too low: the loss moves so slowly it reads as flat; on the optimizers lesson’s quadratic (w − 3)² from w = 10, lr = 1e-4 moves the weight by 1e-4 × 14 = 0.0014 per step, so the loss goes 49.0 → 48.98. Or the gradients are not arriving: dead ReLUs (zero gradient, permanently), a detached graph, or vanishing gradients in a deep sigmoid stack.
The reference numbers from the source are worth keeping at hand:Adam starts at 1e-3, SGD at 1e-1 or 1e-2, and when a run stalls the first experiment is three learning rates spanning 10× — 1e-2, 1e-3, 1e-4 — before any architecture change. Adam adapts each parameter’s step, but it is not immune to a wrong base rate.
Symptom 2 — the loss decreases but the model is bad. Training accuracy hits 99% while test accuracy sits at 55%: the model memorized the batch. Or validation accuracy is suspiciously high — 99.7% on a task where 80% is the state of the art — which usually means leakage: shuffling before splitting, preprocessing with statistics computed on the full dataset, or duplicate rows spanning the split. Fix the order of operations: split first, preprocess second, check for duplicates. A third cause hides in the labels themselves: Northcutt et al. (2021) found 3–6% of labels in major benchmark test sets are wrong, and a model trained on them learns the noise.
Symptom 3 — NaN or Inf in the loss. Cross-entropy computes log(p); if p is exactly 0, the log explodes. Batch normalization divides by the batch standard deviation; a constant batch has std 0. Softmax exponentiates; a large logit overflows to Infinity, and Infinity divided by Infinity is NaN. Each has a standard fix: clamp probabilities to [1e-7, 1−1e-7] (the worst-case loss becomes −log 1e-7 = 16.118, not infinity), add eps to every denominator, and subtract the max logit before exponentiating.
Follow the source’s decision tree: one question at a time, from symptom to the first thing to try. Every leaf carries the number that makes the diagnosis checkable.
No answers yet — pick a symptom on the right.
Pick the symptom that looks most like your run. The questions that follow are the same ones the source's decision tree asks — in the same order.
A triage is not a diagnosis until a number says so. Every leaf here names the measurement that separates it from its neighbours — because “loss is bad” is compatible with a dozen different bugs.
The naive softmax on confident logits overflows, and the fix is one subtraction. With logits (1000, 999):
The same arithmetic explains why “loss went to NaN” is usually a three-step story rather than a mystery: weights grew, one logit crossed ≈709 (ln(1e308)), exp overflowed, and the division produced NaN. By the time NaN appears, the real bug — the oversized step — happened several iterations earlier.
| Symptom | Likely cause | First thing to try |
|---|---|---|
| Loss stuck at −log(1/C) | Uniform predictions — no signal reaching the output | Print (input, label) pairs; verify alignment and scale |
| NaN immediately | log(0) or division by zero | Clamp to 1e-7; add eps to denominators |
| NaN after a few steps | Exploding gradients / overflow | lr ÷ 10; clip the global gradient norm |
| Loss oscillates wildly | lr too high, or batch too small | Reduce lr 10×; raise the batch size |
| Loss decreases, then plateaus | lr too high for the fine-tuning phase | Add a cosine or step-decay schedule |
| Train acc high, test acc low | Overfitting | Dropout, weight decay, more data, early stop |
| Train = test = chance | Model learns nothing | Run the overfit-one-batch test |
| Train = test, both low | Underfitting | Bigger model, more features, longer training |
| Gradients all zero | Dead ReLUs or detached graph | LeakyReLU; check requires_grad and the graph |
| Loss climbs ×100 in one step | lr above the stability limit | Lower lr 10×; clip; normalize inputs |
| Out of memory | Batch too large or graph never freed | Smaller batches; torch.no_grad() for evaluation |
Before you touch the architecture, look at what the model is actually being asked to learn. Five minutes of printing inputs and labels finds a class of bugs that no amount of training fixes: misalignment, unnormalized scale, leakage and duplicates.
Visualize the inputs and labels. The source’s checklist ends with “print 5 random samples with labels”, and there is a reason it stays the first move even after years of practice: it is the only check that sees the pairing. Plot a batch’s features, color them by label, and ask whether the two clouds separate. If they do not, the model is being asked to learn noise — and it will oblige.
Normalize, and understand why the gradients care. Every gradient is linear in its inputs. Feed the network pixels in [0, 255] instead of [0, 1] and you have multiplied the gradient by 255 — a learning rate 255× larger than the one you chose. The standard MNIST statistics tell the story in three rows:
Check for leakage and duplicates. The source lists the classic mistakes: shuffling before splitting, preprocessing with statistics from the full dataset, and duplicate samples spanning the split. Each one produces the same signature — validation accuracy that looks too good — and each is fixed by the same ordering: split first, preprocess second, check for duplicates third. The lab’s validator measures the leakage directly: a scaler fitted on all 64 rows reports a mean of 0.58 where the train-only mean is 0.05 — the pooled statistic has seen the held-out distribution.
Audit the labels themselves. Northcutt et al. found 3–6% label errors in the test sets of MNIST, CIFAR-10, ImageNet and others. Your dataset is not cleaner than those. Label noise is also the one bug the overfit-one-batch test cannot see — a model with capacity memorizes wrong labels as happily as right ones — which is exactly why the held-out set, not the training loss, is the sensor for it.
Flip data bugs on and off and watch the validator from the source’s fourth exercise react. The scatter shows the training set (filled dots, colour = label) against the held-out set (hollow dots) — the shifted test cluster is visible before any check fires.
| split | x₁ | x₂ | label |
|---|---|---|---|
| train | 1.07 | 1.06 | 1 |
| train | 1.43 | -1.16 | 1 |
| train | 1.47 | 0.00 | 1 |
| train | 0.89 | -0.32 | 0 |
| train | -0.44 | 1.96 | 0 |
| train | 1.06 | 0.03 | 0 |
| test | 2.57 | -0.92 | 0 |
| test | 3.09 | 0.92 | 1 |
| test | 2.81 | 0.51 | 1 |
Try labels shuffled alone: the scatter still looks perfectly healthy, because the bugs it catches are invisible in the data’s shape. The label-alignment row is the one that fires — comparable to how a shuffled-label model still passes the overfit-one-batch test.
You have images and labels, and you shuffle everything together before splitting 80/20 into train and test. What is the most likely consequence?
Take 8–32 samples. Train on that batch alone for 200 iterations. A working model drives the loss to nearly zero — if it cannot, do not proceed to full training, because nothing downstream can be trusted.
This is the single most important technique in the lesson, and it is almost embarrassingly cheap. The reasoning: if the model, the loss and the training loop are wired correctly, then on a batch of 8 examples the loss has 8 constraints while the model has dozens of free parameters — 42 in the lesson’s 2→8→2 network — so the optimizer will happily memorize them. Failure to memorize means the problem is not the data being hard. It is plumbing, or capacity, or step size.
The lesson’s lab runs the test on a 2→8→2 network with 8 XOR points, 200 full-batch steps, and a pass line of loss < 0.1. The healthy run writes a curve that is worth memorizing as the reference shape:
Each contrast names a different failure, which is the test’s power: the way it fails points at the bug. A flat line at ln 2 says the model cannot represent the data at all (missing nonlinearity, too few units). A frozen line that never reaches chance says the step size is too small. A NaN says the step is too big. These are different fixes, separated by thirty seconds of compute.
The source lists what the test catches, and the list is worth internalizing: broken loss functions, broken backward passes, architectures too small to represent the data, optimizers not connected to the model’s parameters, and data and labels misaligned. If a tiny batch will not fit, none of the full-data training runs will either — they will just waste hours proving it at greater expense.
The same 8-sample batch, 200 steps — six ways. A working model must drive the loss below the green line by the end. One scenario passes and still should not ship: can you see which, before the readout says?
The test is not “did it learn?” — it is “can this model and this loop learn at all?”. It takes seconds on a batch of 8 and it catches broken losses, broken backward passes, disconnected optimizers and missing nonlinearities before a full training run can waste hours.
def overfit_one_batch(model, x_batch, y_batch, criterion, lr=0.01, steps=200):
optimizer = torch.optim.Adam(model.parameters(), lr=lr)
model.train()
for step in range(steps):
optimizer.zero_grad() # the order matters
output = model(x_batch) # forward
loss = criterion(output, y_batch)
loss.backward() # backward
optimizer.step() # update
if step % 50 == 0 or step == steps - 1:
preds = output.argmax(dim=1)
acc = (preds == y_batch).float().mean().item()
print(f"step {step:3d} | loss {loss.item():.6f} | acc {acc:.1%}")
if loss.item() > 0.1:
print("FAIL: loss did not converge — model or loop is broken")
return False
print(f"PASS: converged to {loss.item():.6f}")
return TrueThe loss tells you that something is wrong. The per-layer activations and gradients tell you where: which layer is saturated, which is dead, and whether the backward signal reaches the first layer at all.
PyTorch lets you attach functions to any module that run during the forward and backward passes. The source’s NetworkDebugger registers one of each per layer and keeps a running summary. The forward hook records what the layer produced; the backward hook records the gradient that flowed back through it.
for name, module in model.named_modules():
if isinstance(module, (nn.Linear, nn.Conv2d, nn.ReLU, nn.LeakyReLU)):
# forward: what did this layer output?
module.register_forward_hook(activation_hook(name))
# backward: what gradient flowed through this layer?
module.register_full_backward_hook(gradient_hook(name))
def activation_hook(name):
def hook(module, inputs, output):
out = output.detach().float()
stats[name] = {
"mean": out.mean().item(),
"std": out.std().item(),
"fraction_zero": (out == 0).float().mean().item(),
}
return hook
def gradient_hook(name):
def hook(module, grad_input, grad_output):
if grad_output[0] is not None:
grad = grad_output[0].detach().float()
stats[name] = {"abs_mean": grad.abs().mean().item(),
"max": grad.abs().max().item()}
return hookActivation statistics: what the forward pass says. The source’s diagnosis table is four rows long and catches most of what goes wrong inside a network:
| Health indicator | Mean | Std | Diagnosis |
|---|---|---|---|
| Healthy | ~0 | ~1 | Learning normally |
| Saturated | ≫0 or ≪0 | ~0 | Stuck at extreme values |
| Dead | 0 | 0 | All zeros — nothing flows |
| Exploding | ≫10 | ≫10 | Growing without bound |
Dead ReLUs, with numbers. A ReLU unit whose input is always negative outputs 0 and receives gradient 0 — the update never pushes it back. The source creates the failure deliberately by initializing weights to −1 and biases to −5. With 10 standard-normal inputs, the pre-activation is a sum of 10 terms each with standard deviation 1, so it has mean −5 and std √10 ≈ 3.16. The probability a unit sees a negative input is Φ(5/√10) = Φ(1.58) ≈ 0.943 — about 94% of units dead before training starts.
Vanishing gradients, with numbers. The sigmoid derivative is σ′(z) = σ(z)(1 − σ(z)), whose maximum is 0.25 at σ = 0.5. Ten layers in series multiply the backward signal by at most 0.25¹⁰ ≈ 9.5×10⁻⁷ — the first layer receives a millionth of the gradient the last one got, and stops learning. Tanh behaves better (~0.5¹⁰ ≈ 9.8×10⁻⁴) and ReLU better still when active (1 per unit, with about half the units silent). Residual connections sidestep the product entirely by adding the gradient identity path.
Gradient norms are a four-number language. When you print param.grad.abs().mean(), one order of magnitude already tells a story:
The healthy pattern is flat flow: every layer’s gradient within ~100× of every other’s. The source’s healthy diagram reads 0.05 / 0.04 / 0.06 / 0.05 across four layers; its vanishing diagram reads 0.0001 / 0.003 / 0.02 / 0.08 — a thousandfold taper from the loss end back to the input end. Exploding gradients are the sign flip: growth as you move backward.
Verify the backward pass itself with finite differences. Backpropagation is easy to get subtly wrong, and the audit is elementary calculus: nudge one parameter, watch the loss, and compare the slope to what autograd reported. This is the one place in the lesson where the expensive way is the trustworthy way — the numerical gradient is the referee, not the algorithm.
A simplified per-layer model: each backward step multiplies the gradient by one factor. Slide it and watch the profile go from a taper (vanishing) through flat (healthy) to growth (exploding). The histogram below is the other half of every autopsy — what the activations look like.
A single batch’s zero fraction is a crude sensor: a healthy ReLU layer legitimately shows ~50% zeros because that is what “half the units are off” means. The real check is a unit that is zero for every sample in every batch — track it across steps before declaring it dead.
The source’s checker computes the central difference for one parameter at a time and compares it to the analytic gradient with a relative metric:
Run it on L(w) = (w − 3)² at w = 4, where the true derivative is 2(w − 3) = 2:
One honest wrinkle explains a line in the source: the checker casts the model and data to double() before running. In float32, the spacing between representable values near 1 is about 1.2×10⁻⁷; dividing that quantization by 2·eps = 2e-4 gives a gradient noise floor of ≈ 6×10⁻⁴ — above the 1e-5 pass threshold, so a perfectly correct backward pass could report a mismatch. In float64 the same division gives ≈ 3×10⁻¹² of noise, and the check is meaningful. Precision is part of the experiment.
A weight's gradient norm has been 1e-7 for a thousand steps, with a learning rate of 1e-3. Roughly how large is the parameter update per step, and what does that imply?
The learning rate is the one hyperparameter whose wrongness shows up as every symptom at once. Two tools replace guesswork: a range test that finds the valley in one run, and clipping that keeps an oversized step from ending the run while you find it.
Smith’s range test (2017) is mechanical: start at a very small rate — 1e-7 — and multiply it by a constant factor every step until the loss diverges, then plot loss against the rate. The source’s sweep multiplies by (10/1e-7)^(1/100) = 10^0.08 ≈ 1.2023 per step, so a hundred steps cover eight orders of magnitude. Its printed curve is the classic shape:
The cliff has an exact explanation on a quadratic. For L(w) = ½λw², one gradient step multiplies the weight by 1 − lr·λ. The run converges when that factor is smaller than 1 in magnitude, which happens when lr < 2/λ. The loss reaches its minimum fastest at lr = 1/λ — the valley floor — and the recommendation of one order of magnitude below it is a bet against noise: real surfaces are not perfect quadratics, and the margin buys stability.
The lab’s numbers for λ = 10: floor at 0.1 (one perfect step), limit at 0.2, and at lr = 0.25 the factor is |1 − 2.5| = 1.5, so the loss multiplies by 1.5² = 2.25 per step — after 25 steps this is 45 × 1.5⁵⁰ ≈ 2.9×10¹⁰ (the lab grid’s next point, 0.2512, reads 4.3×10¹⁰). Geometric growth is why diverging runs die so fast: nothing is gradual past the cliff.
Gradient clipping is the seatbelt. Before the optimizer steps, measure the global gradient norm and scale the whole gradient down if it exceeds a threshold:
Clipping does not care why the gradient was large, which is both its virtue and its limit: it bounds the damage of an oversized step, but the cause — a wrong lr, an unnormalized input scale, one pathological batch — is still there. Normalization is the cause-side fix. In the lesson’s NaN simulator, raw inputs at 30× the unit scale produce a step-0 loss of 12.95 and NaN at step 1; turning on clipping at norm 1.0 keeps the same run alive to a final loss of 1.1×10⁻⁵, and normalizing the inputs instead reaches 0.0091 with no guardrail at all. Do both while you investigate: clip for safety, normalize for the fix.
Sweep lr across four decades and plot the loss each rate reaches. The valley has a floor; the cliff is the stability limit. This is a closed-form one-parameter quadratic — the simplest model that shows the real valley-then-cliff shape (the source’s network sweep goes 2.3 → 0.9 → NaN for the same reason).
Honest caveat: the quadratic is a simplified teaching model — real network curves are noisier and can be non-monotone. But the shape they share is the finding: descend into a valley roughly one decade wide, then a cliff. Pick the rate at the valley’s steepest point and use a tenth of it.
One tiny run, three dials: input scale, learning rate, and whether clipping or normalization is protecting the update. Watch when the curve dies — and which guardrail brings it back.
The demo misbehaving is not exotic: raw images, unmapped features, or an extra ×255 are the usual causes. Note what each fix does — clipping bounds the step but leaves the scale wrong; normalization removes the cause. Practically, do both: the clip is insurance while you find the scale bug.
When a run misbehaves, the goal is not to inspect everything — it is to make the failure smaller. A bug that reproduces in six lines with one batch and a fixed seed is a bug you can fix; the same bug inside a 40 million-parameter pipeline is a week of guessing.
The source’s checklist is the simplification procedure written out. Run it in order, and stop at the first failure — every later observation is contaminated by it:
Two habits make the checklist fast. First, start from a known-good configuration and add one piece at a time — a smaller model, a simpler dataset, a shorter sequence — so the first failure you meet is the first bug you introduced. Second, log every number you might argue about: loss, learning rate, gradient norm, activation statistics, weight norm. Debugging is a reading exercise, and the console line you did not print is the one you will wish you had.
Seeds and determinism. Set every seed — Python, NumPy, PyTorch, the DataLoader’s shuffle — and your runs become reproducible. That is invaluable for bisection: a bug you can replay is a bug you can corner. But be precise about what it buys: a seed makes a wrong run reproducibly wrong. It does not make the result correct, it does not fix a race condition on the GPU, and it does not excuse you from checking the data.
Automate the classification. The source’s debugger sorts a loss history into one of four states —NAN_OR_INF, NOT_DECREASING, OSCILLATING, HEALTHY — by comparing the first ten losses to the last ten and measuring the spread of recent differences. A few lines of such logic, run as an assertion after every epoch, turn “something feels off” into an alert with a name.
Carry the same sensors into production. The deployment pattern is the checklist made continuous: monitoring hooks on the training script, activation and gradient statistics logged to TensorBoard or Weights & Biases every N steps, and alerts on the three loud signatures — a NaN loss, a layer that has gone completely zero, a gradient norm that leaves its usual order of magnitude. Re-run the overfit-one-batch test whenever the architecture or the data pipeline changes; it is the cheapest regression test in the stack.
A training iteration is six lines. Five of the six possible orders fail silently or crash with a confusing error. Rebuild the order, then read the three bugs that survive review most often.
awaiting…awaiting…awaiting…awaiting…awaiting…awaiting…symptom · gradients accumulate across iterations
Each backward() adds into the same .grad buffers, so step k uses g₁ + g₂ + … + gₖ. If each batch's gradient has a consistent part g and a batch-to-batch disagreement δ, the accumulated update grows ~k·g while the disagreement only grows ~√k — the update is systematically k times too large. In the source's demo (SGD lr 0.01, 50 steps) the symptom is a loss that oscillates instead of settling. In a full-batch, well-conditioned toy, accumulation can masquerade as momentum and look harmless; the bug turns damaging exactly when batches disagree, which is the normal case.
The order is not style: each line depends on the last. Bugs that reorder or skip a line produce numbers that look plausible — which is why the checklist exists at all.
| Bug | Symptom | Fix |
|---|---|---|
Forgetting optimizer.zero_grad() | Gradients accumulate; loss oscillates | Zero before loss.backward() |
Forgetting model.eval() | Dropout and batch norm behave differently; test numbers vary | model.eval() + torch.no_grad() |
| Wrong tensor shapes | Silent broadcasting produces wrong results | Print shapes after every operation while debugging |
| CPU/GPU mismatch | RuntimeError: expected CUDA tensor | .to(device) on model and data |
| Not detaching tensors | Graph grows forever; out of memory | .detach() or with torch.no_grad() |
| In-place ops breaking autograd | RuntimeError: modified by in-place operation | Replace x += 1 with x = x + 1 |
| Data not normalized | Loss stuck at the random-chance level | Normalize inputs to mean 0, std 1 |
| Labels as wrong dtype | Cross-entropy expects Long, got Float | Cast: labels.long() |
Loss oscillates wildly around 2.0 and occasionally spikes to 20. Training loss is not improving across epochs. What is the best first move?
Answer before you look. The overfit-one-batch question and the flat-loss question are the two that separate a checklist you have memorized from a mechanism you can reason with at 2 a.m.
0 / 6 answered · 0 correct
01Why is neural network debugging harder than traditional software debugging?
02What is the 'overfit one batch' debugging technique?
03Your loss is NaN after a few training steps. What is the most likely cause?
04Your training loss decreases but validation loss stays flat from the start. What does this indicate?
05What should you check first when your loss curve is completely flat (loss doesn't decrease at all)?
06After loss.backward(), you print the gradients and every parameter's gradient is exactly 0. What are the two most likely explanations?
Click a card to swap the lazy description for what it actually means.
Three problems with exact numbers — build an exploding-gradient detector and choose a clipping value, resurrect dead ReLU units with Kaiming initialization, and write the data pipeline validator. Try first; a worked answer is one click away.
Keep a rolling history of the global gradient norm per step, then flag an explosion when the current norm exceeds 10× the median of the last 100 norms (a robust baseline — a mean is itself dragged up by the outlier). The suggested clip value is 10× the median: healthy enough to leave normal steps untouched, tight enough to bound the outlier. Worked example: a 20-layer ReLU stack with per-layer gain 1.5 grows the backward signal 1.5²⁰ ≈ 3,325×; if the median norm is 0.4 and the current norm spikes to 2,700, the detector says 'clip at 4.0'. Verify clipping preserves direction: ‖[3, 4, −12]‖ = 13, clipped to 4 it becomes [3, 4, −12] × (4/13) = [0.923, 1.231, −3.692]. Two honest caveats. First, clipping is a seatbelt: it keeps the run alive but the cause (unnormalized inputs, an lr above 2/λ, a deep unnormalized stack) is still there — log the norm, fix the scale. Second, the right threshold is per-network: on the source's deliberately broken lr = 10 demo the norm crosses every plausible threshold, while a healthy run's median is the definition of normal. Report both the measured norms and the flag, so the log shows the trend, not just the alarm.
Identification is the part that has to be right: a unit is dead only if its pre-activation is ≤ 0 for every sample in every batch. Track a per-unit counter across ~10 batches and call it dead at 10/10. Then reinitialize that unit's incoming weight row with Kaiming normal (std = √(2/fan_in)) and zero (or slightly positive) bias. Numbers on the source's broken init, which uses weights −1 and biases −5 with 10 standard-normal inputs: pre-activation ~ N(−5, √10), so P(z < 0) = Φ(5/√10) = Φ(1.58) ≈ 0.943 — 94% dead. After Kaiming reinit with std √(2/10) = 0.447 and zero bias, each weight times input contributes variance 0.2, so the pre-activation is N(0, √2 ≈ 1.41) and P(dead) = Φ(0) = 0.5; a +0.2 bias shifts it to Φ(−0.14) ≈ 0.44, i.e. 44% dead instead of 94%. The resurrection test: run the overfit-one-batch sanity check before and after. A network that cannot fit 8 samples with 94% dead units will fit them once the units are alive — if it does not, the unit identification or the fan_in (number of incoming connections) is wrong. Watch for the trap the lesson names: one batch's zero fraction is not deadness — a healthy ReLU layer is ~50% zeros by design, so a per-batch reading of '50% dead' is noise, not a diagnosis.
The lesson's data-doctor lab is exactly this validator, and its output on the clean 64-row dataset versus each corruption is the worked answer. Duplicates: compare feature rows for exact/near-exact equality across splits — 8 copied test rows are found and flagged 'the model has already seen those examples'. Imbalance: count positives and negatives; 3 positives to 45 negatives is 15:1, over the source's 10:1 threshold (a warn zone starts at 3:1). Normalization: report per-feature mean and std; raw pixels at ×80 read std 63.6 and 79.7, the standard failure. NaNs: count non-finite entries — 5 flag as failures, and they propagate into every layer. Label alignment: compute the standardized class gap on each feature (difference of class means divided by pooled std); the clean data separates at 1.28σ on x₁, while shuffled labels collapse to −0.32σ, below the 0.5σ threshold. Leakage: compare statistics fitted on the train split against statistics fitted on the pooled data; with a held-out cluster shifted +2.5 in x₁, the pooled mean 0.58 versus the train-only mean 0.05 is the leak, and it is measured directly rather than suspected. Order matters: run the validator before the first training run, because the overfit-one-batch test cannot see bad labels or a 30× scale — it will happily memorize corruption.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 03, Lesson 13) and the Math Foundations Notebook reference build. The eight labs (the four-failure loss gallery, the triage board, the data doctor, the overfit-one-batch test with six injected bugs, the gradient and activation inspector, the NaN simulator with clipping and normalization switches, the quadratic learning-rate range test, and the find-the-bug ordering board) are original to this page, as are the chance-level arithmetic (ln C, the 0.9/0.1 softmax check, the 0.056 base-rate fraud model), the log-sum-exp worked check, the MNIST scale arithmetic, the live overfit-one-batch trace and its four failing variants, the shuffled-label honesty clause, the dead-ReLU Φ(5/√10) ≈ 94% arithmetic, the gradient-norm language and the float32 gradient-check noise floor, the range test's 10^0.08 multiplier with the exact quadratic stability limits 1/λ and 2/λ, and the memory hook. Every number shown is computed live by the labs or verified by hand in the prose.