The worst bugs don’t crash.
They train beautifully.

That asymmetry is the whole problem. A web app fails in milliseconds and hands you a stack trace with a line number. A training run fails for eight hours and hands you a log that says everything completed. The process exits with status 0, the loss curve descends, and the model has learned nothing. The source’s arithmetic is worth stating plainly: 8 hours × $25/hour = $200, so the implied rental rate behind the example is about $25 per hour — the price of a small multi-GPU node rather than one consumer card, and the reason silent failures are measured in money.

AI debugging happens at three levels, and the source’s diagram is the map of this lesson. Level 1, standard Python: breakpoints, logging, profiling, memory. Level 2, tensor operations: shapes, dtypes, devices, NaN (not a number) and Inf values. Level 3, training dynamics: loss curves, gradient norms, activations. The levels are not a difficulty ranking — they are a search order.

Here is the claim that reorganizes your instincts: most people jump straight to level 3 — staring at TensorBoard — but 80% of AI bugs live at levels 1 and 2. The evidence is in the instruments each level needs. A shape mismatch, a float64 tensor where float32 was intended, a batch left on the CPU (central processing unit), a NaN in a column with zero variance — each one is visible with a single print, before the run even finishes. A loss curve, by contrast, only tells you that something is wrong, and usually only after hours of compute have already been spent. The curves are the most expensive place to look and the least specific: they are a smoke alarm, not a floor plan.

So the workflow this lesson builds is a search that starts cheap. Ask the tensor four questions — shape, dtype, device, NaN — and the run two more: where did the time go, and where did the memory go. When the loss curve is the mystery itself — parked at the chance level, or refusing to move at any learning rate — the diagnosis becomes neural-network-specific, and that is a later lesson: Phase 3, Lesson 13 (Debugging Neural Networks) covers chance levels (ln C), the overfit-one-batch test, gradient checking and the learning-rate range test. This lesson is the general toolbox underneath all of them.

The source’s first tool is a function that never changes: debug_print(name, tensor). It prints seven facts about a tensor — shape, dtype, device, min, max, mean and whether it contains a NaN (not a number) — and it is called after every suspicious operation. When the bug is found, you remove the prints. Simple. The reason a print wins here is bandwidth: stepping through a debugger gives you one value at a time, while a tensor bug is usually a combination of shape, dtype and range that only makes sense when you see them together.

Each field answers a specific question, and it is worth knowing which bug each one catches before you need it:

The second tool is the debugger (pdb, the Python Debugger), and the interesting part is the condition. A 10,000-step run is too long to step through, so the source puts breakpoint() behind the symptom: pause only when the loss is absurd or already NaN. That turns the debugger from a microscope into a trap — it catches the run in the act, at the exact step where the evidence still exists.

The source’s argument for logging is one sentence of realism: when a training run fails at 3 AM, you want a log file, not terminal output that scrolled off screen. Overnight, nobody is looking. The SSH (Secure Shell) session that hosted the run may be gone by morning; the terminal window on your laptop may have been closed; the buffer that held 40,000 step lines is not evidence anywhere. Meanwhile the rented GPU billed you for every minute of the eight hours — which makes the log file the run’s black box.

Three ideas are doing the work in those ten lines. Levels rank severity: DEBUG < INFO < WARNING < ERROR < CRITICAL. Setting level=logging.INFO means a record is emitted if its severity is at or above INFO — so DEBUG records vanish without you deleting anything, and you can raise the floor to WARNING for a quiet run or lower it to DEBUG for a three-minute investigation. Handlers are destinations: the file handler writes training.log; the stream handler writes to the terminal. Two handlers, one stream of records — which is why you can keep the full history on disk while the terminal shows only warnings. Timestamps turn the log into a timeline: the moment the loss spiked, the checkpoint that was never written, the exact minute the run stopped. And one habit to copy from the source: the arguments are passed separately ("...%.4f", value), not f-string-formatted, so the string is only built if the record is actually emitted — in a hot loop that is real work saved.

Logging also settles the timing question, because the thing that follows from “where did it fail” is “where did the time go”. The source’s Timer is a context manager — a Python object usable in a with block that does work on entry and exit — built on time.perf_counter(), the highest-resolution clock Python exposes.

Now the finding that pays for this whole lesson: data loading often takes 60% of training time, and the fix is num_workers > 0 in your DataLoader — not a faster GPU. Here is that claim worked with arithmetic (a teaching model built on the source’s 60%, not a benchmark of your machine). One hundred steps of a small model, with num_workers=0, take 30.0 seconds:

Read the last line carefully: nothing about the model changed. The forward and backward times are identical in both columns — num_workers cannot make a matrix multiply faster. What changed is that the GPU stopped sitting idle waiting for the next batch, and idle hardware is the most expensive thing in the room.

Python ships a profiler in the standard library. Run your script through it and sort the report by cumulative time — the time a function and everything it calls consumed:

Two habits make profiler output readable. First, sort and compare: run once with -s cumtime to find the expensive call trees, then look at the tottime column to see which frames are spending time in their own code rather than inheriting it from children. A function that calls everything accumulates everyone else’s seconds and will always sit near the top. Second, remember the overhead: cProfile instruments every call, so a profiled run is slower than the real one and the absolute numbers shift. The ratios are the finding; the seconds are a hint. When a ratio points at a frame, confirm it with a Timer on a clean run.

When the report points at a function but you need the line, the standard-library profiler stops being enough — it works at function granularity. That is the job of line_profiler:

On the CPU (central processing unit) side, Python’s standard library carries a second profiler: tracemalloc traces every allocation and can attribute the bytes to the line that allocated them. The recipe is four steps — start tracing, run the suspicious code, take a snapshot, print the top statistics.

For line-by-line memory rather than allocation attribution, the third-party memory_profiler decorates a function the same way line_profiler does, and reports MiB per line:

The GPU (graphics processing unit) has its own ledger, and PyTorch exposes it directly. torch.cuda.memory_summary() prints a block-by-block account; the two numbers worth memorizing are memory_allocated() — bytes held by live tensors — and memory_reserved() — bytes held by PyTorch’s caching allocator.

That gap explains both why OOM (out of memory) happens and why empty_cache() is not magic: it returns unused cached blocks to the driver so the process can avoid one more allocation failure — it does not free a single live tensor. With the two ledgers in hand, the source’s OOM checklist becomes a sequence to work down, in order:

One more number makes the OOM conversation concrete. The optimizer state is not free: with Adam, every parameter carries a gradient plus two moment estimates, all in FP32 — 4 + 4 + 4 + 4 = 16 bytes per parameter before a single activation. A 125-million-parameter model therefore needs 125,000,000 × 16 = 2.0 GB just for weights, gradients and optimizer state; everything the forward pass computes sits on top of that. (Scale that model up by 56× to a 7-billion-parameter model and the same arithmetic gives 112 GB — which is why the largest models need sharding and offloading, tools that arrive in later phases.)

The order matters because the checklist is sorted by cost: a config change, then a cache call, then a reference deletion, then a numerical technique, then a compute-for-memory trade. Reaching for gradient checkpointing first is like emptying a house to make room for one suitcase.

Shape mismatch is the most frequent bug in deep learning. A tensor has shape [batch, features] when the model expects [batch, channels, height, width]; a flattened MNIST image is 28 × 28 = 784 numbers, but an ImageNet-shaped image is 3 × 224 × 224 = 150,528 — 192 times wider than the layer that receives it. The error message names two tensors and no intent, so the source’s catcher runs a sample batch through forward hooks — callbacks each layer calls with its input and output — and prints every shape transformation at once.

NaN loss means something exploded — and the source lists the usual suspects: a learning rate too high, a division by zero in a custom loss, a logarithm of zero or a negative number, exploding gradients in recurrent networks. The catcher walks every named parameter and reports which gradient went NaN or Inf, because the first bad gradient is usually one layer below the arithmetic that produced it.

Data leakage is the bug that flatters you. Your model reports 99% accuracy on the test set, the source says — and it is a bug: test samples leaked into training, or a feature contains the answer. The catcher intersects the identifiers of the two splits; anything it finds disqualifies the result. The second leak is temporal: using future data to predict the past — sort by timestamp before splitting, and check that every feature was knowable at prediction time.

Wrong device is the quiet one. Tensors on different devices usually raise a RuntimeError the moment an operation mixes them — but sometimes a tensor silently stays on the CPU while everything else is on the GPU, and training simply runs slowly. The source’s catcher compares every candidate tensor with the model’s device, and the honest detail is that it must be called before the run: once the mismatch is inside an operation the error names the op, not the line that forgot to move.

One boundary is worth drawing explicitly, because it saves a lot of duplicated effort. Everything in this chapter is evidence collection: shapes, gradients, devices, overlap. When the loss curve itself is the mystery — parked at the chance level, busy memorizing one batch, or unresponsive at every learning rate — the diagnosis becomes neural-network-specific. That is Phase 3, Lesson 13 (Debugging Neural Networks): the chance-level arithmetic ln C (0.693 for two classes, 2.303 for ten), the overfit-one-batch test, gradient checking against a numerical gradient, the learning-rate range test, and gradient clipping. This lesson gets the numbers on the table; that lesson reads the neural-network half of them.

The writer loop is small. Create a SummaryWriter pointed at a run directory, add the scalars you watch every step, add histograms of weights and gradients every hundred steps (writing a histogram is far more expensive than writing a scalar), and close the writer when the run ends.

The source gives six patterns and their meanings. Read them as hypotheses, not verdicts — each one narrows the search, and each one has an instrument in this lesson that can confirm it:

Then there is the debugger you already own. Visual Studio Code (VS Code) can attach to your training script with a launch.json — a JavaScript Object Notation (JSON) file in .vscode/ describing how to start a debug session — using the debugpy adapter. Click the gutter to set a breakpoint, use the Variables pane to inspect tensor properties, and use the Debug Console to evaluate expressions mid-run. The justMyCode: false setting is the one to understand: it lets the debugger step into library code, so a breakpoint can land inside PyTorch itself, not only inside your file.

Everything in this lesson now composes into the source’s five-step workflow — the sequence to follow for every training run, from the quiet before it starts to the performance question after it works:

And when the curves are read and the evidence is in — shapes, dtypes, devices, gradients, timing, memory — the diagnosis for a neural-network-specific failure follows the playbook in Phase 3, Lesson 13 (Debugging Neural Networks): compare the loss against the chance level ln C, overfit one tiny batch to prove the model can learn at all, check gradients numerically, and sweep the learning rate. This lesson made the measurements; that one turns them into a diagnosis.