A wrong answer is a debt spread across millions of weights. Backpropagation pays it back in a single reverse sweep — carrying the chain rule backward through the graph, reusing every value the forward pass stored. This is the algorithm that makes learning possible.
A 768×3072 hidden layer owns 2,359,296 weights, and the naive gradient needs one forward pass per weight. Backprop stores every intermediate value during the forward pass, then walks the graph once in reverse — every parameter's gradient, one sweep, no repeated forward passes.
all gradients ≈ parameter count ÷ 3 passesAddition passes the incoming gradient through; multiplication hands each input the other factor; sigmoid multiplies by a(1 − a) using the output the forward pass already stored. Derivatives multiply along a path, and contributions from different paths add where they meet. Everything else is bookkeeping.
incoming × local, then sum the pathsEach layer multiplies the backward signal by its local derivative. Sigmoid's derivative is at most 0.25, so ten layers pass a millionth of the gradient to the first layer; weight scales above 1 make the product explode instead. Gradient checking — nudging one weight and measuring the loss — is the independent referee that proves the backward pass is correct.
0.25¹⁰ ≈ 10⁻⁶ · trust, but verifyBackpropagation is the chain rule wearing a bookkeeping costume: the forward pass records what every operation produced, and the backward pass walks the same graph in reverse, multiplying each incoming gradient by one local derivative and adding the contributions where paths meet.
By the end you will be able to run a backward pass by hand through addition, multiplication and sigmoid; accumulate gradients across shared paths with +=; derive and verify the gradients of a full 2-2-1 network against finite differences; train XOR from scratch with a four-line loop; and explain why sigmoid depth multiplies the signal by at most 0.25 per layer — the bridge to activation functions and initialization.
A hidden layer with 768 inputs and 3072 outputs holds 2,359,296 weights. When the network is wrong, every one of them shares the blame — and assigning that blame is the whole problem. The answer is not a cleverer formula; it is a bookkeeping device called the computational graph.
Your network has a single hidden layer with 768 inputs and 3072 outputs: 2,359,296 weights. It made a wrong prediction. Which weights caused the error? The naive experiment is an experiment per weight: nudge one weight by a tiny amount, run the forward pass again, and watch whether the loss went up or down. That number is the weight’s gradient. Now repeat it for every weight — and every training step, for every example.
Backpropagation computes all 2,359,296 gradients in a single backward pass. That is not an optimisation of the nudge — it is a different operation. The naive route pays one full forward pass per parameter; the backward pass pays once for the whole network and reuses everything it saw on the way forward.
The naive way to get one gradient: nudge one weight, re-run the whole forward pass, measure the loss change. Repeat for every weight. Count the passes — then count what backpropagation needs.
Bar width is logarithmic — a linear bar for backpropagation would be invisible next to 2.36×10^12 naive passes. One epoch of the naive route is 786,432× more work; the backward pass buys every gradient at once because it reuses the values the forward pass already stored.
Simplified counting model: backward ≈ 2× the cost of a forward pass, so backprop is scored as 3 forward-equivalents per example. Real frameworks batch examples and reuse hardware parallelism; the ratio between the two rows is what survives.
The source’s own example: a 768 → 3072 layer holds 2,359,296 weights. Nudging each one means 2.3 million forward passes to score a single step; backprop computes all 2.3 million gradients in one sweep. That difference is why deep learning is trainable at all.
The device that makes this possible is the computational graph. Every forward pass silently builds one: nodes are operations (multiply, add, sigmoid, square), and edges carry values forward. In the reverse direction the same edges carry gradients. Each operation knows exactly one extra thing — its own local derivative, the answer to “if I nudge my input by a little, how much does my output move?”
The forward pass runs left to right and stores every intermediate value it computes. The backward pass runs right to left and gives every node one job: take the gradient arriving from above, multiply by the local derivative, and hand the result to the inputs that produced it. Sigmoid needs its own output to compute a(1 − a); multiplication needs the other factor. The backward pass never recomputes them — it reuses them.
One sigmoid neuron and a squared error. Step forward and watch values fill in; step backward and watch each node hand a gradient to the inputs it came from. Same graph, opposite directions.
The forward pass stores every value it computes; the backward pass never recomputes them — it reads them. That is the whole memory-for-speed trade at the heart of backprop.
A teammate proposes estimating gradients by nudging each weight once per step. Why does backpropagation win?
You met the chain rule in Phase 1, Lesson 05. Here it stops being a calculus exercise and becomes an algorithm: every node carries one local derivative, and the backward pass multiplies them together as it walks the graph.
The chain rule says: if y = f(g(x)), then dy/dx = f′(g(x)) · g′(x). In plain English — to find how a nudge in x moves y, multiply the rates of every stage in between. If a 1% change in x causes a 3% change in g, and a 1% change in g causes a −2% change in y, then a 1% change in x causes roughly a −6% change in y. Rates multiply along a chain; that is the entire idea.
A neural network is one long chain: weights multiply inputs, biases add, activations squash, the loss compares. Backpropagation just runs the chain rule from the end of the chain backwards. Three local derivatives cover almost everything in this lesson:
These are the exact rules the source’s Value class implements: _backward for + adds the incoming gradient to both inputs, _backward for * multiplies it by the other factor, and _backward for sigmoid multiplies it by s(1 − s) — where s is the output the forward pass already stored.
Backprop is not one big derivative — it is a product of small local ones. Walk the chain from the loss to a weight, then ask the network numerically to confirm the product.
Along one path, derivatives multiply. That is why a sigmoid’s 0.25-maximum local derivative is such bad news in a deep stack — the product shrinks with every layer, the subject of chapter 07.
Fixed example used by the labs and the hero: w = 1.5, x = 0.5, b = 0.2, target y = 1.0, squared error L = (a − y)². Simplified teaching neuron — one multiply, one add, one sigmoid, one square.
Read the middle block as a sentence: to move L, the sigmoid stage converts a loss-side rate of −0.55777 into a z-side rate of −0.11217, and the multiply stage converts a z-side rate into a w-side rate of −0.05609 by multiplying through the other factor x = 0.5. The numeric column is the network itself answering the same question with no calculus at all — the two agree to the sixth decimal, which is the subject of chapter 05.
The backward pass is a single walk through a topologically sorted graph: start at the loss with gradient 1, and let every node hand its inputs the product of incoming gradient and local derivative. A node that fed several operations collects the sum.
The source’s backward() does three things. It builds a topological order of the graph — a list where every node appears after all the nodes it depends on, built with a depth-first walk. It seeds the loss with grad = 1.0, because dL/dL is exactly 1: the loss is 100% sensitive to itself. Then it walks the list in reverse, calling each node’s backward rule once. By the time a node is processed, every path that flows through it has already delivered its gradient — which is the entire reason the order matters.
Why can’t you process nodes in any order? Because a node’s gradient is not a single number from a single parent — it is the sum of the gradients arriving from every operation that used it. If a shared node runs its backward rule early, it propagates a partial gradient onward and the rest of the graph quietly computes wrong answers. Topological order guarantees the node’s bucket is complete before it is spent.
That bucket is why the forward source code writes self.grad += other.data * out.grad and self.grad += out.grad — never =. A value used in two operations receives two contributions, and += is how the graph adds across paths while multiplying along them.
Here x feeds two branches that rejoin. Walk the backward pass and watch x receive two separate gradient contributions. Then flip the accumulator to a plain assignment and watch half the gradient disappear.
A value used in several operations needs the sum of every path’s gradient. This is exactly why the engine writes self.grad += … and why every framework zeroes gradients before the next backward pass.
In that example (w1 = 2, w2 = 3, x = 1), the backward pass reaches x twice with contributions 20 and 30. What should x.grad be, and why?
The source derives the backward pass for a full two-layer network: two inputs, two sigmoid hidden units, one sigmoid output, squared error. Same rules as the single neuron — apply them layer by layer, matrices and all.
Forward pass. A layer is a weighted sum followed by an activation. The hidden layer turns the two inputs into two pre-activations z1, squashes them into activations a1, and the output layer does the same once more. Squared error compares the final activation with the target:
Backward pass. Apply the chain rule from the loss backwards. Each line below is one link: incoming gradient × local derivative. The only new move is the transpose-like fan inversion — when the forward pass computes W2·a1, the backward pass sends the output gradient back through W2 to reach a1.
Simplified teaching network, fixed by hand so every number is checkable: W1 = [[0.5, −0.25], [0.25, 0.5]], b1 = [0.1, −0.1], W2 = [0.6, −0.4], b2 = 0.2, input x = [1.0, 0.5], target y = 1.0.
Follow one number end to end: the output unit’s error rate −0.203458 becomes each output weight’s gradient by multiplying by the hidden activation that fed it — a bigger activation fed more of the error, so it earns a bigger gradient. Then the same error rate rides back through the output weights to reach the hidden units, gets multiplied by each hidden sigmoid’s local derivative, and finally becomes each input weight’s gradient by multiplying by the input value. Nothing new was invented; the chain just got longer.
| gradient | formula (incoming × local) | value |
|---|---|---|
dL/dW2[0] | dL/dz2 × a1[0] | −0.125446 |
dL/dW2[1] | dL/dz2 × a1[1] | −0.121808 |
dL/db2 | dL/dz2 | −0.203458 |
dL/dW1[0][0] | dL/dz1[0] × x[0] | −0.028860 |
dL/dW1[0][1] | dL/dz1[0] × x[1] | −0.014430 |
dL/dW1[1][0] | dL/dz1[1] × x[0] | +0.019553 |
dL/dW1[1][1] | dL/dz1[1] × x[1] | +0.009777 |
dL/db1[0] | dL/dz1[0] | −0.028860 |
dL/db1[1] | dL/dz1[1] | +0.019553 |
A sign flipped, a path forgotten, a derivative copied from the wrong function — broken backprop does not crash. It trains a network that gets slightly worse forever. Finite differences are the independent referee: nudge a weight, measure the loss, and compare.
Backpropagation is arithmetic, and arithmetic written by humans is arithmetic that can be subtly wrong. The dangerous bugs are not exceptions — they are mistakes that produce plausible-looking numbers. A missing +=, a sigmoid derivative written as a(1 − a) when a is the pre-activation, a gradient sent through W instead of Wᵀ: the network still trains, just worse. You need a second opinion that does not share code with the first.
That second opinion comes from the definition of a derivative. You do not need calculus to estimate one — you can measure it. Nudge a single weight up by h, run the forward pass, and record the loss. Nudge it down by h and record the loss again. The slope between those two points is the central finite difference:
The estimate is independent of the backward pass: it touches only the forward pass and the loss. If backprop says −0.125446 and the two-point slope says −0.125446, the chain rule was implemented correctly for that parameter. Repeat for every parameter and you have verified the whole gradient, one weight at a time.
Use the same teaching network as worked example B: the one whose output-layer weight W2[0] = 0.6 received backprop gradient dL/dW2[0] = −0.125446. Perturb it by h = 10⁻⁵ in both directions and compute the loss at each point with the forward pass alone.
Read the last two lines together — they are the whole practical lesson. Make h too large and the two points sit far apart on a curve, so the secant slope misses the tangent (truncation error). Make h too small and L(w + h) and L(w − h) agree in the first dozen digits, so their difference is mostly floating-point noise (roundoff error). Somewhere around h = 10⁻⁵ the two errors balance and the check agrees with backprop to twelve digits.
Perturb one weight by ±h, measure how the loss changes, and compare that cheap numerical estimate with the script’s backprop value. Then slide h and watch the error trade truncation against floating point.
The best h here is around 10⁻⁵: small enough that the slope is accurate, large enough that subtracting two nearly equal losses keeps its digits. The U shape is the whole lesson of this chapter — and PyTorch ships the same test as torch.autograd.gradcheck.
def numeric_grad(f, x, h=1e-5):
return (f(x + h) - f(x - h)) / (2 * h)
# analytic gradient from the backward pass
loss = f(x)
loss.backward()
analytic = x.grad.item()
estimate = numeric_grad(f, x)
assert abs(analytic - estimate) < 1e-6 # a one-time audit, not a training stepA teammate sets h = 10⁻¹⁴ for the gradient check to be 'extra precise'. What do they actually get?
Everything so far has been building blocks. The training loop is the assembly: clear the old gradients, run the forward pass, run the backward pass, nudge every parameter downhill. Repeat until the loss stops arguing.
Training a network is a loop with exactly four moves. Zero the gradients so yesterday’s accumulation does not leak into today. Forward the batch and compute the loss. Backward through the graph to fill every parameter’s gradient. Update each parameter against its gradient. Nothing else is required — no hand-derived formulas per layer, no per-parameter bookkeeping. Backprop filled the gradients; the optimiser just spends them.
The source trains a Network([2, 4, 1]) on the four XOR rows with this loop. The batch is all four examples: their individual losses are summed into total_loss, and one backward pass on that sum hands every weight the gradient of the total error. Then the parameters step. That is gradient descent, with the gradients computed by the code you built in the previous chapters.
random.seed(42)
net = Network([2, 4, 1])
for epoch in range(1000):
total_loss = Value(0.0)
for inputs, target in xor_data:
x = [Value(i) for i in inputs]
pred = net(x)
loss = mse_loss(pred, target)
total_loss = total_loss + loss
net.zero_grad()
total_loss.backward()
for p in net.parameters():
p.data -= learning_rate * p.gradNo library. Every gradient in this training loop comes from the Value engine the lesson builds: forward, loss, backward, update. Watch the loss fall and the decision field bend into the XOR shape.
XOR is not linearly separable — a single neuron cannot do it, which is why there is a hidden layer. The network discovers the hidden layer’s own representation using nothing but the gradients backprop produced.
Same architecture and seed as the lab (2-4-1, seed 42, learning rate 1.0). The numbers below are the actual loss and the four predictions at checkpoints on the way down.
Two things are worth noticing. First, the first ~150 epochs are a plateau: the randomly initialised network answers almost 0.5 on every example (total loss ≈ 1.03), and at that symmetric point the four error signals largely cancel, so the loss crawls. Around epoch 200 the hidden units break symmetry — two outputs climb, two fall — and the loss collapses from 0.89 to 0.12 in a hundred epochs. Second, by epoch 300 the four outputs have split into two high and two low: that is the network inventing its own hidden representation. Nobody labelled the hidden units; backprop found them.
The loop sums the losses of all four XOR examples before calling backward(). Why not call backward() four times, once per example?
Backprop multiplies one local derivative per layer. That is a feature — until the factors are all 0.25 and the product is a millionth by layer ten. Depth turns multiplication into exponential decay or exponential blow-up, and the whole modern toolkit exists to keep the product near 1.
In chapter 06 the network was two layers deep, and the gradients arrived at the first layer only slightly smaller than they left the last. Stack twenty sigmoid layers and that changes completely. Each layer multiplies the backward signal by its own local derivative, and the sigmoid’s derivative, a(1 − a), is never larger than 0.25 — it peaks at 0.25 when the unit sits at its midpoint and shrinks toward zero as the unit saturates. A chain of n sigmoid layers therefore multiplies the gradient by at most 0.25ⁿ. At depth 10 that is about 10⁻⁶: the first layers receive a millionth of the learning signal, so they barely move while the last layers look like they are training fine.
The failure runs in both directions. If each layer’s effective factor is even slightly above 1 — because the weights are large, or the activation’s derivative is above 1 over the operating range — the product grows exponentially and the early gradients explode, producing wild updates and NaNs. The same chain rule that makes backprop work makes it fragile: depth multiplies.
A chain is only as strong as its product of local derivatives. Stack sigmoids and the product dies; stack layers whose weight scale exceeds 1 and it explodes. Same chain rule, opposite failure.
The model is deliberately pessimistic — every layer is assumed to sit at its maximum derivative. Real networks average lower still, so the sigmoid’s diagonal is the best case it can hope for, not the worst.
Take the simplified worst-case model from the lab: every layer sits at its maximum local derivative and at a chosen weight scale s, so the factor per layer is s × max f′. The gradient arriving at layer n is the factor raised to n.
The table is deliberately pessimistic — real factors vary per unit and per sample, and healthy units sit where their derivative is decent — so use it to order the failure modes, not to predict exact numbers. The ordering it gives is the historically accurate one: sigmoid networks simply could not be made deep, ReLU plus careful initialization (next lesson) made depth trainable, and residual connections made very deep networks boring.
Your 15-layer sigmoid network's loss has stalled, and you notice the first three layers' gradients are ~10⁻⁸ while the last layers move normally. Which change is most likely to help?
Answer before you look. The += question and the topological-order question are the two that expose whether the backward pass is a mechanism you can debug or a magic trick you memorised.
0 / 5 answered · 0 correct
01What does the chain rule say, in the context of neural networks?
02Why is backpropagation so much faster than estimating every gradient independently?
03In the backward pass, why is the accumulation written as grad += ... instead of grad = ...?
04What causes the vanishing gradient problem in deep sigmoid networks?
05Why does topological order matter in the backward pass?
Click a card to swap the lazy description for what it actually means.
Four problems with exact numbers — trace a backward pass by hand, verify it against finite differences, break the accumulation on purpose, and compute where a sigmoid stack’s gradient crosses the noise floor. Try first; a worked answer is one click away.
Implement __neg__ as self * -1 (the multiply node already exists, so the backward rule comes for free), and __sub__ as self + (-other). Forward: (3 − 1)² = 4. Hand backward: dL/d(a−b) = 2(a−b) = 4; the add node passes 4 to both inputs; the neg node multiplies by −1 on the way to b, so b.grad = −4 and a.grad = 4. Finite-difference check at a: (3.001 − 1)² = 4.004001, so (4.004001 − 4)/0.001 ≈ 4.001 ≈ 4 when h = 10⁻³; a symmetric difference with h = 10⁻⁵ lands within 10⁻⁸ of 4. The two new methods never touched the backward machinery — composition of existing nodes is what autograd calls operator overloading.
Forward returns x if x > 0 else 0; backward passes the incoming gradient unchanged when the input was positive and multiplies it by 0 otherwise. With the same seed and learning rate, the ReLU run escapes the symmetric plateau far sooner — the sigmoid run in this lesson sits near loss 1.0 for the first ~150 epochs because its local derivatives are ≤ 0.25 times small outputs, while ReLU passes factor 1 through every active hidden unit. Expect the ReLU hidden layer to reach loss < 0.01 in a few hundred epochs where the sigmoid version needs roughly 600–900. The catch: watch how many hidden units output exactly 0 on all four rows — those units are dead, receive gradient 0 forever, and are the subject of Lesson 04.
Clipping is one line inside the update loop: p.grad = max(-1, min(1, p.grad)). At weight scale 1.5 the worst-case factor per layer is 1.5 × 0.25 = 0.375, so a 4-layer stack is actually 0.375⁴ ≈ 0.0198 — vanishing, not exploding; to see blow-up you need the derivative side above 1 (tanh at mid-range) or a much larger scale. A better experiment uses tanh: factor 1.5 × 1.0 = 1.5 per layer, so layer-1 gradients are 1.5⁴ ≈ 5.06× the output gradient, and after ten layers 1.5¹⁰ ≈ 57.7×. Without clipping, those early updates overshoot and the loss spikes or turns to NaN; with clipping at [−1, 1], the 57.7× gradient becomes 1×, the update stays bounded, and the curve descends smoothly. The cost is slower progress when large gradients are legitimate — clipping trades speed for stability.
Order the groups output-layer-first: |dL/dW2| ≈ 0.13 per weight, |dL/db2| ≈ 0.20, then hidden layer |dL/dW1| ≈ 0.02 and |dL/db1| ≈ 0.02 — roughly an order of magnitude smaller. The cause is the product of derivatives: a weight in W1 sits behind two sigmoid stages (output-stage derivative ~0.24 and hidden-stage derivative ~0.24), while W2 sits behind one. Each extra sigmoid multiplies the signal by at most 0.25, so early layers get exponentially less gradient. What would change it: ReLU/GELU hidden units (factor 1 or near it), an initialization scaled to keep z near zero, normalization layers that keep units off the saturated tails, or residual connections that give the gradient a path around the multiplications entirely.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 03, Lesson 03) and the Math Foundations Notebook reference build. The seven labs (naive-versus-backprop cost comparison, graph explorer, chain-rule stepper, accumulation checker, finite-difference gradient check with the error-versus-h curve, XOR trainer, vanishing/exploding chart), worked example A for one sigmoid neuron, worked example B for the full 2-2-1 network with its nine-gradient ledger, the h = 10⁻³/10⁻⁵/10⁻⁷ comparison, the seed-42 XOR checkpoint table, the per-activation depth table, and the residual-highway memory hook are original to this page. Every number shown is computed live by the labs or verified by hand in the prose.