dy/dx = dy/du · du/dx One scalar loss flows backward through the graph, picking up a local rate at every node, until all million weights know which way to move. That flow is backpropagation.
A dollar becomes euros at one rate, euros become rupees at another, and the two rates multiply. A composite function is the same chain: dy/dx = f′(g(x)) · g′(x), where each factor is one link's local slope evaluated at the point the chain passes through.
dy/dx = f′(u) · g′(x)The forward pass computes values and caches them. The backward pass starts at the loss with gradient 1 and multiplies by each node's local rate on the way back, so the value at every point is charged exactly what it contributed.
grad += local_rate × upstream_gradA network has millions of weights as inputs and a single loss as output. Reverse mode seeds that one loss with 1 and reaches every weight in one backward pass — roughly the cost of a second forward pass. Forward mode would need a pass per weight.
1 backward pass → all N gradientsNever differentiate the whole composition at once: at each node, compute one local rate, multiply it by the gradient arriving from above, and pass the results on. Seed the loss with 1, walk the graph in reverse, and every weight gets its answer in a single pass.
By the end you will be able to read a computational graph, run a backward pass by hand, explain why PyTorch accumulates with +=, pick a sane step size for gradient checking, and read the 60-line autograd engine like code you wrote yourself.
If y depends on u and u depends on x, then a nudge to x moves u, and that movement in u moves y. The two effects multiply — that is the whole chain rule.
Write the composition as y = f(g(x)). The inner function g turns x into an intermediate value u. The outer function f turns u into y. Each link has its own rate: g′(x) is how much u changes per unit of x, and f′(u) is how much y changes per unit of u.
Plain English: change x by a little; that change is scaled by g′ on the way to u; then it is scaled again by f′ on the way to y. Multiply the two scale factors. Deeper chains simply multiply more factors — a ten-layer network is a ten-link chain.
Drag the point and watch the inner slope, the outer slope and the composite slope move together. The composite always equals their product.
The product is exact; the numeric line is a finite-difference estimate of the same number. Agreement means the two rates really do multiply.
Let δx be a tiny change in x. The inner function turns it into a tiny change in u, scaled by its local rate:
Divide both sides by δx and let it shrink toward zero. The approximations become exact and the product survives as the derivative. (This is the informal teaching version; a real analysis course makes the limit argument carry the error terms along.)
Notice that the outer rate is evaluated at u, not at x. Getting that point wrong is the classic chain-rule mistake: you need f′(g(x)), the outer slope where the chain actually is.
A computational graph turns a formula into a little factory: raw inputs on the left, one operation per station, one output on the right. Building it is what makes the backward pass mechanical.
Every operation becomes a node; every value handed from one operation to the next travels along an edge. The rule is only that the graph flows one way: no cycles, ever. Data moves forward, left to right, and the values computed along the way are kept.
Take the smallest honest neural-network expression: y = relu(x₁·x₂ + b). With x₁ = 2, x₂ = 3 and b = 1, the forward pass is three steps. Multiply first, then add, then bend:
The graph for y = relu(x₁·x₂ + b). Change the inputs, then press Step backward repeatedly: forward values in ink, gradients in orange, arrows lighting up right to left.
Set b to −10 so the pre-activation goes negative: ReLU outputs 0, its local slope becomes 0, and every gradient upstream dies. That is a dead ReLU.
That caching is not an implementation detail; it is the reason the backward pass can be cheap. Every local derivative is evaluated at values the forward pass already computed, so no function is ever re-evaluated. One forward pass, one backward pass, all the gradients.
Start at the output with a gradient of 1, then walk the graph right to left. At each node: multiply the gradient arriving from above by that node’s own local derivative. That is the entire algorithm.
The seed is dy/dy = 1: a tiny nudge to y changes y by exactly that much. From there the rule never changes. An add node hands the gradient unchanged to both of its inputs. A multiply node hands each input the other input’s value, because the other factor is the local rate. A relu node either passes the gradient through (input positive) or kills it (input non-positive).
Gradients only flow to a node once every consumer of that node has pushed its contribution — the dependency order. A depth-first topological sort produces exactly that order, so walking the sorted list in reverse guarantees each node’s gradient is complete before it is read.
| Operation | Local derivative | Why |
|---|---|---|
| out = a + b | ∂out/∂a = 1, ∂out/∂b = 1 | raise an input by δ and the sum rises by δ |
| out = a · b | ∂out/∂a = b, ∂out/∂b = a | the other factor is the exchange rate |
| out = aⁿ | n · aⁿ⁻¹ | power rule (Lesson 4) |
| out = 1 / a | −1 / a² | power rule with n = −1; how division gets a gradient |
| out = eᵃ | eᵃ | the defining property of e; reuse the forward value |
| out = ln a | 1 / a | the inverse of exp (Lesson 4) |
| out = relu(a) | 1 if a > 0, else 0 | a line of slope 1, or flat |
| out = tanh(a) | 1 − tanh²a = 1 − out² | quotient rule, or trust the source; reuse out |
Every line has the same shape: (gradient from above) × (local derivative). Nothing else ever happens in backpropagation. With millions of nodes the bookkeeping grows, but the arithmetic does not.
One fixed expression, f = tanh(a·b + c), and the engine built from two operations. Edit the leaves, then walk the backward pass one node at a time and watch every gradient assemble.
The add node hands the same gradient to both of its inputs; the multiply node hands each input the other input’s value. Nothing else happens, ever.
You can carry derivatives forward from the inputs or pull gradients backward from the output. Neither is wrong; one is dramatically cheaper for the shape of problem training actually is.
Forward mode seeds one input with derivative 1 and carries derivatives through the graph alongside the values. One pass gives you the derivative of every output with respect to that single input. Reverse mode seeds the output with 1 and pulls gradients back. One pass gives you the derivative of that single output with respect to every input.
| Mode | Seed | Direction | One pass gives | Best when |
|---|---|---|---|---|
| Forward mode | dxᵢ/dxᵢ = 1 for one input | input → output | every output's derivative with respect to one input | few inputs, many outputs |
| Reverse mode | dy/dy = 1 at the output | output → input | one output's derivative with respect to every input | many inputs, few outputs — neural networks |
Each node’s backward step does a fixed small amount of work — one multiply-add per input. The backward pass visits every node exactly once. So its total cost is proportional to the number of operations in the forward pass, no matter how many inputs there are.
For N in the millions or billions, that factor is the difference between training being possible and not. Forward mode is not useless — it is the better tool when a function has few inputs and many outputs, and it is elegant to implement with dual numbers — but that is not the shape of a training loss.
Real graphs are not chains — they are webs. One value often feeds several operations, so its gradient arrives along several paths. The total is their sum.
If x is used in two places, a nudge to x changes the loss through both of them. Each route contributes its own product of local rates, and the multivariable chain rule says the total is the sum of the contributions. In the little graph below, L = x·y + x·z, so dL/dx = y + z.
This is not an edge case. Weight tying uses the same matrix twice; recurrent networks reuse a weight at every time step; attention reuses projections. Anywhere a value has more than one consumer, the gradient is a sum.
x feeds two operations, so gradient reaches it along two routes. Step backward and watch the total assemble with += — then simulate the = bug.
PyTorch’s .grad accumulates the same way, which is why optimizers call zero_grad() before each backward pass — otherwise yesterday’s routes add to today’s.
Each route ends with its own += into x’s gradient. Run the routes in any order; the sum assembles itself. That is why the order in which backward closures fire is not delicate — only the rule that every consumer fires before the shared value is read (topological order) matters.
Autodiff is exact arithmetic, but your backward rule can still be wrong. The central difference from Lesson 4 is the referee: an independent estimate that does not use your backward code at all.
The central difference estimates a derivative by evaluating the function on both sides of the point and measuring the slope of the secant: (f(x+h) − f(x−h)) / (2h). It is slower and only approximate, but it knows nothing about your graph, your closures or your topological sort — which is exactly what makes it a good test.
Run both, compare, and treat agreement to roughly 1e−6 or better as a pass. Do it when you add a new operation to an autograd engine, and first when a training loop refuses to converge. Never in production: it costs two extra forward passes per parameter.
Autodiff gives the exact slope; the numerical estimate is a secant. Slide h and watch the error curve: too big is truncation, too small is cancellation.
Use this check when you add a new operation to your engine or when a training loop will not converge. Never in production: it costs two extra forward passes per parameter.
Write out the two evaluations with Taylor series — the function plus its derivatives at x:
The even powers cancel when you subtract — that is why the central difference beats the one-sided (f(x+h) − f(x))/h, whose error shrinks only like h. But no choice of h removes roundoff entirely: at h = 1e−12 the values f(x+h) and f(x−h) differ in their last couple of digits, and the subtraction leaves mostly noise.
def gradient_check(build_expr, x_val, h=1e-7):
x = Value(x_val)
y = build_expr(x)
y.backward()
autodiff_grad = x.grad
y_plus = build_expr(Value(x_val + h)).data
y_minus = build_expr(Value(x_val - h)).data
numerical_grad = (y_plus - y_minus) / (2 * h)
diff = abs(autodiff_grad - numerical_grad)
return autodiff_grad, numerical_grad, diff
def expr(x):
return (x ** 3 + x * 2 + 1).tanh()
ad, num, diff = gradient_check(expr, 0.5, h=1e-5)
print(f"Autodiff: {ad:.8f}") # 0.15252426
print(f"Numerical: {num:.8f}") # 0.15252426
print(f"Difference: {diff:.2e}") # 2.80e-11Three ingredients: wrap every number, record every operation, walk the recorded graph in reverse. That is a complete autodiff engine — the same design PyTorch uses, scaled up.
Wrap. A Value stores its number, its gradient (start at 0), a pointer to its child nodes, and a _backward closure. Record. Every arithmetic operation creates a new Value and fills in a closure that knows that operation’s local derivative. Walk. backward() topologically sorts the graph, seeds the output with 1, and calls each closure in reverse order.
class Value:
def __init__(self, data, children=(), op=""):
self.data = float(data)
self.grad = 0.0 # dL/d(this), filled by backward()
self._backward = lambda: None # how to push my grad to my children
self._prev = set(children)
self._op = op
def __add__(self, other):
other = other if isinstance(other, Value) else Value(other)
out = Value(self.data + other.data, (self, other), "+")
def _backward():
self.grad += out.grad # d(a+b)/da = 1
other.grad += out.grad # d(a+b)/db = 1
out._backward = _backward
return out
def __mul__(self, other):
other = other if isinstance(other, Value) else Value(other)
out = Value(self.data * other.data, (self, other), "*")
def _backward():
self.grad += other.data * out.grad # d(ab)/da = b
other.grad += self.data * out.grad # d(ab)/db = a
out._backward = _backward
return out
def relu(self):
out = Value(max(0, self.data), (self,), "relu")
def _backward():
self.grad += (1.0 if out.data > 0 else 0.0) * out.grad
out._backward = _backward
return out
def backward(self):
topo, seen = [], set()
def build(v):
if v not in seen:
seen.add(v)
for child in v._prev:
build(child)
topo.append(v)
build(self)
self.grad = 1.0 # seed: dL/dL = 1
for v in reversed(topo): # reverse topological order
v._backward()
x1, x2 = Value(2.0), Value(3.0)
y = (x1 * x2 + 1.0).relu()
y.backward()
print(x1.grad, x2.grad) # 3.0 2.0 — matches PyTorch.grad; one call to loss.backward() fills all of them, and the update rule nudges each leaf against its gradient.model = MLP([2, 4, 1])
xs = [[0, 0], [0, 1], [1, 0], [1, 1]]
ys = [-1, 1, 1, -1] # XOR, in tanh's range
for step in range(100):
loss = sum((model(x) - y) ** 2 for x, y in zip(xs, ys))
for p in model.parameters():
p.grad = 0.0 # zero_grad, or yesterday adds in
loss.backward() # one call fills every gradient
for p in model.parameters():
p.data -= 0.05 * p.grad # nudge every weight downhillThis is micrograd: a complete neural network training loop in pure Python. Every commercial deep learning framework does the same thing at massive scale — tensors instead of scalars, compiled kernels instead of closures, millions of nodes instead of dozens.
Answer before you look. The reverse-mode and accumulation questions are the ones that explain why real training code is written the way it is.
0 / 5 answered · 0 correct
01What does the chain rule state?
02What is a computational graph?
03Why does PyTorch use reverse-mode autodiff instead of forward mode?
04Why does the backward function use '+=' instead of '='?
05What is gradient checking and when should you use it?
Click a card to swap the lazy description for what it actually means.
Four short problems. Try first; a worked answer is one click away.
The op creates out = Value(self.data ** n, (self,), '**n'), and its backward closure is self.grad += n * self.data ** (n - 1) * out.grad. At x = 2 with n = 3: 3 · 2² = 12. Numeric check with central differences: ((2.001)³ − (1.999)³) / (2·0.001) = (8.012006001 − 7.988005999) / 0.002 = 12.000001. The power rule and the engine agree.
Forward: t = math.tanh(self.data); out = Value(t, (self,), 'tanh'). Backward: self.grad += (1 - t ** 2) * out.grad — reuse the forward output t, no second tanh call. At 0: tanh(0) = 0 so the slope is 1 − 0² = 1. At 2: tanh(2) ≈ 0.96403, so the slope is 1 − 0.92935 ≈ 0.07065, the stated 0.0707. A quick central difference on tanh at 2 also gives ≈ 0.07065.
Pick w₁ = 1, w₂ = 2, x₁ = 0.5, x₂ = −1, b = 3. Forward: z = 0.5 − 2 + 3 = 1.5, and since z > 0, y = relu(1.5) = 1.5 with relu' = 1. Backward: dy/dw₁ = x₁ = 0.5, dy/dw₂ = x₂ = −1, dy/db = 1, dy/dx₁ = w₁ = 1, dy/dx₂ = w₂ = 2. Check by nudging w₂ up 0.001: z = 1.499, y = 1.499, so Δy/Δw₂ = −0.001/0.001 = −1 ✓. In PyTorch the same five numbers land in w1.grad, w2.grad, b.grad, x1.grad, x2.grad after y.backward().
A Dual is a pair (value, derivative) with rules (a, a') + (b, b') = (a + b, a' + b'), (a, a') · (b, b') = (a·b, a'·b + a·b'), and tanh(a, a') = (tanh(a), (1 − tanh²a)·a'). Seed the input as (x, 1) and read the derivative off the output. For f(x) = tanh(x³ + 2x + 1) at x = 0.5: x³ = (0.125, 0.75), 2x = (1, 2), the sum is (2.125, 2.75), and tanh gives (0.97187, (1 − 0.97187²)·2.75) = (0.97187, 0.15252). Reverse mode gives the same 0.15252 — one pass per input here, versus one pass for all inputs there.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 01, Lesson 05) and the Math Foundations Notebook reference build. Interactive figures, the animated backward-pass hero, worked exercise answers and the autodiff console are original to this page. Every lab runs in your browser.