A loss is a landscape, and the derivative is the slope under your feet. Subtract a small multiple of it from every weight — w = w − lr · dL/dw — and the error falls. That loop is training.
Draw a line through two nearby points on a curve and shrink the gap. The secant pivots until it becomes the tangent at one point — and its slope is the derivative f'(x). For x², that slope is 2x: at x = 2 it is 4.
f'(2) = 4 nudge x by 0.001 → y moves ≈ 0.004A loss depends on every weight at once. ∂L/∂w asks how the loss responds to one weight while all the others are held still. For f(x, y) = x² + 3xy + y², ∂f/∂x = 2x + 3y and ∂f/∂y = 3x + 2y.
∂f/∂x = 2x + 3y (y frozen)Collect the partials into one vector and you get ∇L, the direction of steepest ascent. Training subtracts a multiple of it from every weight. Too large a step and the loss bounces; too small and it crawls.
w ← w − lr · ∂L/∂wTraining is a loop: predict, measure the loss, read the slope of the loss with respect to every weight, step downhill, repeat — and calculus is the part that tells you which way downhill is.
By the end you will be able to read dL/dw and ∂L/∂w without fear, explain why the gradient points uphill, predict what a learning rate does before you run it, and check any gradient by hand with a finite difference.
A derivative answers one question: nudge the input a hair — how much does the output move? The answer is the slope of the curve at that exact point.
Take two points on a curve and draw a line through them: that is a secant line, and its slope is the average rate of change over the gap. Shrink the gap and the secant pivots toward the tangent at a single point. The limit of that slope as the gap goes to zero is the derivative, written f’(x).
For f(x) = x², the derivative is f’(x) = 2x. At x = 2 the slope is 4: nudge x right by 0.001 and f rises by about 0.004. At x = 0 the slope is 0 — the curve is flat, and you are standing at the bottom of the bowl.
Drag the green point along the curve, then shrink h. The orange secant line through the two nearby points becomes the blue tangent line — and its slope becomes the derivative.
Numerically h never reaches zero — a tiny h loses precision to floating-point rounding. The best real-world h sits around 1e-5 to 1e-7. The next chapter shows exactly where it breaks.
The same expansion with xⁿ gives the power rule: d(xⁿ)/dx = n·xⁿ⁻¹. Two more facts you will use constantly: the derivative of a constant is 0, and derivatives add term by term.
There are three ways to get a slope: derive it by hand, approximate it with nearby points, or let a framework differentiate the code itself. The middle one is the only one you can check with a calculator — and the one with a hidden failure mode.
The analytic derivative is a formula: for x², it is 2x. Exact and fast — but somebody has to derive it. The numerical derivative skips the algebra: evaluate the function at two nearby points and take the slope of the line between them. The central form (f(x + h) − f(x − h)) / 2h is more accurate than a one-sided version because the errors from the two sides cancel.
Frameworks take a third road: automatic differentiation (autodiff) applies the chain rule to the code itself, giving exact derivatives without writing any calculus by hand. Lesson 5 builds it. Everything in this chapter is how you verify that machinery: compare its answer against a finite difference and watch them agree.
Compare the analytic derivative with the central difference (f(x+h) − f(x−h)) / 2h as h shrinks from 10⁻¹ to 10⁻¹⁶. The error falls, then explodes — that explosion is floating-point cancellation.
Rule of thumb: h between 1e-5 and 1e-7 for float64. Frameworks never use this in production — they work out exact derivatives symbolically (autodiff), and use finite differences only to check them.
def numerical_derivative(f, x, h=1e-7):
return (f(x + h) - f(x - h)) / (2 * h) # central difference
def central_check(f, df, x, h=1e-5):
numeric = numerical_derivative(f, x, h)
exact = df(x)
return numeric, exact, abs(numeric - exact)
for x in [-2, -1, 0, 1, 2]:
num, exact, err = central_check(lambda t: t ** 2, lambda t: 2 * t, x)
print(f"x={x:2d} numeric={num:10.6f} analytic={exact:5.1f} error={err:.1e}")
# x=-2 numeric= -4.000000 analytic= -4.0 error=0.0e+00
# x= 0 numeric= 0.000000 analytic= 0.0 error=0.0e+00
# x= 2 numeric= 4.000000 analytic= 4.0 error=0.0e+00Write the Taylor expansion of the function at the two neighbours of x, one step h to the right and one to the left:
Subtracting the two expansions kills every even-power term, including the ½f″h² term that dominated the forward difference. With h = 0.001, an error “like h” is around 0.001 while “like h²” is around 0.000001 — a thousand times better from the same two function evaluations.
A loss depends on every weight at once. A partial derivative freezes all of them except one and asks the only question training needs: how does the loss respond to this single knob?
For a function of one variable we write df/dx. For a function of many variables we write ∂f/∂x — the curly “d” means partial: differentiate with respect to x while pretending every other variable is a constant. In ML the loss depends on the weights, so ∂L/∂w is the loss’s sensitivity to one weight, with all the others held still.
Take f(x, y) = x² + 3xy + y². For ∂f/∂x, treat y as a fixed number: x² becomes 2x, 3xy becomes 3y (a constant times x), and y² becomes 0. So ∂f/∂x = 2x + 3y. Likewise ∂f/∂y = 3x + 2y. At the point (1, 2) that is [2 + 6, 3 + 4] = [8, 7].
Pick a function and a point. Each row shows one partial derivative: its formula, its exact value, a numerical check, and the update it would trigger.
f(x,y) = x² + 3xy + y²
| partial | formula | exact | numeric (2h = 0.00002) |
|---|---|---|---|
| ∂f/∂x | 2x + 3y | 8.0000 | 8.0000 |
| ∂f/∂y | 3x + 2y | 7.0000 | 7.0000 |
The exact and numeric columns agree to four decimals because the central difference is exact for quadratics. That is the check backpropagation implementations are verified against.
The trick is to replace the frozen variable by a number and forget it is a variable at all.
Notice that neither partial tells you what happens when both move at once. That is the job of the gradient, which collects them into one vector.
Collect every partial derivative into one vector and you have the gradient. It points in the direction of steepest ascent — so training walks in the exactly opposite direction.
The gradient, written ∇f, is the list of all partial derivatives: ∇f = [∂f/∂x, ∂f/∂y, …]. For f(x, y) = x² + y², it is [2x, 2y]. At the point (1, 1) that is [2, 2]: moving away from the origin in any direction raises f, and the direction that raises it fastest is the diagonal. At the bottom of the bowl, (0, 0), the gradient is [0, 0] — flat in every direction.
Two facts make the whole training loop work. First, the gradient points uphill. Second, it is perpendicular to the contour lines of f, so it is the most efficient direction to move if you want to change f quickly. To decrease the loss, take the gradient and flip its sign.
The bowl, valley and saddle in the lab are teaching models — a real network’s loss lives in millions of dimensions and cannot be drawn. The rule being demonstrated is exactly the same: one partial derivative per parameter, collected into one vector.
Click anywhere to sample a point. Blue arrows show the downhill direction −∇f; the orange circle marks the function’s special point. The contour lines are level sets: the gradient is always perpendicular to them.
Each partial derivative is the slope along one axis with the other frozen. Together they form the gradient, which always points uphill — so training takes its step in the opposite direction.
| Point | ∇f for x² + y² | −∇f (descent) | What it means |
|---|---|---|---|
| (2, 1) | [4, 2] | [−4, −2] | steepest way up is right and slightly up; downhill is the reverse |
| (1, 1) | [2, 2] | [−2, −2] | the diagonal, straight back toward the bowl’s bottom |
| (0, 0) | [0, 0] | [0, 0] | flat everywhere: you are at the minimum |
Forget pictures for a second and do the arithmetic. Suppose you move a tiny step (dx, dy) away from the point (x, y). The change in f is, to first order:
That is a dot product between the gradient and your step. A dot product is largest when the two vectors point the same way, so the step that increases f fastest is the one parallel to ∇f. The step that decreases f fastest is −∇f. Every optimization algorithm in deep learning is a variation on that one observation.
Subtract a small multiple of each partial derivative from its weight. Repeat. That one line, run over every knob, is the whole of neural network training.
The rule is w ← w − lr · ∂L/∂w, applied to every weight at once. The learning rate lr scales the step. The partial derivative says which way increases the loss, so subtracting moves the weight the other way. Because the slope flattens near a minimum, the steps shrink on their own as the model approaches the answer — no schedule required.
Get lr wrong and the curve tells you. Too small: hundreds of tiny steps and still far away. Too large: each step overshoots the valley, the loss bounces between the walls, and past a certain point it grows without bound. The lab is the fastest way to build that intuition.
Pick a loss curve and a learning rate, then step. Each step subtracts lr × slope from x. Too small and it crawls; too large and it bounces or diverges.
On x², lr = 0.1 converges smoothly, lr ≈ 1 oscillates forever (x alternates sign), and lr > 1 diverges. On the double well, the start point decides which of the two valleys you fall into.
Model: prediction = w·x + b. Loss on one example: (w·x + b − y)². Call the bracket err.
xs = [1.0, 2.0, 3.0, 4.0, 5.0]
ys = [3.0, 5.0, 7.0, 9.0, 11.0] # truth: y = 2x + 1
w, b, lr = 0.0, 0.0, 0.01
for epoch in range(200):
dw = db = loss = 0.0
for x, y in zip(xs, ys):
err = (w * x + b) - y # prediction minus target
loss += err ** 2
dw += 2 * err * x # d(err²)/dw = 2·err·x
db += 2 * err # d(err²)/db = 2·err
n = len(xs)
w -= lr * dw / n
b -= lr * db / n
# after 200 epochs: w ≈ 1.98, b ≈ 1.07 — converging on y = 2x + 1A network is a composition: linear, then activation, then linear, then loss. The chain rule says the derivative of a composition is the product of the local derivatives — link by link.
If y = f(g(x)), then dy/dx = f’(g(x)) · g’(x): the outer derivative evaluated at the inner value, times the inner derivative. For y = (3x + 1)², the outer function is u² with derivative 2u, and the inner is 3x + 1 with derivative 3, so dy/dx = 2(3x + 1) · 3 = 6(3x + 1).
This is the whole trick behind deep learning. The forward pass walks left to right through the chain; backpropagation (Lesson 5) walks right to left, multiplying one local derivative at each step. When a value is used in two places, its contributions add — which is why the gradient of a parameter sums over every path through the graph.
σ(x) = 1 / (1 + e⁻ˣ) = (1 + e⁻ˣ)⁻¹. Use the chain rule with outer function u⁻¹ and inner u = 1 + e⁻ˣ.
That shrinkage is exactly why deep networks abandoned sigmoid for ReLU (Lesson 2): ReLU’s derivative is 1 wherever the unit is active, so the chain does not get quieter with depth.
The first derivative gives the direction. The second derivative — the Hessian, in many dimensions — describes how the slope itself bends, and that is what makes better optimizers possible.
A gradient step assumes the function is a straight line in the direction you are walking. Near the minimum that is optimistic: the line keeps going down forever, but the curve flattens. Curvature measures how quickly the slope changes. A large learning rate works when curvature is gentle and misbehaves when it is sharp, because the step leaves the region where the linear approximation was trustworthy.
In many dimensions the second derivatives form the Hessian matrix: entry (i, j) is how the slope of parameter i changes when parameter j moves. At a point where the gradient is zero, the Hessian’s eigenvalues classify the point — all positive is a minimum, all negative a maximum, mixed signs a saddle.
| Method | Uses | Cost per step | Reality check |
|---|---|---|---|
| Gradient descent | first derivatives only | O(N) | slow but reliable |
| Newton's method | full Hessian H | O(N³) | fast, impractical for big N |
| L-BFGS | Hessian estimated from gradient history | O(N) | superlinear, medium models |
| Adam | per-parameter running moments of the gradient | O(N) | the deep learning default |
Taylor’s theorem says any smooth function is locally a polynomial. Write h for the step away from x:
Answer before you look. The gradient, learning-rate and finite-difference questions are the ones that come up every time you debug a training run.
0 / 5 answered · 0 correct
01What does the derivative of a function at a point tell you?
02What is a gradient in ML?
03What does w = w − lr · dL/dw accomplish?
04Why can't Newton's method be applied directly to networks with millions of parameters?
05What is the central-difference approximation for f'(x)?
Click a card to swap the lazy description for what it actually means.
Three short problems. Try first; a worked answer is one click away.
def numerical_derivative(f, x, h=1e-5): return (f(x + h) - f(x - h)) / (2 * h) def numerical_second_derivative(f, x, h=1e-3): return (numerical_derivative(f, x + h) - numerical_derivative(f, x - h)) / (2 * h) f = lambda x: x ** 3 print(numerical_second_derivative(f, 2)) # ≈ 12.0000 The calculus: f'(x) = 3x², so f''(x) = 6x, and 6·2 = 12. The nesting is itself a warning: the outer difference builds two inner differences, so rounding noise is amplified. Use a larger outer h (1e-3) than the inner one (1e-5), or use the direct central second difference (f(x+h) − 2f(x) + f(x−h))/h², which needs only one subtraction of nearby values.
The partials are ∂f/∂x = 2(x − 3) and ∂f/∂y = 2(y + 1), so with lr = 0.1 each coordinate follows x ← x − 0.2(x − 3), i.e. the distance to 3 is multiplied by 0.8 every step. step 0: (0.000, 0.000) step 1: (0.600, −0.200) step 2: (1.080, −0.360) step 3: (1.464, −0.488) ... step 20: (2.965, −0.988) step 30: (2.996, −0.999) The two coordinates have their own slopes and are optimized independently. The steps shrink as the slope flattens, so gradient descent slows down automatically near the minimum. A quadratic bowl like this one has no local minima to get caught in.
Momentum: v ← β·v + f'(x), then x ← x − lr·v, with β around 0.9. Plain descent: x ← x − lr·f'(x). From x = 2.2, f'(x) = 4x³ − 6x. Plain descent with lr = 0.02 walks 2.200 → 1.612 → 1.470 → 1.393 → 1.344 → … toward the valley at x ≈ 1.2247 (f = −2.25). Momentum with lr = 0.01 and β = 0.9: v = 29.39, x = 1.906 v = 42.71, x = 1.479 v = 42.51, x = 1.054 Momentum reaches the valley region in three steps that plain descent needed five or six for — consistent directions accumulate, so it accelerates on the long slope. But it also overshoots: the third step lands left of the minimum, and the velocity has to bleed off before it settles. The lesson is that momentum makes the effective step size larger, so lr must be retuned (and it is exactly why Adam, in Lesson 8, tracks per-parameter moments instead of one global momentum).
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 01, Lesson 04) and the Math Foundations Notebook reference build. Interactive figures, added AI visuals, worked exercise answers, and the four new calculus labs are original to this page. Every lab runs in your browser.