Nine weights.
Every position.

Start with the smallest possible picture of the operation. A 3×3 kernel holds nine weights per input channel. Place it at the top-left corner of an image, multiply each weight by the pixel under it, and sum the nine products — that is a dot product, exactly the operation from Phase 1, applied to a 3×3 window instead of two whole vectors. Slide the kernel one pixel right and do it again. Slide it down a row and do it again. The grid of sums you collect is the output, so a 5×5 input with a 3×3 kernel at stride 1 produces a 3×3 output: nine placements, nine sums.

Why bother with a sliding window instead of a normal dense layer? Do the arithmetic. A 224×224 colour image is 224 × 224 × 3 = 150,528 numbers. One dense layer neuron connected to all of them needs 150,528 weights; a hidden layer of 1,000 such units needs 150,528,000 weights — about 150.5 million parameters, which is 574 MiB in float32 for a single layer, before the network has learned anything. A 3×3 convolution producing 64 channels from those same three input channels needs 64 × 3 × 3 × 3 + 64 = 1,792 parameters — about 7 KiB. The dense layer has 5,376× more weights, and it still has no idea that it is looking at an image.

The problem is not just size. A dense layer treats every pixel position as an independent knob: it learns “bright at row 12, column 40”, not “bright”. Move the cat three pixels to the right and every one of those knobs is wrong. A convolution makes two assumptions instead, both true for photographs:

The lesson’s opening example makes the arithmetic concrete. Take this 5×5 input and the source’s 3×3 kernel, slide with no padding and stride 1, and compute the first output pixel by hand:

Repeat for all nine placements and the whole 3×3 output falls out. The next chapter is about the sliding schedule; for now the important part is that nothing else is happening — nine multiplies and a sum, 81 multiplies in total, produced by ten numbers (nine weights plus one bias). A dense layer that maps the same 25 inputs to the same 9 outputs would need 9 × 25 = 225 weights and nine separate biases: 234 parameters, 23.4× more, with no sharing and no locality.

Here is the complete output — all nine placements, including the ones that touch the bottom-right corner:

The playground below runs this exact arithmetic live. Drag the window, swap in a blur or an edge kernel, edit a single weight, and watch the output map recompute — the same nine numbers doing the work at all sixteen placements.

Padding adds a ring of extra values around the input before the kernel starts sliding. Without it, every convolution shrinks the feature map: a 3×3 kernel at stride 1 removes one row and one column from each side, because the kernel can never centre on the outermost pixels. Stack 20 of those and a 224×224 image is down to 184×184, and any residual connection that expects matching shapes breaks. A ring of zeros (P = 1) lets the kernel centre on pixel (0, 0) and still have three rows and three columns of values to multiply — at the price of inventing values that were never in the image. That trade-off is why padding modes exist: zero (most common), reflect (mirror the edge), replicate (copy the edge) and circular (wrap around, for genuinely periodic data).

Stride is the step size of the slide. Stride 1 visits every position. Stride 2 skips every other column and row, which halves each spatial dimension — the classic way modern CNNs downsample inside the layer instead of with a separate pooling layer. ResNet, ConvNeXt and MobileNet all use strided convolutions in place of max-pool somewhere in the stack.

Read the middle column as the lesson’s three worked cases with the same 5×5 input and the same 3×3 kernel: with no padding and stride 1 you get a 3×3 output (9 placements); add a one-pixel zero ring and stride 1 keeps the input size, 5×5; keep the ring but stride 2 and the kernel lands on three columns and three rows — also 3×3, but from placements spaced two pixels apart instead of one. Every one of those numbers is floor((5 − 3 + 2P)/S) + 1.

“Same padding” is the common name for choosing P = (K − 1) / 2 when the kernel is odd, which makes H_out = H at stride 1: 3→1, 5→2, 7→3. That is a large part of why 3×3 kernels dominate — they are the smallest odd kernel with a true centre, and they cost one pixel of padding per side. With an even kernel there is no symmetric choice: K = 4 needs P = 2 on one side and P = 1 on the other to keep the size unchanged.

One more habit that pays off immediately: when H + 2P − K is not a multiple of S, the floor silently discards the remainder — no warning, no error, just a feature map one row smaller than you expected. The calculator below flags exactly that case, plus the impossible combinations where the kernel is larger than the padded input.

A convolution layer can learn its kernels from data — that is the whole point of a convolutional neural network (CNN). But it is easier to see what a kernel means when you write it by hand. Every design below is one 3×3 matrix, and the property that predicts its behaviour is the sum of its weights, its DC gain:

A kernel that sums to 1 preserves the brightness scale: a flat region of value 40 comes out as 40, so blur and identity read as modified photographs. A kernel that sums to 0 is a difference: a flat region of 40s comes out as exactly 0, and only change survives. That single number explains why an edge map is mostly black — most pixels are not an edge.

To see the difference kernels work, point Sobel-x at a 16×16 image that is 0 on the left half and 1 on the right half. At the placement whose window sits one column left of the step, the three columns are 0, 0, 1:

Two details in that printout are worth keeping. The response is a pulse two columns wide, not one — edges in pixel space have width, and a single-column expectation is the most common Sobel surprise. And the final column reads −4 purely because of padding: a bright image running straight into the zero ring looks like an edge, which is why border artifacts appear in real edge maps and why practitioners often crop or ignore a one-pixel border after filtering.

The sign of a kernel is a convention, not a fact: flip all nine weights of Sobel-x and every response flips sign — the same edges light up, as negative numbers instead of positive. That is why the source’s W (from the previous chapter) and the standard Sobel-x differ only in sign, and why a magnitude image looks identical either way.

These filters are not toys the deep-learning era left behind. The first convolutional layer of AlexNet (2012) and VGG (2014), trained purely by gradient descent on ImageNet, learned banks of edge and colour-blob detectors that look strikingly like Gabor filters — the same shapes this gallery writes by hand. A good image model needs edge detectors no matter what task comes later, so it builds them.

Build the reference first. The smallest primitive is padding: a function that adds p zeros around an H×W array. The trailing-axes trick x.shape[:-2] means the same function works on (H, W), (C, H, W) or (N, C, H, W) without modification — worth doing once, because every convolution in the rest of the phase will use it.

Then the definition itself: one output channel, one output row, one output column, and a multiply-accumulate over C_in × K × K values. That is where the classic “quadruple loop” name comes from — oc, i, j, and the implicit sum over the patch.

The output of a conv layer is a volume, not a map. A real image is (C_in, H, W) — three channels — and one output channel needs one 3×3 slice of weights per input channel. A layer that produces C_out channels stacks C_out of these volumes into a weight tensor of shape (C_out, C_in, K, K). For 64 output channels on a 3-channel input that is 64 × 3 × 3 × 3 = 1,728 weights plus 64 biases: 1,792 parameters, the number the quiz asks you to recognize. The general formula is C_out × C_in × K² + C_out.

Now the trick. Nested Python loops are easy to read but leave the hardware idle; GPUs — graphics processing units — are built to multiply big matrices. im2col — short for image-to-column — rearranges the input so that the convolutionbecomes one matrix multiply: every kernel-sized window of the input is flattened into one column of a matrix, and the kernel is flattened into a row. The lazy version of the same loops still extracts the windows — but the heavy lifting afterwards is a single @.

im2col earns its keep at scale. For a 3×224×224 image with a 3×3 kernel and P=1, the output is 224×224 = 50,176 positions, each column is 3 × 3 × 3 = 27 numbers, and the whole cols matrix holds 27 × 50,176 = 1,354,752 floats — 5.4 MB in float32, exactly 9× the raw image (27/3). One image is fine. A batch of 64 is 1,354,752 × 64 × 4 bytes ≈ 331 MiB of extra memory for a single layer, which is why production kernels often refuse to materialize it: implicit GEMM (general matrix multiply), direct convolution, Winograd for 3×3, and FFT (Fast Fourier Transform) convolution for large kernels are all strategies for getting the matmul speed without the cols tax.

The other number worth carrying around is the work. Every output pixel needs C_in × K² multiply-accumulates per output channel, and there are H_out × W_out × C_out of them:

The receptive field of an activation is the patch of the original input that it depends on. A single 3×3 convolution has a 3×3 field. Put a second 3×3 convolution on top of it and the second layer’s pixels each see 3×3 of the first layer’s map — but a corner pixel of that map already depended on 3×3 of the input, so the total footprint is 5×5. Nothing new was added to the operator; depth composed the windows.

Two things follow from that table. First, you can build any receptive field you want out of 3×3 layers, one ring at a time. Second, the arithmetic is why VGG, ResNet and ConvNeXt all settled on “3×3 all the way down”: two 3×3 layers reach the same 5×5 field as one 5×5 layer with 18 weights per channel pair instead of 25 (28% fewer) and an extra nonlinearity in between — and three 3×3 layers reach 7×7 with 27 weights against 49 for a single 7×7 (45% fewer). For 64 input and 64 output channels the numbers are 2 × 64 · 64 · 9 = 73,728 against 64 · 64 · 25 = 102,400. Smaller and more expressive: an unusually one-sided trade.

Strides change the arithmetic but keep the idea. Each strided layer widens the ring by (K − 1) × the product of all strides before it, so a few downsamples let a ten-layer network cover the whole 224×224 image. Worked example with the source’s function: layers [3×3 s1, 3×3 s2, 3×3 s2] produce fields of 3, then 5, then 9 — the final layer’s ring of 2 is multiplied by the stride product 2 left behind by the stride-2 layer below it, so it adds 4 to the field instead of 2.

This is also the right place to pin down translation equivariance, the property the whole design exists to exploit. Convolution is equivariant: shift the input by (Δi, Δj) pixels and the output shifts by the same amount, because the same weights ran over the shifted window. It is not invariant — the output didn’t stay put, it moved with the input. Invariance comes later, from pooling the whole map down to a single vector for a classifier: that is what lets a network answer “is there a cat?” instead of “where is the cat?”. Two honest caveats: with stride S, exact shifts survive only for multiples of S, and near the borders, padding makes the equivalence approximate rather than exact.

Everything in this lesson transfers directly. nn.Conv2d owns a weight tensor of shape (C_out, C_in, K, K) and one bias per output channel, initializes them the way Phase 3 taught, and runs the forward pass as a single fused kernel on the GPU. Change padding or stride and the output follows the formula from chapter 02 to the pixel.

Three production details are worth knowing before you meet them in a real architecture. First, strided convolutions replace pooling: a 3×3 stride-2 layer with P=1 takes 224×224 to 112×112 and learns how to downsample instead of throwing away the odd pixel. Second, depthwise separable convolution splits the job into a depthwise conv (one K×K kernel per channel,C · K² + C parameters) followed by a pointwise 1×1 conv that mixes channels (C_in · C_out + C_out). For C=64 and K=3 that is 640 + 4,160 = 4,800 parameters instead of 64 · 64 · 9 + 64 = 36,928 — about 7.7× fewer — which is how MobileNetV2 fits in a phone. Third, cross-correlation, not convolution: the mathematical convolution flips the kernel before sliding, and every deep-learning framework skips that flip because the weights are learned anyway. The sign conventions in chapter 03 are signs; the leaky abstraction is universal, so it is worth saying out loud once.

If you want to see the operation living inside a modern architecture, look at the patch embedding of the ViT (Vision Transformer): it is literally nn.Conv2d(3, 768, kernel_size=16, stride=16) — a convolution whose 16×16 kernels are the patches, with stride equal to the kernel size so they do not overlap. The transformer’s “patchifier” is a convolution.