Every CNN since 1998 is convolutions, a nonlinearity and downsampling feeding a small head. The history of vision is one bolt-on per generation: ReLU and dropout, 3×3 stacks, parallel paths, and the identity skip that made 1,000 layers trainable. Learn the ideas in order, then build all three networks in under 40 lines each.
LeNet-5 wrote that recipe in 1998 — two conv-pool blocks, three dense layers, 61,706 parameters, trained on a CPU. Every CNN since is the same list made deeper, wider and smarter about where the downsampling goes. Once you can read the recipe, a new backbone is twenty minutes of reading, not a research project.
input (1,32,32) → (6,28,28) → (6,14,14) → (16,10,10) → (16,5,5) → 400 → 120 → 84 → 10AlexNet (2012) swapped tanh for ReLU, added dropout and scale — 85% top-5. VGG (2014) used only 3×3 stacks and went 16 layers deep. Inception (2014) ran 1×1, 3×3, 5×5 and a pool in parallel, then concatenated. ResNet (2015) added y = F(x) + x and took the same dataset to 96%. No new data; new ideas.
60k params (LeNet) → 60M (AlexNet) → 138M (VGG) → 11.7M (ResNet-18)The degradation problem: a 34-layer plain net has higher training loss than an 18-layer plain net. Both curves are worse, so more data will not fix it. The identity skip y = F(x) + x gives every block a do-nothing option and every gradient a route that does not shrink, which is why 100+ layer vision nets — and every transformer block — train at all.
L stacked 3×3 convs see r = 1 + 2L · two 3×3s cost 18C² vs one 5×5 at 25C²A CNN is one recipe — conv → nonlinearity → downsample — repeated until the map is small, then a small head; every famous family is that recipe with one new idea bolted on, and reading a backbone means finding its stem, its block, its skips and its head.
By the end you will be able to predict any layer’s output shape with out = ⌊(H − K + 2P) / S⌋ + 1; count parameters by hand (C_in·C_out·K²; a 3×3 at 64→64 is 36,864 weights); say which family contributed ReLU, 3×3 stacks, parallel paths and the identity skip; explain the degradation problem and why residual connections fix it; implement LeNet-5, a VGG block and a ResNet BasicBlock in PyTorch, each under 40 lines; and account for the real parameter counts — LeNet-5 61,706, AlexNet ≈60M, VGG-16 138,357,544, ResNet-18 11,689,512, ResNet-50 25,557,032 — before you look at the source.
In 2011 the best ImageNet classifier reached about 74% top-5. AlexNet hit 85% in 2012. ResNet hit 96% in 2015. Same dataset, same silicon generation — the gains were architecture.
The benchmark everyone is climbing is ImageNet: 1.28 million labelled training images, 1,000 classes, 50,000 images held out for validation. “Top-5 accuracy” means the correct class is among the model’s five best guesses. In 2011 the best pipelines got 74 of 100 images into that top five using hand-designed features. One year and one architecture later, a CNN got 85. Three years later, 96.
A working vision engineer has to know which idea came from which paper, for two practical reasons. First, every production backbone you will meet is a recombination of these same pieces — grouped convolutions went from CNNs into transformers, residual connections went from ResNet into every LLM in existence, batch normalisation lives in diffusion models. Second, studying the lineage in order immunises you against a very expensive habit: reaching for the biggest available model when a LeNet-sized one would do. MNIST does not need a ResNet, and the scale difference is not a rounding error.
That is the whole argument in one comparison: the bigger network is three orders of magnitude more expensive and does not automatically solve your problem better. The rest of this lesson is the ordered list of what each family actually added, so that when you read an unfamiliar backbone you can name its parts instead of guessing.
Every network in this lesson is three moves in a loop: a convolution mixes nearby pixels, a nonlinearity bends the result, a downsample shrinks the map. When the map is small enough, a tiny classifier head makes the decision.
LeNet-5 is the recipe written out in full, and its shape trace is the one to memorize. This is the source’s exact forward pass:
Read that list as a sentence and you have the whole field: extract local features, shrink the map so later layers see more of the image, repeat until the map is tiny, then classify. AlexNet has more layers, VGG has more convs per block, ResNet has skips — none of them change the sentence. The only numbers that vary between families are how many repeats, how wide each layer is, and where the downsampling sits.
Shapes are not guesswork. Every conv and pool in every one of these networks follows the formula from Lesson 02:
Four settings cover every layer in this lesson. Valid (no padding) shrinks: a 5×5 on 32 gives 28. Same pads by (K−1)/2 so the size is unchanged: 3×3, P=1, S=1 on 64 gives 64 — which is why almost every 3×3 in a modern backbone has padding=1. Pooling divides: 2×2 stride 2 on 28 gives 14. And stride is the third dial: ResNet’s 7×7 stem uses stride 2, so 224 becomes ⌊(224 − 7 + 6) / 2⌋ + 1 = 112. The output-shape lab below makes you predict each of these before revealing it.
Channels are a separate bookkeeping axis, and they never change the spatial arithmetic. A conv layer is written C_in → C_out, K×K, and its parameter count is C_in × C_out × K² weights, plus one bias per output channel. A 3×3 at 64 → 64 is 64 × 64 × 9 = 36,864 weights. That single multiplication is the reason parameter counts in this lesson vary by four orders of magnitude, and it is the one the parameter-calculator lab makes you do by hand.
Predict every layer before you see it. One formula does all the work: out = ⌊(H − K + 2P) / S⌋ + 1. Get through the LeNet-5 trace and the last three rows ask for dense-layer weights — which is how the whole network adds up to 61,706.
input: 1 × 32 × 32 · What shape comes out of conv 5×5, stride 1, pad 0?
A conv layer also has a channel count — shape traces in the lesson write it as (C, H, W). The spatial arithmetic above never touches C; channels are a separate dial you set per layer.
A kernel of size K placed on a 1-D input of size H needs K positions to fit. Add P zero-padded cells on each side and the padded length is H + 2P, so the number of K-wide windows that fit is (H + 2P − K). If the kernel jumps S cells at a time, the number of jumps is ⌊(H + 2P − K) / S⌋, and the +1 counts the first placement. Run it on LeNet-5:
Two consequences worth keeping. First, odd kernels are the friendly case: P = (K−1)/2 gives same-size output at stride 1, and K=3 is the smallest odd kernel with a centre pixel — which is where VGG’s obsession comes from. Second, anything with stride 2 halves the map (with one rounding wobble for odd sizes), and halving the map is how every family buys depth without buying compute.
A 3×3 convolution with padding 1 and stride 1 receives a 64 × 64 feature map. What comes out?
LeNet wrote the template in 1998. AlexNet, VGG, Inception and ResNet each added exactly one structural idea on top of it. Learn the five ideas and the last thirty years of vision stop being a list of names.
LeNet-5 (1998) — the template. Two conv-pool blocks, tanh activations, average pooling, three dense layers, 61,706 parameters, trained on a CPU for handwritten digits. The only “concession to modernity” in the source implementation is that we use cross-entropy instead of the original Gaussian connections. Everything the modern world calls a CNN is this network with more layers, bigger channels and better activations.
AlexNet (2012) — ReLU, dropout, and scale. Three changes at once, and the first one is the big one. Tanh saturates: for large positive or negative inputs its derivative falls to almost zero, so gradients fade and training crawls. ReLU is piecewise linear — it passes gradient 1:1 for positive inputs and never saturates — and the paper reports roughly a 6× speedup in training time. Dropout in the fully connected head turned regularisation into a layer instead of a trick. And the network was deep and wide enough — five convs, three dense layers, ≈60M parameters — that it had to be split across two GPUs. The paper’s Figure 2 still shows that split as two parallel streams; that part was a hardware workaround, not an architectural insight. ReLU and dropout are still in every model you use.
VGG (2014) — the 3×3 stack. VGG asked a narrow question: what happens if the only kernel you ever use is 3×3, and you go deep? The answer was 16 or 19 conv layers, 138M parameters, and a block so simple — conv 3×3 → conv 3×3 → pool — that it became the reference point for every architecture that came after. The mathematical observation that powers it (two 3×3s see what one 5×5 sees) is chapter 04.
Inception (2014, same year) — parallel paths. Google’s answer to “which kernel size should I use?” was: all of them, in parallel. One Inception block runs a 1×1, a 3×3, a 5×5 and a 3×3 max pool on the same input and concatenates the results along the channel axis. Each branch specialises — 1×1 for channel mixing, 3×3 for local texture, 5×5 for larger patterns, pooling for shift-invariant features — and the concat lets the next layer pick whichever branch is useful. 1×1 convolutions inside each branch act as bottlenecks to keep the parameter count sane.
ResNet (2015) — the identity skip. One line: y = F(x) + x instead of y = F(x). The + x means a block can always choose to do nothing by driving F to zero, so a 1,000-layer network is at worst as bad as a 1-layer network. With that escape hatch the optimizer is willing to make every block slightly useful — and slightly useful, stacked a hundred times, is state of the art. Chapter 06 is the why; this is the what.
Click through 1998 → 2015. Each family’s block diagram is its real layer sequence — LeNet through ResNet — with Inception’s four parallel branches drawn as rows so they fit a stack. The accent rows are the one new idea that family contributed. Read them in order and the lineage stops being a list of names.
Inception and VGG arrived the same year from different teams — VGG went deeper with one kernel size, Inception went wider with four. Both are in every modern backbone’s family tree.
VGG and Inception both arrived in 2014 and both changed a CNN. What was Inception's new idea?
VGG replaced variety with repetition: one block type — conv 3×3, conv 3×3, pool — stacked 16 or 19 layers deep. The reason it works is arithmetic you can do on your fingers.
Start with the observation the whole family rests on. A single 3×3 convolution gives each output unit a 3×3 view of its input. Stack two of them, and the second layer’s unit looks at 3×3 units of the first layer, each of which looked at 3×3 pixels — so it sees a 5×5 patch of the original input. The arithmetic of the view is:
The view is identical, but the price is not. Compare the weights per channel pair between two stacked 3×3 convs and one 5×5 conv, both mapping C channels to C channels:
That is the entire VGG argument: same receptive field, 28% fewer weights, one extra nonlinearity, and a uniform block you can repeat without thinking. VGG-16 repeats one block type — conv 3×3 ×2 or ×3, then pool — five times, thirteen conv layers in total, then ends with three dense layers. The receptive-field lab below lets you grow the stack one 3×3 at a time and watch both the footprint and the bill.
The bill is the famous part. VGG-16 has 138,357,544 parameters, and almost none of them are in the convolutions:
Keep that 89.4% in mind for chapter 07. ResNet’s head replaces all three dense layers with a global average pool and one linear layer: 513,000 parameters for 1,000 classes instead of 123.6M. That single decision, plus the skip connection, is most of how ResNet-18 matches VGG-16’s accuracy with 12× fewer parameters.
Every unit in layer L sees a (2L+1) × (2L+1) window of the input — that is r = 1 + 2L for a stack of 3×3 stride-1 convolutions. Watch the footprint grow, then compare what the stack costs against one kernel of the same reach.
This is the stride-1 stack the lesson cares about: padding keeps the map size fixed and each extra 3×3 widens the window by exactly two cells. Depths in real backbones reach the whole image — ResNet-18 ’s theoretical field is 435 × 435 on a 224 × 224 input.
The whole lesson compiles to three small classes: LeNet-5 (the template), a VGG block (the repetition), and a ResNet BasicBlock (the skip). Write them once and you can read a decade of architecture papers as variations on your own code.
LeNet-5 first, because every line of it will still be there in ResNet-50. Two Conv2d layers, two average pools, three linear layers, and tanh — the original activation, kept faithful:
import torch
import torch.nn as nn
class LeNet5(nn.Module):
def __init__(self, num_classes=10):
super().__init__()
self.conv1 = nn.Conv2d(1, 6, kernel_size=5)
self.conv2 = nn.Conv2d(6, 16, kernel_size=5)
self.pool = nn.AvgPool2d(2)
self.fc1 = nn.Linear(16 * 5 * 5, 120)
self.fc2 = nn.Linear(120, 84)
self.fc3 = nn.Linear(84, num_classes)
def forward(self, x):
x = self.pool(torch.tanh(self.conv1(x)))
x = self.pool(torch.tanh(self.conv2(x)))
x = torch.flatten(x, 1)
x = torch.tanh(self.fc1(x))
x = torch.tanh(self.fc2(x))
return self.fc3(x)
net = LeNet5()
x = torch.randn(1, 1, 32, 32)
print(f"output: {net(x).shape}")
print(f"params: {sum(p.numel() for p in net.parameters()):,}")
# → output: torch.Size([1, 10])
# → params: 61,706The 61,706 is not a magic constant — it is five multiplications, and you can reproduce every one of them with the calculator lab below:
Now the VGG block. One class, two 3×3 convs with batch norm, a ReLU after each, a max pool at the end — and a tiny model that stacks three of them and calls an adaptive average pool before the final linear layer. That head is the modern move: instead of a giant dense layer on a 7×7×512 map, squash the map to 1×1 and take one number per channel.
import torch.nn as nn
import torch.nn.functional as F
class VGGBlock(nn.Module):
def __init__(self, in_c, out_c):
super().__init__()
self.conv1 = nn.Conv2d(in_c, out_c, kernel_size=3, padding=1)
self.bn1 = nn.BatchNorm2d(out_c)
self.conv2 = nn.Conv2d(out_c, out_c, kernel_size=3, padding=1)
self.bn2 = nn.BatchNorm2d(out_c)
self.pool = nn.MaxPool2d(2)
def forward(self, x):
x = F.relu(self.bn1(self.conv1(x)))
x = F.relu(self.bn2(self.conv2(x)))
return self.pool(x)
class MiniVGG(nn.Module):
def __init__(self, num_classes=10):
super().__init__()
self.stack = nn.Sequential(
VGGBlock(3, 32),
VGGBlock(32, 64),
VGGBlock(64, 128),
)
self.head = nn.Sequential(
nn.AdaptiveAvgPool2d(1),
nn.Flatten(),
nn.Linear(128, num_classes),
)
def forward(self, x):
return self.head(self.stack(x))
# 3 VGG blocks + one linear head = 288,746 params — plenty for CIFAR-10,
# and about 1/480th of the real VGG-16.And the block that changed everything. Read it twice: the arithmetic is three lines of convolution, one batch norm, one addition.
class BasicBlock(nn.Module):
def __init__(self, in_c, out_c, stride=1):
super().__init__()
self.conv1 = nn.Conv2d(in_c, out_c, kernel_size=3, stride=stride, padding=1, bias=False)
self.bn1 = nn.BatchNorm2d(out_c)
self.conv2 = nn.Conv2d(out_c, out_c, kernel_size=3, stride=1, padding=1, bias=False)
self.bn2 = nn.BatchNorm2d(out_c)
if stride != 1 or in_c != out_c:
self.shortcut = nn.Sequential(
nn.Conv2d(in_c, out_c, kernel_size=1, stride=stride, bias=False),
nn.BatchNorm2d(out_c),
)
else:
self.shortcut = nn.Identity()
def forward(self, x):
out = F.relu(self.bn1(self.conv1(x)))
out = self.bn2(self.conv2(out))
out = out + self.shortcut(x) # ← the whole idea
return F.relu(out)Build a network one layer at a time and do the arithmetic the way the papers report it: a conv layer costs C_in × C_out × K² weights (+ one bias per output channel); a dense layer costs in × out. Load LeNet-5 at the bottom and the five layers sum to exactly 61,706.
No layers yet. Configure one on the right and press “add layer” — or load LeNet-5 to see a whole network accounted for.
Batch norm learns one γ and one β per channel, which is why conv layers followed by BN drop their bias: BN’s β already shifts the output. That is the bias=False convention in every ResNet block.
Stack four groups of BasicBlocks — channels double, spatial size halves at the start of each group — and you have a working ResNet for 32×32 inputs. The source’s TinyResNet comes out to 2,797,610 parameters, and the group table is worth reading carefully because it shows a pattern every ResNet shares: cost concentrates in the last group, where the channels are widest.
A BasicBlock with in_c == out_c and stride 1 is exactly two 3×3 convs plus two BN layers: 2 × 9,216 + 2 × 64 = 18,560 parameters at 32 channels. The stride-2 blocks add a 1×1 shortcut convolution (2,048 weights at 32→64) plus its batch norm. Batch norm itself is always 2 × C — one γ, one β per channel — which is why BN never shows up in the parameter budget (0.14% of TinyResNet) even though it is everywhere in the code.
By 2015, VGG-19 worked and VGG-32 did not. Past roughly twenty plain layers, both training and test loss climbed. That is not overfitting — it is the optimizer failing to find useful weights in a very long chain. One line fixed it.
The failure has a precise signature, and it is worth stating because it separates it from every neighbour in the diagnostic table. A 34-layer plain network does not merely generalise worse than its 18-layer twin — it has higher training loss. The deeper network is harder to optimize, even on the data it is looking at. More data does not help, because the data is not the problem. Neither does more regularisation; there is nothing to regularise yet.
The source’s explanation is multiplicative gradients, and the arithmetic is short. In a plain stack the output is y = f_L(f_(L−1)( ... f_1(x) ... )) — layer after layer, each applied to the previous result. The gradient reaching an early layer is a product of one term per layer on the way back, and each term has magnitude roughly (weight magnitude) × (activation gain). Stack them with a gain below 1 and the product collapses:
Batch normalisation, published the same year, kept activations well-scaled and let 19 layers train; but even BN could not rescue depth beyond roughly thirty layers. What was missing was not better scaling but a different topology. He, Zhang, Ren and Sun proposed one change:
The consequence is stronger than “gradients flow better”. A residual block can always choose to do nothing: drive F(x) to zero and the block returns its input unchanged. So a 1,000-layer ResNet is at worst as bad as a 1-layer network, because every extra block has a trivial escape hatch. With that guarantee in place, the optimizer is willing to make each block slightly useful — and slightly useful, stacked a hundred times, is state of the art.
Two block shapes show up everywhere, and after chapter 07’s parameter table you will know how to tell them apart on sight:
The paper’s Figure 1 is the picture to remember: the 34-layer plain network converges to higher training error than the 18-layer plain network, and the 34-layer ResNet beats both. Depth stops being a liability and becomes a dial.
A teaching model of the backward pass: each plain block multiplies the signal by one gain q; a residual block always carries the straight-through path at magnitude 1. Flip the switch and watch the first block’s gradient go from ~1e-5 to about 1.
Simplified model, labelled as one: a real backward pass multiplies full Jacobians and the identity term can cancel. What is not simplified is the shape of the fix — the residual path gives every block a route that does not shrink with depth.
Your 34-layer plain network has worse training loss than your 18-layer plain network. Which explanation fits the evidence?
Before you open the source: name the four parts, predict the shapes, count the parameters. ResNet-18 is the test case — 11.7M parameters, five shape numbers, one skip pattern per block.
A modern backbone is read like a table of contents, in four moves: stem, stages, head, skip pattern. The stem is the first convolution that turns a 3-channel image into base feature width. The stages are repeated blocks where channels double and the spatial map halves. The head turns the final feature map into class scores. The skip pattern tells you whether you are looking at a ResNet (every block) or a plain stack (none).
ResNet-18 in one breath: a 7×7 stride-2 stem (224 → 112), a 3×3 stride-2 max pool (112 → 56), four stages of two BasicBlocks each (56 → 28 → 14 → 7) with channels 64 → 128 → 256 → 512, a global average pool that collapses 512 × 7 × 7 to 512 × 1 × 1, and a single linear layer to 1,000 classes. Now predict the parameters before scrolling:
| group | output map | parameters | share |
|---|---|---|---|
| stem · conv 7×7 3→64 + BN | 112 × 112 (after maxpool: 56 × 56) | 9,536 | 0.1% |
| layer1 · 2 × BasicBlock, 64 channels | 56 × 56 | 147,968 | 1.3% |
| layer2 · 2 × BasicBlock, 64→128, stride 2 | 28 × 28 | 525,568 | 4.5% |
| layer3 · 2 × BasicBlock, 128→256, stride 2 | 14 × 14 | 2,099,712 | 18.0% |
| layer4 · 2 × BasicBlock, 256→512, stride 2 | 7 × 7 | 8,393,728 | 71.8% |
| head · global avg pool + Linear 512→1000 | 1 × 1 → 1,000 logits | 513,000 | 4.4% |
| total | 11,689,512 | 100% |
Two facts fall straight out of that table. First, the cost concentrates in layer4 — 72% of all parameters — because each channel-pair at 512 → 512 costs 512 × 512 × 9 = 2,359,296 weights. If you need a cheaper backbone, cut width in the last stage. Second, the head is cheap: 513,000 parameters, because the classifier is one linear layer on a pooled vector. Compare VGG-16, where the three dense layers hold 123,642,856 of 138,357,544 parameters — 89%. Same task, same dataset, and the residual family spends its budget on features instead of on a fully connected head.
The parameter efficiency is the headline numbers: ResNet-18: 11.7M parameters, 69.8% ImageNet top-1; VGG-16: 138M parameters, 71.6% top-1. Similar accuracy, twelve times fewer parameters. That is why ResNet variants dominated from 2016 until ViT arrived in 2021, and why they still win wherever compute is the constraint.
ResNet-50 scales up with the Bottleneck block instead of stacking more 3×3s: a 1×1 conv shrinks the channel width, the 3×3 works at that reduced width, and a second 1×1 expands back — with the skip around all three. The arithmetic is the reason a 3×3 can afford to sit at 256 channels at all:
One honest caveat about reading receptive fields off a diagram: they are theoretical. Run the arithmetic through ResNet-18 and the last block’s window is 435 × 435 on a 224 × 224 image —
— but the effective receptive field, the region an output actually depends on in a trained network, is much smaller and centre-weighted. Theoretical fields tell you the maximum reach; they do not tell you where the network looks.
import torch.nn as nn
from torchvision.models import resnet18, ResNet18_Weights, vgg16, VGG16_Weights
r18 = resnet18(weights=ResNet18_Weights.IMAGENET1K_V1); r18.eval()
v16 = vgg16(weights=VGG16_Weights.IMAGENET1K_V1); v16.eval()
print(sum(p.numel() for p in r18.parameters())) # 11,689,512
print(sum(p.numel() for p in v16.parameters())) # 138,357,544
print(r18.layer1[0]) # the BasicBlock you just wrote
# transfer learning is always this shape: freeze, swap the head, train it.
for p in r18.parameters():
p.requires_grad = False
r18.fc = nn.Linear(r18.fc.in_features, 10)In ResNet-50's Bottleneck, why is there a 1×1 convolution before the 3×3?
Every group total in the table above is the sum of its layers’ C_in·C_out·K² (plus BN’s 2C and the head’s 512 × 1000 + 1,000): 11,689,512 for ResNet-18 against the source’s quoted 11.7M, and 25,557,032 for ResNet-50 against 25.6M. The counts match torchvision’s 11,689,512 exactly.
Answer before you look. The ReLU question, the 3×3-stack question and the degradation question are the three that separate a memorized timeline from a mechanism you can reason with on a new paper.
0 / 5 answered · 0 correct
01What single architectural idea did AlexNet (2012) introduce that made training deep CNNs practical on GPUs?
02Why did VGG prefer stacks of 3×3 convolutions over a single larger kernel?
03ResNet introduced residual connections as y = F(x) + x. What problem does this solve?
04In a ResNet BasicBlock with in_channels=64, out_channels=128, stride=2, what is the role of the shortcut branch?
05ResNet-18 has ~11.7M parameters and matches or beats VGG-16 (138M params) on ImageNet. What does this imply about VGG?
Click a card to swap the lazy description for what it actually means.
Three problems with exact numbers — count TinyResNet’s 2.8M by hand, rebuild it with Bottleneck blocks, and reproduce He et al.’s Figure 1 by deleting one line. Try first; a worked answer is one click away.
Walk the groups, remembering that BasicBlock with in_c == out_c and stride 1 has no shortcut parameters and that BN is always 2 × C. Stem: 3×3 conv 3→32 is 3 × 32 × 9 = 864 (bias=False because BN follows) plus BN 64 = 928. Layer1, two blocks at 32 channels: each is 9,216 + 64 + 9,216 + 64 = 18,560, so 37,120. Layer2: the stride-2 block adds a 1×1 shortcut, giving 18,432 + 128 + 36,864 + 128 + 2,048 + 128 = 57,728, and the second block is 36,864 + 128 + 36,864 + 128 = 73,984 — group total 131,712. Layer3: 73,728 + 256 + 147,456 + 256 + 8,192 + 256 = 230,144, plus 295,424 for the second block — 525,568. Layer4: 294,912 + 512 + 589,824 + 512 + 32,768 + 512 = 919,040, plus 1,180,672 — 2,099,712. Head: Linear(256, 10) = 2,560 + 10 = 2,570. Total: 928 + 37,120 + 131,712 + 525,568 + 2,099,712 + 2,570 = 2,797,610, matching the printout's ~2.8M. The budget is overwhelmingly in the convolutions (99.8%), and 75.1% of the whole network lives in layer4 alone — the widest stage (128→256) with spatial maps already small. BN is 4,800 parameters (0.17%): 64 + 256 + 640 + 1,280 + 2,560 across the stem and the four stages, the stride-2 shortcuts' BNs included, and the head 2,570 (0.09%). The practical takeaway: if you need to shrink a ResNet-class model, cut width in the last stage, not BN and not the head.
The block mirrors BasicBlock with three convs and a 4× channel expansion: first 1×1 maps in_c → width (C/4), the 3×3 runs at that width, the second 1×1 expands back to out_c, and the skip crosses all three — a 1×1 stride-2 conv whenever in_c != out_c or stride != 1. Cost at equal width C with width = C/4: C × C/4 + 9 × (C/4)² + (C/4) × C = C²/4 + 9C²/16 + C²/4 = 17C²/16 ≈ 1.06C², against 2 × 9C² = 18C² for a BasicBlock at width C — about 17× cheaper per block at the same width. That saving is what lets ResNet-50 run its 3×3s on much wider stages: at 256 channels the plain 3×3 is 589,824 weights and the bottleneck trio is 16,384 + 36,864 + 16,384 = 69,632, 8.47× less. The CIFAR build uses a 3×3 stride-1 stem (no 7×7/maxpool) and four groups of [3, 4, 6, 3] bottlenecks with widths [64, 128, 256, 512] and expansion 4 — which lands near the ImageNet ResNet-50's 25,557,032 parameters minus the stem difference, i.e. roughly 25M against TinyResNet's 2.8M. The fair comparison is not 'bigger or smaller' but 'what the same budget buys': the bottleneck spends its parameters on channels, not on kernel area.
Build two identical stacks and delete exactly one line in one of them — out = out + self.shortcut(x) — so the only difference is the identity path. Add an 18-block plain twin as the control, then run all three with the same schedule, seed and augmentation budget, and plot training loss (not accuracy). The expected shape: plain-34 sits above plain-18 (that is Figure 1's degradation result), while ResNet-34 drops below both. The mechanism is visible in the gradient model the lesson's lab implements: without a skip, the backward signal reaching block i is a product of one gain per layer crossed. At q = 0.75 and 34 blocks that product is 0.75³³ ≈ 7.5 × 10⁻⁵ by the input side — small enough that early layers barely move; with the skip the straight-through path contributes 1 per block, so the ratio stays near 1 instead of a millionth. Two honest caveats. First, the effect is noisy at 10 epochs on CIFAR-10 — compare the training-loss curves over three seeds, not one run's test accuracy. Second, this is the plain-versus-residual comparison, not a claim that batch norm alone cannot train 34 layers: with BN, plain-34 often trains, but it still converges to a higher loss than plain-18, which is exactly the degradation the paper isolated.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 04, Lesson 03) and the Math Foundations Notebook reference build. The five labs (the clickable architecture timeline, the parameter calculator that sums to LeNet-5's 61,706, the receptive-field visualizer, the gradient-highway explorer with the skip switch, and the output-shape calculator) are original to this page, as are the per-layer parameter breakdowns (LeNet-5's five layers, VGG-16's 123.6M classifier, ResNet-18's 11.7M group table, TinyResNet's 2,797,610), the receptive-field arithmetic r = 1 + 2L with the 18C² vs 25C² comparison and ResNet-18's 435 × 435 theoretical field, the bottleneck cost check (69,632 vs 589,824 weights at 256 channels), the degradation arithmetic (0.75³⁰, 0.75¹⁰⁰, 0.25¹⁰), the cheque-reader scenario (25,557,032 vs 61,706 parameters and ~9,800× the arithmetic), and the five-family memory hook. Every number shown is computed live by the labs or verified by hand in the prose.