Lesson 05 — Transfer Learning & Fine-Tuning, visualized

A MILLION GPU HOURS

Training from scratch is a budget.
Transfer is a decision.

ResNet-50 on ImageNet costs roughly 2,000 GPU-hours. Very few teams have that for every task they ship. What almost every team actually ships is a pretrained backbone and a new head trained on a few hundred or a few thousand task-specific images.

ImageNet is 1.28 million training images across 1,000 classes. Training ResNet-50 on it is about 2,000 GPU-hours — and that is one model, one dataset, one afternoon of hyperparameter luck. Put the arithmetic next to a single fine-tune and the lesson writes itself:

ImageNet, ResNet-50 (2,000 × 3,600 s) ÷ 1,280,000 images ≈ 5.6 GPU-seconds per image at $2.50 per GPU-hour 2,000 × $2.50 ≈ $5,000 per full training run a 700-image head-only run 5 epochs on one T4 ≈ 20 minutes ≈ $0.12 the trunk is the expensive part, and you can rent it for free

(Those are teaching numbers with round assumptions — cloud prices and throughput vary by a factor of a few — but the ratio is the point: four orders of magnitude separate a scratch run from a head swap.) The reason this works at all is that the features a CNN learns on ImageNet are not specialised to its 1,000 categories. They are specialised to the statistics of natural images: edges at specific orientations, Gabor-like filters, textures, contrast patterns, shape primitives. Nature has a limited vocabulary of those, and they are stable across almost every visual domain a human can name.

The first 90% of the hierarchy transfers almost unchanged; the last 10% is what you actually train. Teaching schematic — the real boundary between “generic” and “task-specific” is empirical and moves with the domain.

Where the hierarchy specialises is the top. The last stage of an ImageNet CNN is a detector for 1,000 particular categories, and those combinations are mostly irrelevant to your task. That is why the standard move is to cut the top off and train a new head on the borrowed representation — and why the same recipe shows up everywhere: medical imaging, industrial inspection, satellite data, agriculture. The features transfer; the labels do not.

There is one caveat that separates a working transfer from a broken one. “Transfer” is not a single technique: it is a dial between freezing everything and training everything, and the dial has three classic failure modes waiting on it — feature drift (a too-high learning rate destroys the borrowed features), BatchNorm collapse (the running statistics keep normalising for ImageNet), and catastrophic forgetting (the pretrained features are overwritten before the new task is learned). This lesson walks through all three on purpose.

Quick check

Why do the earliest convolution filters of an ImageNet-trained network transfer to X-rays when ImageNet contains no X-rays?

TWO REGIMES

Freeze and train the head,
or thaw the trunk too.

Feature extraction keeps the backbone frozen and trains a new head. Fine-tuning lets the backbone move as well, usually much more slowly. The choice is not taste — it is a function of how much data you have, how far your pixels sit from ImageNet, and how much compute you can spend.

The two regimes are the same model with different amounts of gradient. In feature extraction the trunk is a frozen function: images go in, a 512-dimensional feature vector comes out, and only the new head learns. In fine-tuning every parameter is trainable, but the trunk moves at a much smaller learning rate because the features it holds are worth preserving.

Feature extraction

Trunk frozen — no gradient, no optimizer state for 11.7M params
New head trained at a normal rate (1e-3 with Adam, higher with SGD)
Runs in minutes; the cheapest accurate baseline that exists
Ceiling: the quality of the borrowed features — the probe floor

Fine-tuning

Every parameter trainable, with a small backbone rate — 1e-5 to 1e-4
Late stages move more than early ones (discriminative LRs, next chapter)
Needs enough data that the new task can steer 11.7M weights
Ceiling: usually a few points above the probe — and occasionally much more

The source’s rule of thumb is a table, and it is worth memorising in its original form:

dataset size	domain distance	recipe
< 1k images	close to ImageNet	freeze backbone, train head only
1k–10k	close	freeze the first 2–3 stages, fine-tune the rest
10k–100k	any	fine-tune end-to-end with discriminative LRs
100k+	far	fine-tune everything; consider training from scratch if the domain is far enough

“Close to ImageNet” means natural RGB photos with object-like content. Medical CT scans, overhead satellite imagery and microscopy are far domains: the features still help, but you will need to let more layers adapt. Compute budget is the third input — it does not change what is possible, it changes what you should try first. On four GPUs, the freeze column gets you a shippable baseline in an afternoon; on forty, unfreezing earlier is worth the extra experiments.

How large is the gap between the regimes? On CIFAR-10 with a ResNet-18 trunk, the source’s numbers are worth keeping:

zero-shot linear probe ~70% (a quick head on frozen features) head trained to convergence ~86% (the probe floor) fine-tuned end-to-end, 5 epochs ~93% (the ceiling) an ImageNet backbone with a new linear head already beats 80% on CIFAR-10 without a single gradient reaching the trunk

Two of those three numbers tell you different things. The jump from 70 to 86 is the head learning which borrowed features to weight; the jump from 86 to 93 is the trunk adapting to the task. If your frozen probe is already at 95 and fine-tuning adds nothing, you have a close domain and a small dataset — ship the probe.

The strategy map

Set a dataset size and how far the domain sits from ImageNet, and read the recipe off the map. The four columns are the source’s decision table; the marker is your task; the card below names the backbone state, both learning rates, and how many of the 11.7M parameters are actually updating.

labelled images4,000

size 4,000 images (4.0k) domain close — natural RGB photos with object-like content — same statistics as ImageNet table column 1k–10k recipe Partial fine-tune — freeze the early stages, adapt the late ones backbone frozen: stem + layer1 + layer2 (≈683k params) head lr 1e-3 backbone lr 1e-5 … 1e-4 for layer4/layer3 (discriminative) trainable 10,498,570 / 11,689,512 = 89.81% why · The middle of the network learns object parts; the last stages learn category-like combinations. With a few thousand images there is enough signal to move those, but not enough to safely re-tune the edge and texture detectors. watch · Keep the early stages frozen at least through the first few epochs; unfreeze later if validation has plateaued and still has room.

The two knobs are the only ones the source uses: how much data you have, and how far your pixels sit from ImageNet. Compute budget moves the whole map one column left — with four GPUs you prefer the freeze column; with forty you can afford to unfreeze earlier.

Quick check

You have 500 labelled images for a new task that looks like ImageNet's natural photos. Which regime makes the most sense?

TWENTY LINES TO A BASELINE

Load the backbone.
Cut off the head.

Every torchvision classifier has the same anatomy: a stem, four stages, and a classifier head. You keep everything except the head — and for a first baseline you keep it frozen, so the only thing that learns is the part that speaks your classes.

ResNet-18 is the workhorse of this lesson: 11,689,512 parameters, a 512-dimensional feature vector after global pooling, and a head called fc that maps 512 features to 1,000 ImageNet classes. The other backbones follow the same shape with different names for the head — fc for ResNet, classifier[1] for EfficientNet and MobileNet, heads.head for torchvision ViT. Replacing it is one assignment.

Step 1 · load a pretrained backbone and look insidepython

import torch
import torch.nn as nn
from torchvision.models import resnet18, ResNet18_Weights

backbone = resnet18(weights=ResNet18_Weights.IMAGENET1K_V1)

print("classifier head:", backbone.fc)              # Linear(in_features=512, out_features=1000)
print("feature dim:", backbone.fc.in_features)      # 512
print("params:", sum(p.numel() for p in backbone.parameters()))  # 11,689,512

The ImageNet checkpoint downloads to your torch cache. `ResNet18_Weights.IMAGENET1K_V1` pins the exact weights version, so the numbers in this lesson reproduce.

The head is now a 1,000-way classifier you do not want. The next fifteen lines freeze the trunk, swap the head, and print the two counts that tell you the recipe is what you think it is:

Step 2 · feature extraction in one functionpython

def make_feature_extractor(num_classes=10):
    model = resnet18(weights=ResNet18_Weights.IMAGENET1K_V1)
    for p in model.parameters():
        p.requires_grad = False                          # 1. freeze the trunk
    model.fc = nn.Linear(model.fc.in_features, num_classes)  # 2. new head, trainable by default
    return model

model = make_feature_extractor(num_classes=10)

trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
frozen = sum(p.numel() for p in model.parameters() if not p.requires_grad)
print(f"trainable: {trainable:>10,}")   #      5,130
print(f"frozen:    {frozen:>10,}")      # 11,684,382

Order matters. Freeze first, then attach the new head: a freshly constructed nn.Linear has requires_grad=True, so it survives the freeze. Swap first and freeze after, and you will train exactly nothing.

Where did 5,130 come from? A linear head from d features to C classes is d × C weights plus C biases:

512 features → 10 classes 512 × 10 + 10 = 5,130 (0.044% of 11,689,512) 512 features → 2 classes 512 × 2 + 2 = 1,026 (good / defective) 512 features → 100 classes 512 × 100 + 100 = 51,300 2048 features → 10 classes 2048 × 10 + 10 = 20,490 (ResNet-50, 25,557,032 params) the head is a rounding error in the parameter count — and it is the only part whose weights have never seen an image

Then the loop is the ordinary loop, with one line that matters: the optimizer only needs the parameters that require grad. Passing frozen parameters to it is harmless but wasteful; filtering is clearer and becomes essential when we start unfreezing stages.

Step 3 · train the head (the whole difference)python

optimizer = torch.optim.SGD(
    [p for p in model.parameters() if p.requires_grad],
    lr=1e-3, momentum=0.9, weight_decay=1e-4, nesterov=True,
)

for epoch in range(5):
    model.train()                       # training mode: BN may update, dropout on
    for x, y in train_loader:
        logits = model(x)
        loss = F.cross_entropy(logits, y, label_smoothing=0.1)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

    model.eval()                        # evaluation mode: BN uses running stats
    correct = total = 0
    with torch.no_grad():
        for x, y in val_loader:
            correct += (model(x).argmax(-1) == y).sum().item()
            total += y.numel()
    print(f"epoch {epoch}  val {correct / total:.3f}")

Training only the head to convergence plateaus around 86% — the probe floor. Five epochs of the same loop with the trunk unfrozen reaches ~93%.

That is a working baseline, and it is often good enough to ship. The second regime — letting the trunk move too — is where the interesting failures live, and it is worth seeing them before you run them:

Three fine-tuning runs

The same 10-class task, the same new head, three backbone treatments. The dashed line is the frozen probe floor; the thin green line is a healthy progressive schedule; the thick line is your run as you move the backbone learning rate and the unfreeze epoch. Watch what happens when the trunk unfreezes before the head has stabilised.

backbone learning rate1.00e-4

unfreeze epoch15

backbone lr 1.00e-4 head lr 1e-3 (fixed) unfreeze epoch 15 final loss 0.244 floor 0.241 quality q 98.4% adaptation a 84% low point 0.244 climbed back — verdict · healthy fine-tune — adaptation without drift quality ended at 98% and the floor fell to 0.24 below the 0.35 probe floor frozen probe floor 0.35 · adapted floor 0.22 · chance ln 10 = 2.303

A frozen head plateaus at the probe floor. A healthy fine-tune spends its backbone budget after the head has stabilised, so the error signal that reaches the trunk is small. A too-high backbone LR spends it early — 100× the healthy rate — and the features the head just learned to read are overwritten underneath it.

TRAP · THE FREEZE THAT ATE THE HEAD

The most common silent bug in this chapter is a one-line ordering mistake:

model.fc = nn.Linear(model.fc.in_features, 10)   # new head
for p in model.parameters():
    p.requires_grad = False                      # ← freezes the new head too
# trainable: 0

The loss will sit almost still, the run will look stable, and the validation accuracy will be chance forever. Print sum(p.numel() for p in model.parameters() if p.requires_grad) every time you write a freeze loop. 5,130 means what you think it means; 0 means you froze your own head.

Quick check

You freeze the whole backbone and replace model.fc with a new nn.Linear. You then build an SGD optimizer over model.parameters() — every parameter, frozen or not. What happens to the trunk's weights when you train?

ONE MODEL, FIVE RATES

Early layers should move slower.
Much slower.

When you unfreeze the trunk, a single learning rate forces a compromise: high enough for the new task, low enough not to wreck the trunk — and it cannot be both. Discriminative learning rates give each stage its own rate, large at the head, tiny at the stem.

The reasoning is the hierarchy from chapter 01. The stem encodes edges and orientations that are right for any natural image; you want to preserve them. Layer4 encodes ImageNet-specific combinations that are mostly wrong for your task; it needs to move a lot. One learning rate for both ends means the early layers get battered or the late layers crawl. In PyTorch this is not new machinery — it is a list of parameter groups passed to the optimizer.

the source's recipe (stage relative to base_lr = the head's rate) stem + first group base_lr / 100 mostly fixed stage 1 base_lr / 10 stage 2 base_lr / 3 last backbone group base_lr new head base_lr (or slightly higher) with the code's decay of 0.3, each stage is 0.3× the one above: head 1e-3 · layer4 3e-4 · layer3 9e-5 · layer2 2.7e-5 · layer1 8.1e-6 · stem 2.43e-6

Those stem numbers look absurd — the stem learns at 0.24% of the head’s rate — and the source says exactly that: “extreme sounding; empirically it works.” The intuition is that the stem does not need to learn anything new; it needs to not forget. The practical band people ship is head 1e-3 and backbone 1e-5 to 1e-4; the exact decay is a hyperparameter, the ordering is not.

Discriminative parameter groups (the source's helper)python

def discriminative_param_groups(model, base_lr=1e-3, decay=0.3):
    stages = [
        ["conv1", "bn1"],   # stem
        ["layer1"],
        ["layer2"],
        ["layer3"],
        ["layer4"],
        ["fc"],             # the new head
    ]
    groups = []
    for i, names in enumerate(stages):
        lr = base_lr * (decay ** (len(stages) - 1 - i))
        params = [p for n, p in model.named_parameters()
                  if any(n.startswith(k) for k in names) and p.requires_grad]
        if params:
            groups.append({"params": params, "lr": lr, "name": "_".join(names)})
    return groups

model = resnet18(weights=ResNet18_Weights.IMAGENET1K_V1)
model.fc = nn.Linear(model.fc.in_features, 10)
groups = discriminative_param_groups(model)
print([(g["name"], g["lr"]) for g in groups])
# [('conv1_bn1', 2.43e-06), ('layer1', 8.1e-06), ('layer2', 2.7e-05),
#  ('layer3', 9e-05), ('layer4', 0.0003), ('fc', 0.001)]

One model, five learning rates, zero extra code: `SGD(groups, momentum=0.9, weight_decay=1e-4, nesterov=True)`. The `requires_grad` filter is what makes the same helper work for progressive unfreezing.

Modern transformer fine-tunes replace the stage groups with a smooth per-layer gradient. Instead of five rates, every layer gets a slightly smaller one than the layer above it:

lr(layer k) = base_lr × decay^(L − k) L = 12 blocks, decay = 0.75, head at 1e-3: block 11 (nearest the head) 1e-3 × 0.75^0 = 1.0e-3 block 0 (nearest the input) 1e-3 × 0.75^11 ≈ 4.2e-5 0.75^11 ≈ 0.042 — the first block learns at 4.2% of the head's rate

For CNNs the stage-grouped version is usually enough; for ViT/BEiT fine-tunes the smooth version is standard because all blocks have the same shape, so the natural grouping is “one group per block.” Either way, the rule is the same: the further from the task you get, the smaller the update.

The second half of the recipe is progressive unfreezing: start with only the head trainable, then unfreeze one stage per epoch from the end toward the beginning. It costs a few extra epochs and it buys a safety property — by the time the stem gets any gradient, the head has already stabilised, so the error signal reaching the stem is small.

The unfreezing scheduler

Read the timeline like a train schedule: each row is a block of ResNet-18, each column an epoch. A coloured cell means that block is receiving gradient; the outline marks the epoch it unfroze; the number on the right is its learning rate. The head always goes first and always trains fastest — early blocks join last and move least.

base learning rate (head)1.0e-3

stage decay0.30×

epoch2 / 6

epoch 2 / 6 schedule progressive (head → stem, one per epoch) trainable 10,498,570 / 11,181,642 = 93.9% fc (new head) live 1.0e-3 unfreezes ep 0 layer4 live 3.00e-4 unfreezes ep 1 layer3 live 9.00e-5 unfreezes ep 2 layer2 frozen 2.70e-5 unfreezes ep 3 layer1 frozen 8.10e-6 unfreezes ep 4 stem (conv1 + bn1) frozen 2.43e-6 unfreezes ep 5 note · the trainable set changes at every unfreeze — rebuild the optimizer (a new parameter group list), or the frozen stages' cached moments keep drifting

The recipe is one sentence: the head learns fastest, and each stage below it learns slower. The default decay of 0.3 gives the stem 0.3⁵ ≈ 0.0024× the head’s rate — 2.4e-6 when the head is at 1e-3. It sounds extreme; the source reports it works.

THE BUFFERS YOU FORGOT

BatchNorm remembers ImageNet.
Your data is not ImageNet.

Every BatchNorm layer holds two running buffers — running_mean and running_var — computed over the batches it saw during pretraining. They are not parameters, they receive no gradient, and if your pixel distribution is different, they are quietly wrong.

BatchNorm normalises each channel using a mean and variance. During training it computes those statistics from the current batch, then nudges two buffers toward them with a momentum:

running_mean ← (1 − momentum) × running_mean + momentum × batch_mean running_var ← (1 − momentum) × running_var + momentum × batch_var momentum = 0.1 (the PyTorch default) one batch moves the buffer 10% of the way to the batch's statistic 20 batches close 1 − 0.9^20 = 87.8% of the gap — if every batch looks alike a buffer starting at 0.45 watching batches centred at 2.0: step 0 0.450 step 5 1.085 step 1 0.605 step 10 1.460 step 20 1.812 ← still 0.19 short of 2.0

At evaluation time BatchNorm stops looking at the batch and uses the buffers. That is the whole point — a single image has no batch statistics — and it is also where the silent failure lives. The buffers encode the mean and variance of ImageNet’s RGB photos. If your task is grayscale CT scans, thermal imagery, or a different sensor, the normalisation is subtract-the-wrong-mean, divide-by-the-wrong-std, and every later layer receives a systematically shifted signal. The source’s estimate of the damage is 5–15% accuracy, silently.

There are three fixes, in order of preference:

Fine-tune with BN in training mode — let the buffers adapt along with the weights. The default when the task dataset is medium-sized (roughly 5k examples or more), because the moving average of a small batch is noise.
Freeze BN statistics in eval mode — keep ImageNet’s statistics, train only the weights. The right call for small datasets: a head trained on 500 images cannot fix an RGB mean anyway.
Replace BatchNorm with GroupNorm — removes the moving-average problem entirely. Standard in detection and segmentation backbones where the per-GPU batch is tiny.

Freeze BN statistics without freezing BN's weightspython

def freeze_bn_stats(model):
    for m in model.modules():
        if isinstance(m, (nn.BatchNorm1d, nn.BatchNorm2d, nn.BatchNorm3d)):
            m.eval()                            # use the running buffers
            for p in m.parameters():
                p.requires_grad = False         # and do not train gamma/beta
    return model

for epoch in range(epochs):
    model.train()               # flips everything to training mode...
    if freeze_bn:
        freeze_bn_stats(model)  # ...then reverses it for BN only
    ...

Call it after model.train(), every epoch. model.train() walks the whole module tree and puts every BN layer back into training mode — one call, and your carefully frozen statistics start moving again.

The lab below shows the two statistics side by side on a tiny batch from a hypothetical new domain: what the layer uses in train mode, what it uses in eval mode, and how far off zero the normalized activations land. Its defaults — a +2.0σ domain gap, 0.1 momentum, three batches fed through — leave the ImageNet buffers far enough behind that eval mode normalizes with a +1.52σ offset. Watch how many batches it takes before the buffer walk is even halfway to the new domain — and note that the healthy answer is often “do not let it walk at all.”

BatchNorm: running stats versus batch stats

A tiny BN layer sees eight activations from your new domain (slider: the domain gap), while its running buffers still remember ImageNet. Feed it batches and watch the buffers walk toward the new mean. Then flip to eval mode and see which statistics the layer actually uses — and how far off zero the normalized activations land.

8 ACTIVATIONS · RAW → NORMALIZED

quantity	batch (this domain)	running buffers (ImageNet)
mean	2.000	0.542
variance	0.727	0.926
raw values	0.6 · 1.1 · 1.5 · 1.8 · 2.1 · 2.5 · 2.9 · 3.3

Running mean per batch step — the walk toward 2.00.

domain gap (batch mean shift)+2.0σ

BN momentum0.10

batches fed through3

mode eval — running buffers, no update stats used mean 0.542 variance 0.926 batch stats mean 2.000 variance 0.727 buffers mean 0.542 variance 0.926 normalized mean +1.515σ offset +1.515σ steps 3 · 22 steps to close 90% of the gap batchnorm update · running = 0.90·running + 0.10·batch

In train mode the layer never uses the buffers — it normalizes with the batch it just saw and moves the buffers 10% of the way. In eval mode it uses the buffers only. When the two disagree, eval sees the domain gap as a constant offset (here +1.52σ) — which is exactly the silent 5–15% accuracy loss the source warns about.

Quick check

You fine-tune a ResNet on 800 grayscale medical images (replicated to 3 channels) and accuracy sits at 10% — chance for 10 classes. What is the most likely cause?

THREE WAYS TO BREAK IT

The transfer bugs do not crash.
They cost you points.

Freezing too much starves the model of information; unfreezing too fast destroys the features; BatchNorm carries the wrong statistics. Each one has a signature in the numbers, and each one has a fix that is one line of configuration.

The source lists the three classics and their first fixes. Read the middle column until you can name the symptom from memory — the whole skill of transfer learning is recognising which of these you are looking at:

failure	what you see	the fix
Feature drift	val loss turns upward right after you unfreeze the trunk, then parks above the frozen probe	backbone LR down to 1e-5–1e-4 with discriminative groups; unfreeze later; one stage at a time
BN collapse	train loss looks fine, eval accuracy is chance or wobbles; worst on tiny, shifted datasets	freeze BN in eval mode after model.train(); or GroupNorm; or a BN warmup pass on the target data
Catastrophic forgetting	new task accuracy climbs while the pretrained features are overwritten — the model gets worse at everything else	lower the backbone LR, freeze more, shorten the schedule; keep the frozen probe as a floor and early-stop when it is beaten

Notice what all three have in common: the trunk was moving faster than the task needed, or moving at all when it should not have. That is why the two-number ritual matters. Before you fine-tune, train the frozen probe and write its accuracy down. After you fine-tune, print both numbers:

pretrained-only accuracy the head on a frozen trunk — your floor fine-tuned accuracy the same model after end-to-end training — your ceiling fine-tuned < pretrained-only ⇒ a learning-rate or BatchNorm bug, not "transfer doesn't work here". The probe is a special case of fine-tuning with backbone LR = 0, so the ceiling cannot legitimately sit below the floor.

That inequality is the single most valuable diagnostic in this lesson. A team that prints both numbers gets a 15-minute debugging session; a team that prints only the fine-tuned number ships a worse model than the baseline they already had. Everything else in this chapter — drift, BN, forgetting — is a way of producing that inversion.

The fine-tuning triage board

Six transfer-learning failures, six symptoms. Pick the one that looks like your run: the board names the most likely cause, the first fix, the number that proves it, and the reflex that wastes an afternoon.

SYMPTOM → CAUSE → FIRST FIX

FEATURE DRIFTValidation loss spikes the moment I unfreeze the backbone

What you see · The frozen-head epochs are fine — val loss 1.61 and falling. The epoch after the unfreeze it jumps to 1.85 and never returns to its old level.

Most likely cause · The backbone learning rate is far above the healthy band (1e-5 … 1e-4). The first trunk steps are enormous while the head is warm but not converged, so the features the head learned to read are overwritten — feature drift, and the error signal accelerates it.

First fix · Use discriminative learning rates (head 1e-3, backbone 1e-5 … 1e-4), unfreeze after the head has stabilised, unfreeze one stage per epoch, and print the frozen probe floor beside every run so a regression is visible immediately.

lab curve · healthy 1e-4 ends at 0.24; the same schedule at 1e-2 dips to 0.28, then climbs to 1.82 — 5.2× the frozen probe floor.

Reflex to resist · Training longer. The curve is not slow, it is being actively made worse by every trunk step.

symptom 1 / 6 tag feature drift the transfer debugging order 1. print the frozen probe accuracy — your floor 2. print the fine-tuned accuracy — your ceiling 3. if ceiling < floor: LR, BN, optimizer — in that order 4. if the val curve spiked at an unfreeze: backbone LR 5. if train is fine and eval collapses: BN buffers 6. if nothing moved: check requires_grad and the optimizer's param list

Every cause on this board is checkable with one number: a trainable parameter count, a probe accuracy, a BN offset. “Loss is bad” is compatible with all six — the number narrows it to one.

For most real tasks you do not have to write any of the machinery in this lesson. torchvision.models plus three lines gives you a fine-tune; timm gives you roughly 800 pretrained vision backbones with consistent defaults; transformers.AutoModelForImageClassification.from_pretrained(name, num_labels=N) does the same for ViT-family models. The heavier machinery — discriminative groups, progressive unfreezing, frozen BN — is what you reach for when the default run hits one of the three failures above.

The source’s “ship it” artifacts are worth stealing as habits: a fine-tune planner that picks feature-extraction vs progressive vs end-to-end from dataset size, domain distance and compute budget; and a freeze inspector that, given a PyTorch model, reports which parameters are trainable, which BatchNorm layers are in eval mode, and whether the optimizer is actually being fed the trainable parameters. The inspector is twenty lines and it catches the two bugs that produce every silent failure in this chapter.

CHECK YOURSELF

Five questions.
Then the terms worth keeping.

Answer before you look. The BatchNorm question and the probe-versus-fine-tune question are the two that separate “I followed the recipe” from “I can debug the run at 2 a.m.”

0 / 5 answered · 0 correct

01You have 500 labelled images for a new task close to the ImageNet distribution. Which regime makes the most sense?

02Why do early conv layers of an ImageNet-pretrained network transfer to medical images, even though ImageNet contains no X-rays?

03When fine-tuning end-to-end with discriminative learning rates, why should early layers get a smaller rate than late layers?

04You fine-tune a ResNet on a 10-class medical dataset of 800 grayscale images (replicated to 3 channels) and accuracy is 10% — chance for 10 classes. What is the most likely cause?

05You compare two runs: (a) a linear probe on the frozen ImageNet backbone reaches 82%, (b) an end-to-end fine-tune reaches 78%. What should you conclude?

Key terms, demystified

Click a card to swap the lazy description for what it actually means.

Exercises from the lesson

Three problems with exact numbers: race the linear probe against a full fine-tune and read both gaps, sabotage the backbone learning rate on purpose and recover with discriminative groups, and run the three-regime experiment on a medical dataset. Try first; a worked answer is one click away.

Train a ResNet-18 as a linear probe (backbone frozen) and as a full fine-tune on the same synthetic-CIFAR dataset. Report both accuracies side by side. Explain which gap tells you the features transfer well and which tells you they do not.
Show one worked answer
Run both with the same data split and the same head, differing only in requires_grad and the backbone rate. Expected numbers on the source's CIFAR-10 setup: the probe converges to ~86% (the floor), the fine-tune to ~93% after 5 epochs. Three readings. First, the probe's absolute accuracy is the feature-transfer signal: 86% with a 5,130-parameter head means the ImageNet features already separate most CIFAR classes, so the domain is close and the early layers are doing real work. Second, the probe-to-fine-tune gap is the adaptation signal: ~7 points is what the trunk still needed to move; a gap near zero would mean the frozen features are already task-perfect and fine-tuning is optional. Third, a fine-tune that ends below the probe is not a data property — it is a bug (backbone LR too high, BN mishandled): the probe is fine-tuning with backbone LR = 0, so the ceiling cannot sit below the floor. Watch the epoch count: with a tiny dataset (say 2k images) the fine-tune overfits before it beats the probe, and the honest conclusion is 'probe wins at this size.'
Introduce a bug on purpose: set base_lr = 1e-1 on the backbone stage instead of the head. Show the training loss explode, then recover by applying the discriminative_param_groups helper. Record the learning rate at which each stage starts diverging.
Show one worked answer
At base_lr = 1e-1 the backbone's first steps are ~1,000× the healthy 1e-4 rate, so the trunk moves far enough in one update to invalidate the features the head is reading: expect the loss to dip for a few steps, then climb by an order of magnitude or more, with validation accuracy falling to chance. The lesson's teaching model reproduces the shape — the 1e-2 curve dips to 0.28 and parks at 1.82, above the frozen probe's 0.35 — and 1e-1 collapses almost immediately. Recovery is one function: discriminative_param_groups(model, base_lr=1e-3, decay=0.3) hands the optimizer head 1e-3, layer4 3e-4, layer3 9e-5, layer2 2.7e-5, layer1 8.1e-6, stem 2.43e-6. Stage thresholds worth recording: the head tolerates 1e-1 with SGD (the source's own probe uses 3e-2), the last backbone stage starts misbehaving around 1e-2 (100× the 1e-4 band), and the stem diverges at anything above ~1e-3. The permanent lesson is that the trunk's safe band is ~1e-5–1e-4 regardless of what the head can survive — they need different rates, not one compromise.
Take a medical imaging dataset (CheXpert-small, PatchCamelyon, or HAM10000) and compare three regimes: (a) ImageNet-pretrained frozen backbone + linear head; (b) ImageNet-pretrained fine-tune end-to-end; (c) scratch training. Report accuracy and compute cost for each. At what dataset size does scratch training become competitive?
Show one worked answer
A workable protocol: hold out a fixed test split, run (a) 1–5 epochs of head-only training, (b) 5–15 epochs end-to-end with discriminative rates and a frozen-BN choice stated explicitly, (c) 100+ epochs from scratch with augmentation and the same schedule family, and log accuracy, GPU-hours and the trainable parameter count for each — 5,130 for (a), 11,689,512 for (b) and (c). Expectation from the published medical-imaging literature: ImageNet-pretrained models beat scratch by large margins on datasets in the hundreds-to-tens-of-thousands of images, because natural-image edges and textures transfer even to grayscale modalities; the gap narrows as the training set grows into the hundreds of thousands. The pretrained fine-tune usually edges the probe by a few points on such far domains — more than on CIFAR, because the domain gap gives the trunk real work to do — but it is also where BN handling decides the run: with small batches, freeze BN or switch to GroupNorm. Where scratch becomes competitive is the honest answer, not a fixed number: the source's table puts 'consider training from scratch' at 100k+ images on a far domain, and the only way to know for your dataset is the three-way experiment. Compute honesty: (a) is minutes, (b) is hours, (c) is days — so run them in that order and stop when the accuracy is shippable.

Terms this lesson borrows from later lessons (or outside)

You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.

backpropagation (Phase 3, Lesson 03) — The reverse-mode chain rule that decides which parameters receive gradient. `requires_grad = False` cuts a parameter out of that graph — that is the entire mechanism behind freezing.
optimizers and parameter groups (Phase 3, Lesson 06) — SGD with momentum (and AdamW) accept a list of parameter groups, each with its own learning rate. Discriminative learning rates are just that list, built by stage.
learning-rate schedules (Phase 3, Lesson 09) — Cosine annealing, warmup and their stability limits. A fine-tune usually runs 5–15 epochs with a cosine decay — much shorter than a from-scratch schedule, because the trunk starts close to a good place.
regularization (Phase 3, Lesson 07) — Weight decay, dropout and label smoothing, all standard in the fine-tune loop this lesson shows; they matter more here because the task dataset is small.
PyTorch modules and tensors (Phase 3, Lesson 11) — Sequential containers, nn.Linear, named_parameters() and state_dict() — the API surface every head swap uses.
CNNs and ImageNet features (Phase 4, Lessons 03–04) — Convolution, pooling, receptive fields and the ResNet lineage. This lesson takes the feature hierarchy those lessons build and asks which parts to keep.
LoRA and parameter-efficient fine-tuning (Phase 11, Lesson 08) — The same head-swap arithmetic scaled to language models: freeze the base weights, train a small low-rank add-on, and pay attention to learning rates exactly as here.

KEEP GOING

A picture is a start.
Practice is the rest.

This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.

Original lessonTransfer Learning & Fine-TuningAI Engineering from Scratch · the source text, quiz, and main.py with the feature extractor, the discriminative parameter groups, the BN-freezing helper and the minimal fine-tuning loop.Canonical tutorialTransfer Learning for Computer Vision TutorialPyTorch · the official walk-through of exactly this recipe: load a pretrained convnet, freeze the trunk, replace the classifier, and fine-tune the head — with the data-augmentation and scheduler details a real run needs.Original paperDeep Residual Learning for Image RecognitionHe et al. (2015) · ResNet: the residual blocks that make 11.7M-parameter trunks trainable on 1.28M images, and the 1000-class ImageNet benchmark every backbone in this lesson was pretrained on.Original paperHow transferable are features in deep neural networks?Yosinski et al. (2014) · the paper that quantified transferability layer by layer: early layers transfer across tasks, late layers specialise, and fine-tuning them together recovers most of the loss.Original paperBatch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate ShiftIoffe & Szegedy (2015) · introduces the running statistics this lesson's fifth chapter spends its time on: what BatchNorm stores, how momentum updates it, and why eval mode behaves differently from train mode.

Lesson text adapted from AI Engineering from Scratch (Phase 04, Lesson 05) and the Math Foundations Notebook reference build. The five labs (the strategy map, the three-run loss-curve comparison, the progressive unfreezing scheduler, the BatchNorm train/eval statistics board, and the six-symptom triage board) are original to this page, as are the GPU-hour arithmetic (1.28M images ÷ 2,000 GPU-hours ≈ 5.6 GPU-seconds per image; ≈ $5,000 per ImageNet run versus ≈ $0.12 for a 700-image head-only run), the head-parameter arithmetic (512→10 = 5,130 = 0.044% of 11,689,512; 512→2 = 1,026; 2048→10 = 20,490), the discriminative-LR ladder with decay 0.3 and the transformer layer-wise 0.75¹¹ ≈ 0.042, the BatchNorm momentum walk (10% per batch, 87.8% after 20, 0.45 → 1.812), the probe-floor/ceiling ritual, and the memory hook “move the trunk last, move it least.” The loss-curve and hero dynamics are a labelled teaching model of fine-tuning (feature quality, adaptation and a head that converges to their floor), because a full training run cannot be replayed in a canvas; every other number shown is computed live by the labs or verified by hand in the prose.

Somebody else trained the trunk.
You train the head.

Somebody already paid for the first 90%.

Pick the regime from data size, not taste.

Drift, BN collapse, forgetting.

Training from scratch is a budget.
Transfer is a decision.

Freeze and train the head,
or thaw the trunk too.

Feature extraction

Fine-tuning

The strategy map

Load the backbone.
Cut off the head.

Three fine-tuning runs

Early layers should move slower.
Much slower.

The unfreezing scheduler

BatchNorm remembers ImageNet.
Your data is not ImageNet.

BatchNorm: running stats versus batch stats

The transfer bugs do not crash.
They cost you points.

The fine-tuning triage board

Five questions.
Then the terms worth keeping.

Key terms, demystified

Exercises from the lesson

A picture is a start.
Practice is the rest.

Somebody else trained the trunk.You train the head.

Somebody already paid for the first 90%.

Pick the regime from data size, not taste.

Drift, BN collapse, forgetting.

Training from scratch is a budget.Transfer is a decision.

Freeze and train the head,or thaw the trunk too.

Feature extraction

Fine-tuning

The strategy map

Load the backbone.Cut off the head.

Three fine-tuning runs

Early layers should move slower.Much slower.

The unfreezing scheduler

BatchNorm remembers ImageNet.Your data is not ImageNet.

BatchNorm: running stats versus batch stats

The transfer bugs do not crash.They cost you points.

The fine-tuning triage board

Five questions.Then the terms worth keeping.

Key terms, demystified

Exercises from the lesson

A picture is a start.Practice is the rest.

Somebody else trained the trunk.
You train the head.

Training from scratch is a budget.
Transfer is a decision.

Freeze and train the head,
or thaw the trunk too.

Load the backbone.
Cut off the head.

Early layers should move slower.
Much slower.

BatchNorm remembers ImageNet.
Your data is not ImageNet.

The transfer bugs do not crash.
They cost you points.

Five questions.
Then the terms worth keeping.

A picture is a start.
Practice is the rest.