ImageNet-1k’s 1.28M labels cost around $10M. Self-supervised vision learns the features from the unlabelled pile instead: SimCLR pulls two views together, DINO distils an EMA teacher into its student, MAE hides 75% of the patches and paints them back. Seven chapters and seven labs take the three families apart — the InfoNCE loss, the batch-as-negatives arithmetic, centring and sharpening, the 75%-vs-15% mask argument, and the linear probe that puts the frozen features to work on your 10k labels.
SimCLR pulls two augmented views of one image together and pushes every other image away. DINO trains a student to predict an EMA teacher's output — no negatives at all. MAE hides 75% of patches and reconstructs the pixels. All three are trained on unlabelled images and produce an encoder you freeze or fine-tune.
frozen linear probes: SimCLR ≈71 · DINO ≈77 · MAE ≈76 · DINOv2 ≈86InfoNCE asks one view to find its partner among all 2N − 1 candidates in the batch: batch 32 gives 62 negatives per positive, 512 gives 1,022, 4096 gives 8,190. More negatives, sharper signal. A constant encoder scores exactly ln(2N − 1) — 4.14 at batch 32, 9.01 at 4096 — which is why the loss is read against that floor.
L = −log( exp(sim(zᵢ,zⱼ)/τ) / Σₖ exp(sim(zᵢ,zₖ)/τ) ) · τ = 0.07–0.5Contrastive: the negatives. DINO: centring (subtract the per-dim EMA mean) plus sharpening (teacher temperature 0.04 → 0.07). MAE: the 75% mask itself — at 15% a blur solves the task, which is why text can mask 15% and images cannot. Augmentations define the invariances: crop and colour jitter say what the encoder must ignore.
196 patches → 147 masked · 49 visible · 50 encoder tokens · 112,896 pixels scoredSelf-supervised pretraining is a made-up task whose only non-trivial solution is real visual features — pull two views together (SimCLR), predict a lagged copy of yourself (DINO), or paint the 75% of patches you were not allowed to see (MAE). The labels arrive only at the end, on a head with a few thousand parameters.
By the end you will be able to write InfoNCE from memory and say why batch 32 starves it, compute the negative count and the ln(2N − 1) floor for any batch, explain centring and sharpening as two halves of one anti-collapse mechanism, justify MAE’s 75% against BERT’s 15% with the entropy argument, and choose between a frozen DINOv2 probe and an MAE fine-tune for your own label budget.
ImageNet-1k is 1.28M labelled images, and that label set is estimated to have cost around $10M. Self-supervised pretraining throws the labels out of the expensive part: learn features from unlabelled pictures first, then fine-tune on whatever small labelled set you can afford.
Supervised ImageNet has 1.28M labelled images; at the commonly quoted annotation estimate that works out to roughly 1.28M labelled images for about $10M, or $7.81 per image. Medical imaging and industrial inspection datasets are tinier and cost more per label, because a radiologist or a specialist has to look at every one. Meanwhile the world is drowning in pictures nobody labelled: YouTube frames, web crawls, factory webcam footage, satellite sweeps. The pretraining corpora are named for their scale: JFT-300M (300M images), LAION-5B (5B image–text pairs), and DINOv2’s 142M-image curated set — three to four orders of magnitude more pictures than any label budget reaches. The question that reshaped vision from 2018 onward: can a model learn visual features from the cheap data, and only use labels at the end?
The conceptual trick is that the pretext task — the thing the model is trained to do — does not have to be the downstream task. Predicting the colour of a grayscale image, classifying the rotation someone applied, masking patches and reconstructing them: all were tried. What matters is not whether the task is useful, but whether solving it requires useful features. Three families scaled:
All three are trained on unlabelled images and produce an encoder you then freeze (for a linear probe) or fine-tune (for real). The supervised recipe — collect labels, train end to end — is not replaced; it is moved to the end of the pipeline, where a 10k-image labelled set is enough.
The anchor numbers for this lesson, all on ImageNet-1k with a frozen encoder and one linear classifier on top: SimCLR (ResNet-50) ≈71% in 2020, DINO (ViT-S/16) ≈77% in 2021, MAE (ViT-L/16) ≈76% in 2022, DINOv2 (ViT-g/14) ≈86% in 2023. The last one is a purely self-supervised model whose frozen features rival fully supervised ones — trained on 142M curated images and zero labels.
Pick a family to read its objective, its negatives, its collapse guard and its compute; then flip the guard off and watch what the family would collapse into without it. The green strip is the piece that keeps the representation alive.
The three families are three answers to one failure mode: a model that maps everything to the same vector is useless. Contrastive learning keeps representations apart with negatives; DINO centres and sharpens the teacher; MAE makes the task hard enough that only real features solve it.
Why is it fine for the pretraining task (reconstruct patches, match views) to be completely different from the downstream task (classify flowers)?
SimCLR’s idea fits in one sentence: take an image, augment it twice, and train the encoder so the two views land close together while every other image in the batch stays far away. No labels, no decoder — just a softmax that must pick the right partner.
The pipeline is four steps. Sample an image. Apply two independent random augmentations (two views). Push both through the same encoder and a small projection head. Minimise a loss that says “these two embeddings are the same picture” and “these two embeddings are different pictures”. The encoder and projection head are trained; the projection head is thrown away afterwards and only the encoder is kept.
The loss is InfoNCE. For an anchor view z_i with positive partner z_j, it is a softmax over cosine similarities divided by a temperature τ, where the positive is the target class and every other view is a distractor:
Read it as a multiple-choice exam: the anchor asks “which of these 2N − 1 views is my partner?”, the softmax gives a probability to each candidate, and the loss is the negative log of the probability it assigned to the right answer. A perfect encoder puts all probability on the positive and scores 0. A constant encoder spreads probability evenly and scores ln(2N − 1) — the exam with all answers equally plausible.
Take a microscope-sized batch: 2 images, so 4 views. Suppose the encoder says the positive pair has cosine similarity 0.8, and the two other pairs sit at 0.2. With τ = 0.5 the logits are 1.6, 0.4 and 0.4:
Two things to carry forward: a smaller τ makes the softmax sharper, so a clean positive is rewarded almost perfectly; and the same sharpness makes any negative that looks like the positive expensive. That is what “hard negatives” means, and it is why the number of negatives has to be large — see the next chapter.
import torch
import torch.nn.functional as F
def info_nce(z1, z2, tau=0.1):
"""z1, z2: (N, D) L2-normalised embeddings of paired views."""
N, D = z1.shape
z = torch.cat([z1, z2], dim=0) # (2N, D): view 1s then view 2s
sim = z @ z.T / tau # (2N, 2N) cosine similarity / tau
mask = torch.eye(2 * N, dtype=torch.bool, device=z.device)
sim = sim.masked_fill(mask, float("-inf")) # a view is not its own answer
targets = torch.cat([torch.arange(N, 2 * N), torch.arange(0, N)]).to(z.device)
return F.cross_entropy(sim, targets) # softmax with the positive as target
# sanity check from the source:
z1 = F.normalize(torch.randn(16, 32), dim=-1)
z2 = z1.clone()
print(info_nce(z1, z2, tau=0.1).item()) # identical pairs → near 0
z_random = F.normalize(torch.randn(16, 32), dim=-1)
print(info_nce(z1, z_random, tau=0.1).item()) # random pairs → ≈4.75 at τ=0.1
# ln(2N−1) = ln(31) ≈ 3.43 is the constant-encoder floor: identical
# embeddings make every candidate equally plausible, while random ones
# sit above it because their similarities are spread out.A batch of embeddings, two views each. The loss is a softmax: pick the positive out of all 2N − 1 candidates. Move the temperature, the batch size or how similar the positives are, and watch the loss and the similarity matrix react.
This is the real loss, computed on toy embeddings — not a trained network. The mechanism is the point: the batch is the label, and the denominator is where the negatives live.
In supervised learning a batch is just a gradient estimate. In contrastive learning the batch is the supervision: every other view in it is a negative the positive must be ranked against. Shrink the batch and you starve the softmax of distractors.
Count the negatives. A batch of N images produces 2N views — each with its own positive partner — and for every anchor the denominator sums over the other 2N − 1 candidates, one of which is the positive. So each positive competes against 2N − 2 negatives:
Why does the count matter so much? Because the loss is a softmax: it asks the anchor to find its partner among all candidates. With 62 negatives, a mediocre embedding can already win most of the exams — the gradient signal is weak and noisy. With 8,190, the positive has to beat thousands of candidates, so the encoder is pushed to separate images that genuinely differ. SimCLR’s own ablation shows top-1 accuracy climbing steeply from batch 256 to 4096; at batch 32 the loss still decreases, but the features barely improve.
The fix when your GPU cannot hold a big batch is MoCo’s momentum queue: keep the features of the last 65,536 images in a queue, use them as negatives, and decouple the negative count from the batch size. A 256-image batch plus the queue gives 65,792 negatives — the memory cost of the queue (65,536 × 128 dims × 4 B = 32 MiB) is tiny next to a 4096-image batch of activations.
Suppose the encoder ignores its input and returns the same unit vector for every view. Every cosine similarity is then 1, so every logit is 1/τ — identical for the positive and for all 2N − 2 negatives. The softmax has no reason to prefer the target:
This is also the number to watch in a training log. A contrastive run whose loss sits at ln(2N − 1) has learned nothing; one that goes to zero too fast has found a shortcut. The healthy curve passes through a few nats and keeps falling.
InfoNCE ranks a positive against every other view in the batch, so the batch size is not an efficiency dial — it is how much supervision the loss gets. Slide it and watch the arithmetic: negatives, the constant-encoder floor, and the memory of the similarity matrix.
Bars are √-scaled so 62 stays visible next to 8,190. Stops are powers of two; SimCLR’s default is 4096.
512–1024 negatives per positive is where SimCLR-style contrastive learning starts to work. SimCLR's own ablation shows a steep accuracy climb from 256 to 4096; below ~256 the benefit is marginal.
The honest caveat the paper adds: negatives must also be *diverse*. A queue of stale features works because it lags the encoder, but a batch of near-duplicate images gives you fewer useful negatives than its size suggests.
SimCLR needs batches of 512–8192 while supervised ImageNet training is happy at 256. Why?
DINO drops negatives entirely. A student network sees one view and predicts a teacher’s output for the other; the teacher is not trained — it is an exponential moving average of the student’s own weights. The trick is keeping that echo from collapsing into a constant.
Two networks, same architecture. The student has gradients; the teacher is a copy whose weights are updated by a slow exponential moving average of the student’s:
Each image produces two augmented views. View 1 goes to the student, view 2 to the teacher, and the student’s output is trained to match the teacher’s — then the roles swap. There is no negative anywhere: the teacher is the target. But a target that is an average of the model itself invites the laziest solution, which is to output the same vector for everything. DINO’s pretraining loss without guards is minimised by collapse.
Two guards prevent it, and both act on the teacher’s output. Centring keeps a running mean of the teacher’s output per dimension and subtracts it: if one dimension starts to win every image, its mean grows and the subtraction cancels its advantage. Sharpening divides the teacher logits by a small temperature (0.04 early on, then 0.07), which makes the teacher’s distribution peaked. Centring alone would make the output uniform; sharpening alone would let one dimension dominate. Together they produce a distribution that is flat across dimensions, peaked per image — exactly what you want the student to match.
class DinoHead(torch.nn.Module):
"""Toy head from the lesson source; real DINO uses a deeper MLP."""
def __init__(self, in_dim=64, out_dim=128, momentum=0.9):
super().__init__()
self.proj = torch.nn.Linear(in_dim, out_dim)
self.register_buffer("centre", torch.zeros(out_dim))
self.momentum = momentum
def student(self, x, temp=0.1):
return F.log_softmax(self.proj(x) / temp, dim=-1)
def teacher(self, x, temp=0.04):
out = self.proj(x)
# subtract the running centre, then sharpen with a low temperature
return F.softmax((out - self.centre) / temp, dim=-1).detach()
@torch.no_grad()
def update_centre(self, teacher_out): # momentum 0.9
self.centre.mul_(self.momentum).add_(teacher_out.mean(dim=0), alpha=1 - self.momentum)DINO has no negatives. Two cheap tricks keep the teacher honest: centring subtracts the per-dimension running mean, and sharpening divides by a small teacher temperature. Turn each one off and watch the output collapse — one dimension dominating, or everything going flat.
Largest dim mean = how much probability mass the busiest output dimension takes across all images (collapse is above ~0.15); mean entropy = how peaked each image’s distribution is (uniform collapse is above ~0.93 of 1.000).
healthy — peaked per image (entropy 0.078) and spread across dimensions (top dim mean 11.0%)
The numbers come from a deterministic toy head, not a trained network: 64 fixed feature vectors through a random projection. What transfers is the failure mode — the same collapse DINO’s centring and sharpening are designed to prevent.
What stops DINO from collapsing to a constant output?
MAE takes a ViT, masks 75% of the patches at random, shows the encoder only the visible quarter, and trains a small decoder to reconstruct the missing pixels. Then it throws the decoder away — the encoder is the product.
The arithmetic first, because it explains the speed. A 224×224 image at 16px patches is 196 patches; mask 75% and 147 patches are hidden, 49 stay visible. The encoder — the big ViT — only ever sees those 49 plus the [CLS] token: 50 tokens instead of 197, so its attention and MLP cost is roughly a quarter of the full image. The 147 mask tokens are inserted only in the decoder, which is deliberately small: 8 transformer blocks, 512 dimensions wide, versus the encoder’s 12 blocks at 768. The loss is mean squared error on the masked patches only — 147 × 16 × 16 × 3 = 112,896 pixel values scored per image.
That asymmetry is the key design choice: the decoder never sees the task that matters; it exists so the encoder can be cheap. Because the encoder skips the masked tokens entirely, MAE pretrains about 3× faster than BEiT, which pushes all 197 tokens through the encoder. After pretraining, the decoder is discarded. The encoder is the feature extractor that gets fine-tuned or probed.
And the ratio? MAE’s paper sweeps 50%, 75% and 90% and finds 75% the sweet spot for both fine-tuning and linear probing. That number is not arbitrary. Natural language has high entropy per token: BERT can mask 15% of words and each blank still has many plausible completions, so the model must understand the sentence. Image patches have low entropy: neighbouring pixels are so correlated that a masked 16×16 patch is almost determined by the patches around it. Mask 15% of an image and a blur reproduces it — no semantics required. You have to mask aggressively until local extrapolation fails, and the encoder is forced to represent the object rather than the texture.
def random_mask_indices(num_patches, mask_ratio=0.75, seed=0):
"""Return the visible and masked patch indices for one image."""
g = torch.Generator().manual_seed(seed)
n_keep = int(num_patches * (1 - mask_ratio)) # 196 * 0.25 = 49
perm = torch.randperm(num_patches, generator=g)
visible = perm[:n_keep]
masked = perm[n_keep:]
return visible.sort().values, masked.sort().values
visible, masked = random_mask_indices(196, mask_ratio=0.75)
print(f"visible: {len(visible)} / 196") # 49
print(f"masked: {len(masked)} / 196") # 147
# the encoder sees the visible patches only:
# shuffle the 49 visible patches (MAE adds no mask tokens to the encoder),
# prepend [CLS] -> 50 tokens, run the big ViT.
# the decoder gets:
# 50 encoder tokens + 147 learned mask tokens, un-shuffled to 196 positions,
# then a small 8-block / 512-dim transformer predicts the pixels.
# loss = MSE(reconstructed, original) on the 147 masked patches only.Slide the MAE mask ratio from 15% to 90%. The left grid is all the encoder ever sees; the right grid fills the blanks with a local baseline (average the nearest visible patches) and reports its real mean error. The curve is why 75% is not arbitrary.
BERT masks 15% of text tokens because a masked word still has many plausible completions. Image patches do not: a 16×16 patch is almost predicted by its neighbours, so the ratio has to go up until the task stops being a blur.
MAE masks 75% of patches; BERT masks 15% of tokens. Why the difference?
A positive pair is only useful if the two views differ in a way the encoder can learn to ignore. Random crop says “where the object sits does not matter”; colour jitter says “its colours do not matter”. Choose them carelessly and you teach the model to ignore your labels.
Everything self-supervised is defined by its augmentations, because they are the only source of variation in the data. A contrastive pair is the same image under two random draws of the augmentation policy, so the policy is a statement about which variations should map to the same feature. SimCLR’s pipeline is four transformations, and each one buys a specific invariance:
Strength is a dial with two failure modes. Turn it down and the positive pair is nearly identical, so the model can win the exam with a trivial solution and the features barely change. Turn it up and the two views share so little that the positive is indistinguishable from a negative — the task becomes noise, or worse, the encoder learns to be invariant to exactly the thing your downstream task needs. That second failure is the one that costs real projects.
import torch
import torchvision.transforms as T
two_view_train = lambda: T.Compose([
T.RandomResizedCrop(96, scale=(0.2, 1.0)),
T.RandomHorizontalFlip(),
T.ColorJitter(0.4, 0.4, 0.4, 0.1),
T.RandomGrayscale(p=0.2),
T.ToTensor(),
])
class TwoViewDataset(torch.utils.data.Dataset):
def __init__(self, base):
self.base = base
self.aug = two_view_train()
def __len__(self):
return len(self.base)
def __getitem__(self, i):
img, _ = self.base[i] # the label is read and discarded
v1 = self.aug(img) # draw 1
v2 = self.aug(img) # draw 2 — independent
return v1, v2
# Every __getitem__ returns two augmented views of one image.
# The label in base[i] is never used: that is the whole point.Two views of the same image are only useful if they differ in a way the encoder can learn to ignore — and only if they still share what matters. Toggle each augmentation and watch the two views rewrite themselves.
The source’s pipeline is exactly this: RandomResizedCrop(96, scale=(0.2, 1.0)), RandomHorizontalFlip, ColorJitter(0.4, 0.4, 0.4, 0.1), RandomGrayscale(p=0.2). The strength is a design choice — too weak and the task is trivial, too strong and you train a model that ignores your labels.
The standard evaluation of a self-supervised model is a linear probe: freeze the encoder, train one linear classifier on top, report top-1. It isolates feature quality from fine-tuning dynamics — and it is also the cheapest thing you can run on your own data.
The recipe is three lines: run images through the frozen encoder, store the feature vectors (a DINOv2-B CLS token is 768 numbers), and fit Linear(768 → num_classes) on your labels. No backbone gradients, no augmentations needed, minutes on a CPU for a small set. The published anchors show what that buys, all on ImageNet-1k:
The 15.6-point gap between MAE ViT-B’s probe (68.0) and its fine-tune (83.6) is the honest caveat of the whole field: frozen features are good, but a linear classifier can only use the directions already present in the feature space. Fine-tuning reshapes the features for the task, which is why it wins when labels are plentiful — and why a probe can win when they are scarce and the fine-tune starts overfitting. The board below walks that crossover across label budgets.
Two more things frozen features do with no training at all. Retrieval: embed every image in a gallery, embed the query, rank by cosine similarity — that is image search, duplicate detection and “more like this” without a single label. k-NN classification: embed the labelled set, embed the test image, vote among its nearest neighbours; within a few points of a linear probe and zero training. Frozen DINOv2 features also feed depth, segmentation and correspondence heads (they are the backbone in 2020s dense-prediction stacks), because the patch tokens carry dense spatial information, not just the CLS summary.
import torch
from transformers import AutoImageProcessor, AutoModel
processor = AutoImageProcessor.from_pretrained("facebook/dinov2-base")
model = AutoModel.from_pretrained("facebook/dinov2-base")
model.eval()
with torch.no_grad():
inputs = processor(images=[pil_image], return_tensors="pt")
outputs = model(**inputs)
embedding = outputs.last_hidden_state[:, 0] # CLS token, 768 dims
# zero-shot retrieval: cosine similarity against a gallery of embeddings
# linear probe: LogisticRegression().fit(gallery_embeddings, labels)
# dense tasks: outputs.last_hidden_state[:, 1:] are the patch tokens
# MAE-style fine-tuning instead? The timm repo ships every MAE checkpoint:
# model = timm.create_model("vit_base_patch16_224.mae", pretrained=True)Self-supervised pretraining gives you a frozen encoder. A linear probe on top measures the features; fine-tuning the whole encoder is what you ship when you have labels to spend. Move the budget and watch the two lines trade places.
Bars span 40–90% top-1 so the differences stay readable. The random baseline is the 26% you get from a frozen encoder with no training at all — the floor every other row has to beat.
· start with the frozen encoder and a linear probe: it is the cheapest run, it trains in minutes, and it tells you whether the pretrained features already separate your classes.
· the illustrative curves put the probe ahead of full fine-tuning at this budget (49.4 vs 44.0) — with few labels, fine-tuning's millions of free parameters overfit before they help. That low-shot crossover is a published pattern, not a law; measure it.
· if the probe is close to the accuracy you need, spend your money on more *unlabelled* data instead of more labels.
The curves are a teaching model — shape and crossover follow the SSL literature, endpoints are pinned to MAE ViT-B/16’s 68.0 / 83.6 at 1.28M labels. Your own crossover will move with dataset size and similarity to the pretraining data.
The batch-size question and the collapse question are the two that separate “I have heard of SimCLR” from “I could debug a contrastive run”. The 75%-vs-15% question is the one people get wrong because they memorise the number instead of the reason.
0 / 5 answered · 0 correct
01Why does SimCLR need batch sizes of 512–8192 while supervised ImageNet training works with batch 256?
02What prevents DINO from collapsing to a constant output?
03MAE masks 75% of patches. BERT masks 15% of tokens. Why the difference?
04After self-supervised pretraining, a 'linear probe' evaluation trains what?
05Why does MAE use an asymmetric encoder–decoder design?
Click a card to swap the lazy description for what it actually means.
Three problems with real numbers: sweep τ and watch the loss respond, break a DINO head by removing the centre buffer, and pretrain MAE on CIFAR-100 and compare linear probes at 10, 50 and 200 epochs. Try first; a worked answer is one click away.
L2-normalise two sets of 16 paired embeddings (dim 32): one where positives really are close (alignment ≈0.9) and one where the 'positive' is random. Run the lesson's info_nce for each τ and print both losses. With a fixed seed the table looks like this — aligned pair (cos ≈ 0.9), random pair (cos ≈ 0): τ = 0.05 → 0.0001 / 7.669; τ = 0.07 → 0.0013 / 5.903; τ = 0.1 → 0.0163 / 4.750; τ = 0.2 → 0.410 / 3.773; τ = 0.5 → 1.855 / 3.481. Three things to read off it. (1) For aligned pairs the loss falls as τ falls: colder softmax puts more probability on the positive — 0.0001 at τ = 0.05. (2) For random pairs the loss rises as τ falls: the softmax tries to pick one of 31 candidates whose similarities are all near zero but not equal, and a colder softmax amplifies that spread. ln(31) = 3.434 is the floor for a constant encoder — identical embeddings, every candidate equally plausible — while the random encoder already measures 4.750 at τ = 0.1, above it, and climbs further as τ falls. (3) At τ = 0.5 the two curves are far apart but the aligned loss is no longer tiny (1.855), because a warm softmax spreads probability over the negatives. The practical reading: a small τ needs a big batch (many negatives) and a positive that is genuinely closer than the negatives; that is why SimCLR uses τ = 0.1 at batch 4096, not batch 32.
Build a toy head: a linear projection from 64-dim features to 16 or 128 outputs, a centre buffer initialised to zeros, a student log_softmax at τ = 0.1 and a teacher softmax at τ = 0.04 over (logits − centre); update the centre after every batch with c ← 0.9·c + 0.1·mean(teacher output over the batch), and update the teacher weights with θ_t ← 0.996·θ_t + 0.004·θ_s. Run both directions (student on view 1 vs teacher on view 2, then swap) with a cross-entropy loss. With centring on, the diagnostic numbers the lesson's collapse lab computes are: largest per-dimension mean of the teacher's output ≈0.11 (uniform is 1/16 = 0.0625) and mean normalised entropy ≈0.08 — distribution flat across dimensions, peaked per image. Turn centring off and the same head gives largest dim mean ≈0.63: one dimension absorbs most of the probability mass on every image, and the student's best strategy is to predict that dimension blindly — the loss falls fast while the representation dies. Watch two numbers per epoch: the max per-dimension mean should stay under ~0.15 and per-sample entropy should stay low; if the max dim mean climbs past ~0.2 or entropy goes to ~1.0, you are collapsing and no amount of training will fix it.
Do the shapes before touching code. CIFAR-100 is 32×32 pixels; with 4×4 patches that is 8×8 = 64 patches. Mask 75%: 48 masked, 16 visible, so the encoder sees 16 + 1 [CLS] = 17 tokens instead of 65, and the MSE scores 48 × 4 × 4 × 3 = 2,304 pixel values per image — 49× fewer than ViT-B's 112,896 on 224², which is why this fits on one GPU. Pretrain with AdamW, a small learning rate and random resized crops, then throw the reconstruction head away. For the evaluation, freeze the encoder and fit a single linear layer on top of the pooled features using three labelled subsets: 1,000, 10,000 and 50,000 images of CIFAR-100. Report three numbers per pretraining checkpoint (10/50/200 epochs) and compare against two baselines: (a) a linear probe on the same frozen randomly-initialised backbone and (b) a small CNN trained from scratch supervised on the same 1,000 images. The expected pattern from the MAE paper, at miniature scale: the random-features probe barely beats chance (you should see roughly 10–20% with good pooling), the from-scratch supervised model fits the 1,000 images and then overfits, and the MAE-pretrained probe improves with pretraining epochs before plateauing — beating the from-scratch probe on 1,000 images and closing on supervised training as labels grow. If your pretrained probe never beats the random-features probe, check three things in order: the mask ratio is actually 75%, the encoder never sees masked patches, and the linear probe uses the frozen features (no gradients into the backbone).
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 04, Lesson 17) and the Math Foundations Notebook reference build. The seven labs — the InfoNCE playground, the three-family comparator, the batch-size board, the DINO collapse board, the masking lab, the augmentation lab and the linear-probe board — are original to this page, as is the annotation arithmetic ($10M ÷ 1,281,167 ≈ $7.81 per ImageNet label), the negative counts (62 / 1,022 / 8,190) with the constant-encoder floor ln(2N − 1) and its derivation, the worked one-anchor losses (0.471 at τ = 0.5; 0.2147 for a hard negative at τ = 0.07), the DINO EMA half-life (0.996¹⁷³ ≈ 0.5), MAE's patch arithmetic (49 visible, 147 masked, 50 encoder tokens, 112,896 scored pixel values) and its CIFAR-100 variant (64 patches → 48 masked, 17 tokens, 2,304 values), the 268 MB similarity matrix at batch 4096 and MoCo's 32 MiB queue of 65,536 features, the augmentation-invariance trap, and the illustrative probe-vs-fine-tune curves calibrated to MAE ViT-B/16's published 68.0 / 83.6 anchors. Every number shown is computed live by the labs or verified by hand in the prose.