Train an image encoder and a text encoder together so that matching (image, caption) pairs land at the same point in a shared space. That is the whole trick — and it is why a class can be a sentence you write, a search engine is a vector lookup, and every VLM you have met in this phase starts with a vision tower trained this way.
An image encoder (ViT or ResNet) and a text encoder end in projections to the same dimension, L2-normalised so a dot product is a cosine. ViT-L/14 is 303,179,776 image parameters and 123,060,480 text parameters — 427.6M, the paper's 428M — with 256 patches + CLS = 257 tokens at 224px.
24 blocks, width 1024 → 768 · 49,408-word vocab · one shared spaceThe loss is a symmetric cross-entropy over the N × N matrix: each row and each column has to find its diagonal. At batch 32,768 that is 1,073,709,056 negatives (32,767 per positive); τ starts at 0.07, is clipped at 100, and is learned. A model that knows nothing pays log N — 2.079 for 8, 10.397 for 32,768.
loss = ½(CE(sim) + CE(simᵀ)) · τ ≈ 0.07 · log NWrite one prompt per class, encode them once, encode the image, take the argmax of the cosine row. The prompt is the only lever: the default template is worth ~1.3 points on ImageNet and 80 averaged templates ~3.5 more. Zero-shot ViT-L/14 reaches 76.2% — matching a supervised ResNet-50 — with no labels at all.
"a photo of a {}" +1.3 · ensemble of 80 +3.5A two-tower model is a ruler, not a mouth: it measures how close an image and a sentence are, and every higher-level trick — zero-shot classes, retrieval, open-vocabulary detectors, the vision tower inside a VLM — is built on that one measurement.
By the end you will be able to do the batch arithmetic (N pairs → N positives and N²−N negatives; log N at initialisation) and explain why the loss is symmetric; count a CLIP checkpoint’s parameters tower by tower; run zero-shot classification as prompt → cosine → argmax and improve it with prompt templates and ensembling; read the learned temperature (init 0.07, learned near 0.01) and say what it does and does not change; choose between CLIP, SigLIP, OpenCLIP and a LLaVA-style VLM for a given job; and name the five blind spots — order, counting, negation, relations, handwriting — before a stakeholder finds them for you.
A 1,000-class classifier can only ever predict 1,000 things. CLIP (Radford et al., 2021) threw the head away and trained two encoders against each other on 400 million web (image, caption) pairs — so a new class is a sentence you write, not a dataset you label.
This is the closed-vocabulary problem: a model whose last layer has 1,000 outputs cannot name a 1,001st category, and every new category means new labels, a new head and a re-train. Detection, segmentation, retrieval and generation all inherit the same wall in different clothes. The fix is not a bigger head; it is to stop predicting labels and start predicting alignment.
CLIP’s bet is that the internet is already labelled — with captions. Scrape 400 million (image, text) pairs, and the supervision is free. The architecture that exploits it is deliberately plain: two encoders that share no weights, each ending in a linear projection to the same embedding dimension, each output L2-normalised so that a dot product is a cosine. The image tower is a ViT (or a ResNet); the text tower is a transformer; and nothing in either tower knows the other one exists until their outputs meet in the loss.
For ViT-L/14, the shapes are worth tracing once. A 224×224 image with 14-pixel patches divides into 16 positions per side: 16 × 16 = 256 patches, plus the class token = 257 tokens into a 24-block transformer of width 1024. At 336 pixels the same arithmetic gives 24 × 24 = 576 + 1 = 577 tokens. The caption is a BPE sequence of at most 77 tokens over a 49,408-word vocabulary, running through 12 layers of width 768. Both towers then project into the same 768-dim space (512 dims in the smaller B/32 checkpoint) — and that shared space is the product.
Two numbers in that table explain the family. First, the text tower is 123M parameters — and 37.9M of them are the token-embedding matrix, one vector per vocabulary word. Captions are short, so the tower is only 12 layers; the vocabulary is what makes it big. Second, the two towers are wildly asymmetric: 303M for pixels, 123M for words. That is not an accident of design — images are harder to compress into a concept than the sentence that describes them.
The rest of the family is this recipe with the dials turned. B/32 uses 32-pixel patches (7 × 7 = 49 + 1 = 50 tokens), a 768-wide image tower and a 512-dim shared space for 151,277,313 parameters in total; B/16 is 149,620,737; L/14@336px reuses the L/14 weights at a higher resolution. The bigger open checkpoints — OpenCLIP’s ViT-H/14, EVA, the SigLIP so400m — run the same objective with a 1024- or 1152-dim space and a 632M-plus image tower. Same interface, same space, more dimensions.
import torch
import torch.nn as nn
import torch.nn.functional as F
class TwoTower(nn.Module):
"""The lesson's stand-in: real CLIP is ViT + transformer, these are MLPs
over pre-extracted features, but the interface is identical."""
def __init__(self, img_in=128, txt_in=64, emb=64):
super().__init__()
self.image_proj = nn.Sequential(nn.Linear(img_in, 128), nn.ReLU(),
nn.Linear(128, emb))
self.text_proj = nn.Sequential(nn.Linear(txt_in, 128), nn.ReLU(),
nn.Linear(128, emb))
# the logit scale is stored as a log-parameter: ln(1 / 0.07) = 2.6592
self.logit_scale = nn.Parameter(torch.ones([]) * 2.6592)
def encode_image(self, x):
return F.normalize(self.image_proj(x), dim=-1)
def encode_text(self, x):
return F.normalize(self.text_proj(x), dim=-1)
def forward(self, img_feats, txt_feats):
return self.encode_image(img_feats), self.encode_text(txt_feats), self.logit_scale.exp()A ViT-L/14 CLIP tower runs at 224×224 with 14-pixel patches, and the same weights are run at 336×336. How many tokens does the transformer see in each case?
There is no classifier head and no target tensor of class indices. The supervision is which caption belongs to which image — so a batch of N pairs quietly creates N positives and N²−N negatives, and the loss is a cross-entropy that has to find the diagonal.
Take a batch of N (image, caption) pairs. Encode every image and every caption, L2-normalise both, and multiply: an N × N matrix where entry (i, j) is the cosine between image i and caption j. The matching pairs are the diagonal; everything else is a mismatch that the model must push down. Then apply cross-entropy to each row — the target is the column index that equals the row index — and again to each column, and average. That is the whole objective.
Why both directions? Because both queries exist at inference. Zero-shot classification asks “given this image, which text is closest?” — a row of that matrix. Retrieval asks “given this caption, which image is closest?” — a column. Training only one direction leaves the other weak. The paper’s pseudocode calls it loss_i2t and loss_t2i and averages them.
That last line is the sanity check the source’s first code block prints. It is not a coincidence that the initial loss is log N — it is the definition. A model that has learned nothing puts roughly equal mass on every caption in the batch, and the cross-entropy of a uniform distribution over N options is exactly log N. The lab below measures it the other way: crank the temperature up until every row flattens and a trained model pays 2.020 against log 8 = 2.079.
The temperature is a learned scalar, and its journey is worth knowing. CLIP initialises τ = 0.07 — the logit scale is stored as a log-parameter, ln(1/0.07) = 2.6592, exactly the number in the source’s TwoTower — and clips exp(s) at 100 because unclipped logit scales made training unstable. In the released ViT-L/14 checkpoint the learned value is 4.605170 = ln(100) exactly, the ceiling, so τ = 0.01. (SigLIP’s base/16 checkpoint learns 4.765, τ = 0.0085, plus a bias of −12.93 — the extra term a per-pair loss needs to calibrate how rare matches are.) The init works because of a fact about high dimensions: two random unit vectors in d dimensions have cosine ≈ ±1/√d, so CLIP’s 512-dim initialisation gives cosines of ±0.044, logits near zero, and a loss near log N. A 64-dim toy model has ±0.125 — about 3× larger, because the spread scales as 1/√d (1/√64 against 1/√512) — which is why a toy can pay visibly more than log N at the same τ.
Spin the towers up from random weights and watch the diagonal light up; then change τ and watch the same row turn into a sharp or a flat softmax. The line under the matrix is the distribution the loss is computed from.
At τ = 4 the loss sits at 2.020, a hair under log 8 = 2.079: a high temperature flattens every row until the model is paying the price of guessing. At τ = 0.07 the same trained matrix pays 0.305 — temperature does not change what the model knows, only how confidently it is scored.
def clip_loss(image_emb, text_emb, logit_scale):
"""image_emb, text_emb: (N, d), L2-normalised.
logit_scale is exp(learned log-parameter), the 1/τ."""
N = image_emb.size(0)
sim = logit_scale * image_emb @ text_emb.T # (N, N)
targets = torch.arange(N, device=sim.device) # the diagonal
l_i = F.cross_entropy(sim, targets) # image → text
l_t = F.cross_entropy(sim.T, targets) # text → image
return (l_i + l_t) / 2
# the source's sanity check: a random model, batch 8, prints ~ln 8
# batch size: 8 loss: 2.08Once images and sentences live in the same space, classifying, searching and ranking are the same three operations: encode, take a cosine, sort. The temperature decides how sharply you read the ranking — it never changes the ranking itself.
Start with the two-class version, because it is the arithmetic every other use inherits. A trained toy CLIP scores an image of a dog at cosine 0.734 against “a photo of a dog” and 0.607 against “a photo of a car” — its runner-up among all six classes, a plain gap of 0.127. Put just those two prompts in the list and the gap becomes 1.81 logits after dividing by τ = 0.07, and the softmax gives the dog 0.86. At τ = 1 the same pair of cosines gives 0.53 — barely better than a coin flip. Nothing about the model changed; the temperature only decides how much separation you ask the softmax to read out of the same similarity.
With all six classes enabled the mass is shared and the dog takes 0.709 of it (the car drops to 0.115) — which is the second thing to internalise about the space: every score depends on the other classes in the list. Turning a class off in the lab redistributes probability across the rest. A cosine is a local comparison, not a calibrated confidence, and the softmax only makes it look like one.
The payoff is retrieval. Encode a gallery once, keep the vectors, and a query — an image or a sentence — is one embedding and a nearest-neighbour search. The arithmetic is friendly: 768 float32 dimensions is 3,072 bytes per image, so 1,000 images index to 2.93 MiB and a million images to 2.86 GiB, and one query against a million images is 1.5 GFLOP of dot products. You score it with recall@K: the fraction of queries whose true match lands in the top K. That is exactly the rank of the diagonal in the batch matrix from chapter 02 — the lab’s trained 8-pair matrix gets recall@1 = 8/8.
Pick an image and a prompt style, then slide the temperature. The argmax is decided by the cosines; τ only decides how much confidence the softmax puts on it. Add or remove classes and watch the normalisation change.
Cosine is not a probability and does not change when τ changes — a point the OpenCLIP example makes explicit by hard-coding a scale of 100 (τ = 0.01, the ceiling the CLIP paper clipped training to). Everything downstream of the softmax is a decision, not a score.
import torch
import torch.nn.functional as F
@torch.no_grad()
def zero_shot_classify(model, image_feats, class_text_feats, class_names):
"""image_feats: (N, img_in)
class_text_feats: (C, txt_in) — one averaged embedding per class"""
i = F.normalize(model.encode_image(image_feats), dim=-1) # (N, emb)
t = F.normalize(model.encode_text(class_text_feats), dim=-1) # (C, emb)
sim = i @ t.T # (N, C)
pred = sim.argmax(dim=-1)
return [class_names[p] for p in pred.tolist()]
# Note what is NOT here: no head, no training, no labels.
# The (C, emb) matrix of text embeddings *is* the classifier.
# Swap in real CLIP and the same three lines run production zero-shot.Zero-shot classification is a loop you could write on a napkin: prompt every class, encode everything, take the argmax. The only lever left is how you phrase the class — and the paper measured that lever at almost five points on ImageNet.
The procedure, exactly as it runs against a production checkpoint:
(C, d) matrix, where C is the number of classes. This is cached: it does not depend on the test images at all.(1, d) vector.I @ T.T is a (1, C) row of cosines — the same row the contrastive loss computes during training.That is the entire method, and it is why the phrase “open vocabulary” is not marketing: the classifier is a matrix of sentences, computed at inference, discardable and replaceable. The 1,000-class ImageNet classifier you would have trained in 2021 is now a 1,000-row text matrix you can rebuild in milliseconds.
Then the interesting part: the prompt is a hyperparameter. The paper wrapped every ImageNet class in “a photo of a {label}.” and measured a gain of about 1.3 points over the bare class name — because “dog” alone is not a sentence, and the text tower was trained on sentences. Ensembling 80 templates (“a blurry photo of a {label}”, “a sketch of a {label}”, “a photo of many {label}s”, …) and averaging their embeddings added 3.5 points more over the default template. Together, prompt engineering and ensembling improve ImageNet zero-shot accuracy by almost 5 points — with no training data and no gradient step.
The templates are also domain-specific, and the paper says so in as many words: “a photo of a {label}, a type of pet” for pets, quotes around the text to be recognised for OCR datasets, “a satellite photo of a {label}” for overhead imagery. In every case the sentence is doing the work a labelled training set used to do.
Pick an image and read down the list. The thin top bar is the class margin on this image; the bottom bar is the same margin averaged over the six toy images. The bare label and the adjective-free template are not close.
The 80 templates are generated from ten modifiers × eight framings — the spirit of the list OpenAI published. All seven recipes are scored by the same text encoder; nothing is re-trained. Teaching model: the embedding space is hand-built, so read the ordering and the arithmetic, not the absolute cosines.
The paper’s numbers for the same idea: “a photo of a {label}” beats the bare class name by ~1.3 points on ImageNet, and ensembling 80 templates adds ~3.5 more — together almost 5 points, with no training whatsoever.
# the bare label — worst, because it is not a sentence
prompts = ["dog"] # toy margin 0.084
# the default template — the paper's +1.3 points over the bare label
prompts = ["a photo of a dog"] # toy margin 0.127
# style and quality variants — each a different point in text space
prompts = ["a blurry photo of a dog", # 0.159
"a close-up photo of a dog", # 0.171
"a sketch of a dog", # 0.169
# the 80-template ensemble — average the embeddings, then re-normalise
texts = [t.format("dog") for t in TEMPLATES] # 80 strings
embeds = encode_text(tokenizer(texts)) # (80, d), unit vectors
class_embedding = F.normalize(embeds.mean(dim=0), dim=-1) # toy margin 0.168
# why averaging works: each template = concept + a quirk; the mean's
# quirk shrinks by ~sqrt(80) ≈ 8.9×, so the centroid is a steadier
# description of "dog" than any single phrasing.You are shipping a 12-class plant-disease classifier with no labelled images. You drop 'a photo of a {}' and prompt with the bare disease names to save time. What happens?
CLIP, SigLIP, OpenCLIP and LLaVA-style VLMs are easy to blur together. They split cleanly on two axes: the loss (softmax or sigmoid) and the output (an embedding or a sentence).
CLIP is the reference: a softmax contrastive loss over the whole batch, 400M web pairs, batch 32,768, a 428M ViT-L/14 and 76.2% zero-shot ImageNet. Its weakness is structural — the softmax needs negatives, and the negatives come from the batch, so small batches starve the signal.
SigLIP (Zhai et al., 2023) fixes exactly that. Each pair gets its own binary decision: y = +1 for a match, −1 otherwise, and the loss is log(1 + exp(−y·sim)). There is no denominator over the batch, so batch size stops being the constraint. The paper’s curves put the sigmoid loss ahead of the softmax below 32k and comfortably ahead below 16k, and it names roughly 32k as the practical sweet spot even after trying batches up to a million. Its base/16 model reaches 71.0% zero-shot ImageNet after three days on 16 TPU-v4 chips. Combined with locked-image tuning, SigLiT reached 84.5% on four chips in two days. Its released checkpoints are 203M parameters for base/16 — and, pleasingly, the text tower (110.3M) is bigger than the image tower (92.9M).
OpenCLIP is the open reproduction: CLIP’s reimplementation and re-training on LAION data, in many sizes, with published recipes. Its ViT-H/14 (632M image tower) trained on LAION-2B reaches 78.0% zero-shot ImageNet and 73.4% recall@5 on COCO retrieval. For most production pipelines it is the default, because you can read the data card and ship the weights.
LLaVA-style VLMs answer a different question. Take a frozen CLIP-family tower, add a connector — LLaVA-1.5 uses a two-layer MLP — and attach a language model; now the model writes an answer instead of producing a vector. The 336px tower turns its 24 × 24 = 576 patches into 576 visual tokens for the LLM; nothing about that path looks like the N × N matrix. It cannot build your retrieval index, and a text→image index cannot answer a question.
# CLIP: one softmax per row and per column — the batch IS the normalisation
def clip_loss(i, t, logit_scale):
N = i.size(0)
sim = logit_scale * i @ t.T
targets = torch.arange(N, device=sim.device)
return (F.cross_entropy(sim, targets) + F.cross_entropy(sim.T, targets)) / 2
# SigLIP: every pair gets a binary label — no batch denominator at all
def siglip_loss(i, t, logit_scale, bias):
N = i.size(0)
sim = logit_scale * i @ t.T + bias
labels = 2 * torch.eye(N, device=sim.device) - 1 # +1 diagonal, −1 off
return -F.logsigmoid(labels * sim).mean()
# At inference the difference is visible: CLIP gives a softmax distribution
# over classes; SigLIP gives one sigmoid probability per pair, so several
# classes can be 'yes' at once — and none of them has to sum to 1.Pick the constraint your project actually has. The ranking below names the family that fits first and says why — including when the answer is a generative model rather than a contrastive one.
| family | loss | batch | output | headline |
|---|---|---|---|---|
| softmax contrastive over the batch (InfoNCE) | the best runs used 32,768; small batches starve the softmax of negatives | one L2-normalised embedding per image, one per caption | 76.2% zero-shot ImageNet (ViT-L/14) — the supervised ResNet-50 it matches scored 76.1% | |
| per-pair sigmoid: log(1 + exp(−y·sim)), y = ±1 | no batch-level normalisation: the paper's curves put it ahead below 32k and clearly ahead below 16k; 32k is the practical sweet spot | a logit per pair; sigmoid gives independent match probabilities, not a softmax over classes | 71.0% zero-shot ImageNet for base/16 after three days on 16 TPU-v4 chips; SigLiT reached 84.5% with locked-image tuning on four chips | |
| CLIP's softmax contrastive, reimplemented | anything from 1k to 33k depending on the run; each checkpoint documents its own recipe | the same two-tower interface, plus a training script and a data card | ViT-H/14 trained on LAION-2B: 78.0% zero-shot ImageNet and 73.4% recall@5 on COCO retrieval | |
| generative: next-token prediction on image-conditioned instructions | instruction data, not pairs; the vision tower is frozen and a connector is trained | text — an answer, a caption, a reasoning trace — not an embedding | not a zero-shot classifier; measured on VQA and instruction-following suites instead |
Numbers from the papers and the model cards: CLIP’s 428M ViT-L/14 (303.2M image tower + 123.1M text tower) and its 76.2% zero-shot ImageNet; SigLIP’s 71.0% base/16 after three days on 16 TPU-v4 chips and the 84.5% SigLiT result; OpenCLIP’s ViT-H/14 at 78.0% on LAION-2B; LLaVA-1.5’s CLIP ViT-L/14@336 tower plus a two-layer MLP and a 7B/13B language model.
The dividing line is what the model produces. A two-tower model produces one vector per input, so every task becomes a distance computation — index-friendly, cheap at inference, blind to anything that needs a sentence back. A VLM produces tokens, so it can answer a question but cannot be a vector index.
Your lab has four GPUs and 30M (image, caption) pairs. You want to train a two-tower model at batch 256. Which loss is the right default, and why?
CLIP matches concepts, and a concept bag has no word order, no count, no negation and no relations. The paper lists these failures itself — which is why the honest lesson is not “CLIP sees everything” but “CLIP sees these things and is blind to those.”
The mechanism is easy to state once you have seen the objective. Two captions that use the same nouns are nearly the same point in text space, because the loss only ever asked the model to separate different images from different captions — never to parse the sentence. “A dog next to a car” and “a car next to the dog” overlap almost completely; so do “two dogs” and “three dogs”, and “a dog with a leash” and “a dog without a leash”. Counting, negation, spatial relations, fine-grained species and handwriting are the five places this shows up most reliably.
The paper is unusually direct about it. Its analysis names counting objects in synthetic scenes (CLEVRCounts) among the tasks where zero-shot CLIP is weak, says CLIP “struggles with more abstract and systematic tasks such as counting”, and finds fine-grained classification — car models, flower species, aircraft variants — markedly weaker than its object-level results. The OCR result is the sharpest single example: CLIP learns a good text representation for rendered text (Rendered SST2 is one of its strongest results), but it reaches only 88% on MNIST handwritten digits, where an embarrassingly simple logistic regression on raw pixels beats it. That is the signature of a model that has seen digital text on screens and almost no handwriting: the failure is about the training distribution, not about the concept.
Keep the balance in view, though, because the same paper shows CLIP outperforming a Noisy Student EfficientNet-L2 on 21 of 27 datasets, and being markedly more robust to natural distribution shift than supervised ImageNet models. Where a few labels exist, the ceiling is higher still: a linear probe on frozen CLIP features reaches 85.4% on ImageNet against 76.2% zero-shot. The practical reading is not “CLIP is bad” — it is that a similarity score is a tool with a known shape, and it is the wrong tool for counting or relations.
Each row is two prompts that differ in one way — order, number, negation, a spatial word, a breed, a script. A model that reads concepts rather than sentences gives them nearly the same score. Select a row for the explanation and the fix.
Stand-in: a bag-of-concepts model — the same trick as pooling the sentence’s word vectors with no order. Real CLIP is a transformer over the caption, so these pairs are not literally identical to it; the failure pattern is the documented one, and the paper’s own numbers (CLEVRCounts, MNIST 88%, fine-grained sets) are quoted in each row.
The pattern behind all six: CLIP scores how much the image and the sentence are about the same concepts. Anything the concepts cannot express — order, count, negation, relations, script style — is a place where a detector, an OCR model, a VLM or a handful of labels will beat a zero-shot prompt.
CLIP is rarely the product. It is the layer where pixels and sentences first understood each other — and every open-vocabulary system since inherits that layer, from detectors and segmenters to the VLMs and text-to-image models you have already met in this phase.
Once a shared embedding space exists, the vision-and-language tasks become distance computations with different things on the other end. Detection becomes “find regions whose embedding is close to the text” (Grounding DINO, OWL-ViT wrap a CLIP text tower around a detector). Segmentation becomes “classify each region against the prompt” (CLIPSeg; SAM takes text prompts through a CLIP-family encoder). Generation becomes “steer the sampler with a text embedding” (Stable Diffusion and DALL·E 3 both condition on CLIP-family text features). And VLMs use the vision tower as the bridge between an image and a language model.
The two artefacts the source lesson ships are the right summary of what you can build today. The first is a zero-shot class picker: given a domain and a list of class names, produce the prompt templates worth trying, then measure them on a small held-out set if you have one. The second is a text→image retriever: encode a gallery once, store the vectors, and answer natural-language queries with a nearest-neighbour search. Both are written in a day and both start with the same question: what is the text that describes the thing I am looking for?
Two practical notes for 2026. First, the checkpoint you pick matters less than you think — B/32, L/14, SigLIP base, H/14 — because the interface is identical and the ranking is usually preserved; pick for your latency and licence budget, then spend your time on prompts and data. Second, the family has moved to SigLIP-2 and friends for new training runs, but the CLIP-style objective is still the thing that makes the space work. Reading this lesson is reading the vocabulary of every multimodal system that follows.
import open_clip
import torch
from PIL import Image
model, _, preprocess = open_clip.create_model_and_transforms(
"ViT-B-32", pretrained="laion2b_s34b_b79k")
tokenizer = open_clip.get_tokenizer("ViT-B-32")
image = preprocess(Image.open("dog.jpg")).unsqueeze(0)
text = tokenizer(["a photo of a dog", "a photo of a cat", "a photo of a car"])
with torch.no_grad():
image_features = model.encode_image(image)
text_features = model.encode_text(text)
image_features = image_features / image_features.norm(dim=-1, keepdim=True)
text_features = text_features / text_features.norm(dim=-1, keepdim=True)
probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print(probs)You are indexing 1,000,000 photos with CLIP ViT-L/14 and a 50 ms query budget. What do you store, how big is it, and what does a query cost?
The batch-arithmetic question and the zero-shot question are the two you will be asked in a design review. The SigLIP question separates the families; the prompt question is the one that decides whether your label-free classifier lands at 88% or 90%.
0 / 6 answered · 0 correct
01CLIP's contrastive loss is symmetric (image-to-text + text-to-image). Why both directions?
02Zero-shot classification with CLIP works by…?
03SigLIP replaces CLIP's softmax with a per-pair sigmoid. What does that buy?
04A practitioner reports 88% zero-shot top-1 on CIFAR-10 with CLIP ViT-B/32 using one prompt per class, and 90% with the same model using 80 averaged templates. Why does averaging help?
05Why do modern VLMs (LLaVA, Qwen-VL, InternVL) use a CLIP-family vision encoder instead of a supervised ImageNet ResNet?
06A batch of 32,768 (image, caption) pairs gives how many positives and negatives, and what does a freshly initialised CLIP pay?
Click a card to swap the lazy description for what it actually means.
Three problems with exact numbers — run OpenCLIP zero-shot on CIFAR-10 with the 80-template set, quantify what template averaging buys, and build a text→image retrieval index with a hand-written query set that includes the compositional cases the model is known to fail.
The whole pipeline is 20 lines: model, _, preprocess = open_clip.create_model_and_transforms('ViT-B-32', pretrained='laion2b_s34b_b79k'); tokenizer = open_clip.get_tokenizer('ViT-B-32'). Build the classifier once — for each of the 10 classes take the 80 template embeddings, average them, and re-normalise, giving a (10, 512) matrix; the text tower only runs 800 times in total. Then loop the 10,000 CIFAR-10 test images (32×32, upsampled to 224 by the preprocessing transform), encode each to a (512,) vector, and compute image @ text.T with both sides normalised. Report top-1 and the per-class confusion: the errors cluster between visually close classes (cat↔dog, deer↔horse, automobile↔truck). Useful checks along the way: the single default template 'a photo of a {}' typically costs 1–3 points against the ensemble, and the logits should be scaled by 100 (τ = 0.01, the ceiling the paper clips to) before the softmax — skip the scaling and the predictions flatten, which is the temperature lesson in one bug.
Run the pipeline twice with the same image encodings; only the (C, d) text matrix changes. Expect the ensemble to win by roughly 1–3 points on CIFAR-10 (the paper reports ~1.3 from the default template over the bare label and ~3.5 more from ensembling on ImageNet). Mechanically: for class c the ensemble stores T_c = normalise(mean over t of normalise(enc(text_t))). Averaging unit vectors cancels the template-specific components — if each template is the concept plus a random offset, the mean's offset shrinks by about √80 ≈ 8.9× — so T_c is a more robust centroid of the concept in text space, and the cosine to a real image is less sensitive to any one phrasing's quirks. Two traps worth reporting: re-normalise after averaging (the mean of unit vectors has norm < 1, and forgetting the re-normalisation silently shrinks every logit), and expect the ensemble to win on the *average*, not on every image — in a 10-class problem a favourable single template can beat it for a particular picture, which is exactly why nobody tries to pick the best template without labels. The lesson's prompt lab shows the same shape at toy scale: mean single template 0.149 average margin, ensemble 0.171.
Encode once, query often. With ViT-L/14 the gallery is 1,000 × 768 float32 = 3,072,000 bytes ≈ 2.93 MiB — it fits in memory with room to spare, and the same arithmetic scales linearly (1M images = 2.86 GiB). A FAISS IndexFlatIP on L2-normalised vectors is exact and needs no training; the query is one text encode plus 768 dot products per gallery item, i.e. 1.5 GFLOP for a million images, which is why a 50 ms budget is realistic. Write the 20 queries before you look at the results and split them by type: object queries ('a red car'), attribute queries ('a fluffy white dog'), scene queries ('a kitchen with a wooden table') and the adversarial ones the lesson warns about — counting ('three dogs'), negation ('a dog without a leash') and relations ('a cat on top of a car'). Expect recall@5 near 1.0 for the first group, high for the second and third, and near chance for the last. Report the failures with the retrieved images; the honest conclusion is that recall@5 for the queries the model can represent is very high, and the queries that fail are the compositional ones — which is a property of the objective, not of your index.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 04, Lesson 18) and the Math Foundations Notebook reference build. The five labs — the canvas shared-space explorer (8 pairs, the N × N matrix, the symmetric loss at any temperature), the canvas zero-shot classifier (six class prompts, an enable/disable class set, cosine then softmax), the prompt-wording board (seven recipes scored two ways), the model-family chooser and the blind-spots board — are original to this page, as is the arithmetic they compute: the contrastive batch bookkeeping (8 → 8 and 56; 256 → 256 and 65,280; 32,768 → 32,768 and 1,073,709,056) with the log N baseline (2.079 for 8, 10.397 for 32,768), the exact ViT-L/14 parameter budget read off the released checkpoint's tensor shapes (24 × 12,596,224 vision blocks; 303,179,776 + 123,060,480 + 1,376,257 = 427,616,513), B/32's 151,277,313 and B/16's 149,620,737, the temperature's journey (init 0.07 = logit scale 2.6592, clipped at 100, the released ViT-L/14 checkpoint's learned logit scale exactly ln(100) = 4.605170 so τ = 0.01, and SigLIP base/16's 4.765 with a bias of −12.93), the retrieval-index arithmetic (3,072 B per image; 2.93 MiB for 1,000; 2.86 GiB for a million; 1.5 GFLOP per query), the prompt-rung margins (bare 0.084 → default 0.127 → ensemble 0.168 on the dog image, and 0.149 mean template vs 0.171 ensemble across the six toy images) and the blind-spot gaps (word order 0.000, counting 0.002, negation 0.009, spatial 0.010, fine-grained 0.036, printed-vs-handwritten 0.212). The toy two-tower model and the bag-of-concepts stand-in are labelled teaching models wherever they are used; every paper number shown (76.2%, 71.0%, 84.5%, 78.0%, 88.3%, 400M pairs, batch 32,768, MNIST 88%, 21 of 27 datasets) is verified in the prose.