A retrieval system is three moves: turn the query into a vector, look that vector up in an index, and rank the catalogue by distance. The product decision hides in the third move — distance has to mean what your application means — and metric learning is the discipline of shaping the space until it does.
The query — image or text — goes through the same encoder the catalogue did. Every catalogue item becomes one vector; the index stores those vectors and returns the nearest ones. The encoder decides what “similar” can possibly mean; the index only decides how fast you find it.
query → encoder → embedding → index → top-k by cosineTriplet loss pulls a positive in and pushes a negative out past a margin; contrastive loss does it with pairs; proxy-based losses score against one learned prototype per class and skip mining entirely. All of them do the same job: move the encoder until distances agree with your labels.
L = max(0, d(a,p) − d(a,n) + margin) · margin 0.2–0.5recall@K is the fraction of queries whose correct item appears anywhere in the top K — a per-query 0 or 1, averaged over a held-out query set. Report @1, @5 and @10 together: a high @10 with a low @1 means the structure is right and only the ordering is noisy, and a re-ranking stage fixes it.
recall@1 = 0.42 · recall@5 = 0.86 · recall@10 = 0.95 → re-rankRetrieval is ranking, not labelling: embed the query and the whole catalogue into one space, index the catalogue for fast nearest-neighbour search, and measure with recall@K on queries the index has never seen. Metric learning is what makes the distances in that space mean your product’s notion of similar.
By the end you will be able to draw the embed → index → rank pipeline and say which decision each stage owns; pick between triplet, contrastive and proxy losses for a given dataset; state the triplet formula in both distance conventions and explain why semi-hard mining beats easy and hardest; normalise before you compare and show with numbers why cosine and raw dot product disagree; compute a FAISS index’s memory by hand (1M × 512 × 4 B = 2.048 GB) and choose flat vs IVF vs IVF-PQ; and report recall@{1,5,10} on a held-out query set without the self-match leak.
Duplicate detection, reverse image search, “find similar products”, face re-identification — the product question is always the same: given this query, rank my catalogue. Underneath, every one of them is embed → index → rank, and each of the three moves owns a decision.
Start with the question a classifier cannot answer. A classifier takes an image and returns one label from a fixed list; a retrieval system takes a query and returns an ordering of your entire catalogue. That is a different contract: nothing is collapsed to a single answer, so the system can be right about the top item and useful even when it is wrong about ranks 4 and 5.
The pipeline has exactly three moves. Embed: one encoder turns an image (or a text query) into a vector of 384–1024 floats. Index: the catalogue’s vectors are stored in a structure built for nearest-neighbour search. Rank: the query vector’s nearest neighbours, by cosine similarity or Euclidean distance, become the results.
The arithmetic is small enough to hold in your head. A million-image catalogue at 512 dimensions is 1,000,000 × 512 × 4 B = 2,048,000,000 B ≈ 2.048 GB of float32 vectors — about 1% of the ~200 GB of raw JPEGs those images occupy, and the only part the search ever touches. One query is a single vector of 512 floats, so a brute-force ranking is a million dot products of length 512 — half a billion multiply-adds per query, before any index.
Notice what is not in the arithmetic: the size of the images. Retrieval cost is set by the embedding dimension and the catalogue count, which is why 384-d backbones are popular at scale and why product-quantised indexes (chapter 5) are the difference between a server and a data centre.
Two of the three moves are the same in every production system, which is exactly why they stopped being the interesting part. In 2026 you reach for DINOv2 or CLIP for the embedding and FAISS (or a managed vector database wrapping it) for the index. That raises the bar: the hard part is deciding what counts as similar for your application, then shaping the space so distances agree with it. A sneaker query that returns the same sneaker in a different colour is a success for one product and a failure for another; no library can make that call for you.
The distinction also decides which query types you can serve. An image-only encoder answers “find images like this image”. A text query needs text and images in one space, which is a training objective (CLIP/SigLIP), not an index setting — chapter 7 compares the backbones.
Every metric-learning loss is asking for one thing: move the encoder until similar things are close and different things are far. They differ in what labels they need, how many negatives each step sees, and how much mining you have to do to keep the gradient alive.
Contrastive loss is the minimal version. You have pairs that should be close — two augmentations of one image, two photos of one product — and everything else in the batch implicitly plays the negative. It is easy to collect pairs (augmentation labels them for free) and it works with any dataset. The catch is that a loss with one positive and one negative per item learns slowly: gradients are small and noisy, and most of the batch information goes unused. Contrastive pretraining compensates with enormous batches.
Triplet loss makes the negative explicit. You choose an anchor, a positive and a negative, and the loss enforces a margin between the two distances. This is the loss with the clearest intuition, and the one with the clearest bill: the number of possible triplets explodes combinatorially. Take a batch with P = 16 identities and K = 4 images each: 64 anchors, 3 positives and 60 negatives per anchor, which is 11,520 valid triplets. Almost all are easy and contribute nothing, so training without mining is like studying with the answer key face down. Chapter 3 is the triplet chapter.
NT-Xent / InfoNCE is the softmax version: an anchor scores every other item in the batch, and the loss is cross-entropy over those similarities with one correct answer. Because every batch item is a negative for every anchor, batch size is negative count — SimCLR uses 4096+ (4094 negatives per anchor; a batch of 128 gives only 127) — or you keep a queue of old embeddings (MoCo) so that batch size stays modest while the negative pool does not. This is the objective inside CLIP and SigLIP: two encoders, image and text, each batch item’s caption as the correct pairing.
Proxy-based losses (ProxyNCA, ProxyAnchor) remove the mining problem entirely. Every class gets one learned proxy vector; a sample is pulled toward its own class’s proxy and pushed from the others with a softmax cross-entropy over proxy similarities. There are no pairs to construct and no negatives to search, so convergence is fast and stable, and training cost is independent of batch size. The risk moves elsewhere: proxies are free parameters, and on a small dataset a proxy can drift away from the class it is supposed to represent and take the whole cluster with it.
| Loss | Requires | Pros | Cons |
|---|---|---|---|
| Contrastive | (anchor, positive) pairs; negatives come from the batch | Simplest loss that works with any pair label | Converges slowly unless the batch (or a queue) supplies many negatives |
| Triplet | (anchor, positive, negative) triplets | Direct margin control; the loss you can reason about on paper | Hard-triplet mining is the expensive part, and it is mandatory |
| NT-Xent / InfoNCE | Pairs plus batch-mined negatives; a temperature τ ≈ 0.07–0.2 | Scales to huge batches; the objective behind CLIP and SimCLR | Needs a big batch or a momentum queue to have enough negatives |
| Proxy-based (ProxyNCA, ProxyAnchor) | Class labels only — no pairs, no triplets, no mining | Fast and stable; every step uses every sample | Can overfit the proxies on small datasets when a proxy drifts |
Your dataset has only class labels — no pairs, no instance ids. Which loss family fits with the least data engineering?
The triplet loss is one line of code and one page of arithmetic. The page is worth reading, because the choice of distance convention changes the number on your dashboard, and the choice of which triplets you show the loss decides whether training moves at all.
Three images: an anchor a, a positive p that should be near it, and a negative n that should not. The paper writes the loss with squared distances and margin alpha; the source code’s F.pairwise_distance is plain L2 with a default margin of 0.2. Both enforce the same ordering — the squared loss is a monotone function of the plain one for a fixed pair — but the numbers you quote are convention-specific:
Work the numbers by hand on unit-norm embeddings, where the identity ‖a−b‖² = 2 − 2·cos(a,b) converts cosines into distances. Take cos(a,p) = 0.85 → d²(a,p) = 0.30, d(a,p) = 0.548; and cos(a,n) = 0.78 → d²(a,n) = 0.44, d(a,n) = 0.663. The negative is further away than the positive, so this is a semi-hard triplet, and the hinge is open in both conventions:
Now the labels that matter. A triplet is easy when the negative is already outside the margin (d(a,n) ≥ d(a,p) + margin): the hinge is closed, the loss is exactly 0, and the gradient through it is exactly 0. It is hard when the negative is closer than the positive (d(a,n) < d(a,p)): large gradient, but the triplet is often an outlier or a mislabel, and a batch full of hard triplets can yank the encoder around. It is semi-hard when d(a,p) < d(a,n) < d(a,p) + margin: the hinge is open, the gradient is bounded by the margin, and every selected triplet carries information. That window is the FaceNet recipe, and the source code selects it literally:
import torch
import torch.nn.functional as F
def triplet_loss(anchor, positive, negative, margin=0.2):
d_ap = F.pairwise_distance(anchor, positive, p=2) # plain L2, not squared
d_an = F.pairwise_distance(anchor, negative, p=2)
return F.relu(d_ap - d_an + margin).mean() # the hinge
def semi_hard_negatives(emb, labels, margin=0.2):
dist = torch.cdist(emb, emb)
same_class = labels[:, None] == labels[None, :]
N = emb.size(0)
# hardest positive = the furthest same-class item
positives = dist.clone()
positives[~same_class] = float("-inf")
positives.fill_diagonal_(float("-inf"))
pos_idx = positives.argmax(dim=1)
# semi-hard: d(a,p) < d(a,n) < d(a,p) + margin
semi_hard = dist.clone()
semi_hard[same_class] = float("inf")
d_ap = dist[torch.arange(N), pos_idx].unsqueeze(1)
semi_hard[dist <= d_ap] = float("inf") # closer than the positive
semi_hard[dist >= d_ap + margin] = float("inf") # already outside the margin
neg_idx = semi_hard.argmin(dim=1)
# no semi-hard candidate in range: fall back to the hardest negative
fallback = semi_hard[torch.arange(N), neg_idx] == float("inf")
if fallback.any():
hardest = dist.clone()
hardest[same_class] = float("inf")
neg_idx = torch.where(fallback, hardest.argmin(dim=1), neg_idx)
return pos_idx, neg_idxDrag a, p and n (or select one and nudge it). The hinge opens only when the negative gets inside the margin band — and the loss number depends on which distance convention you write down.
d(a,n) is beyond d(a,p) but inside the margin — the FaceNet recipe
distance conventionmove pointPush the negative inside the dashed circle and the loss opens; push it far out and the gradient dies. That is all mining is: choosing which triplets to show the loss.
Count the triplets in one batch and the reason for mining becomes arithmetic rather than folklore.
This is why “batch all triplets” with soft-margin or a batch-hard variant is a common production compromise: use all of them, but shape the loss so most of the signal comes from the active region. Whichever you pick, log the fraction of triplets with an open hinge — if it sits near zero, your loss is not training the encoder, no matter what the dashboard number says.
A batch produces a triplet where d(a,p) = 0.31 and d(a,n) = 0.82, with margin 0.2 (plain L2). What does this triplet contribute?
Chapter 1 said the index is a commodity. That is only true if the vectors you put in it are comparable — and raw encoder outputs are not. The one-line fix, L2 normalisation, is the difference between ranking by similarity and ranking by whichever vector is loudest.
L2 normalisation divides a vector by its own length: v̂ = v / ‖v‖ with ‖v‖ = √Σvᵢ². Every normalised vector lands on the unit sphere, so its length is 1 and the only thing left to compare is direction. Cosine similarity is the inner product of two normalised vectors — and the general formula is just the inner product with the lengths divided out:
Work one example end to end. Take a = (3, 4) and b = (4, 3). Both have norm 5; the dot product is 12 + 12 = 24, so the cosine is 24/25 = 0.96. Normalise first and check: â = (0.6, 0.8), b̂ = (0.8, 0.6), â · b̂ = 0.48 + 0.48 = 0.96 ✓. The squared L2 distance between the normalised vectors is (0.6−0.8)² + (0.8−0.6)² = 0.08, and 2 − 2(0.96) = 0.08 ✓ — the identity in action. Same ordering, two monotone-equivalent numbers.
Now the failure. Skip the division and the inner product rewards length as much as direction: an item with cosine 0.6 but norm 2.0 scores 1.2, while an item with cosine 0.9 and norm 1.0 scores 0.9. The raw dot product ranks the less similar item first. This is not a corner case: embedding norms routinely vary 2–3× across samples, and the lab in this chapter is built so you can watch it happen. Its catalogue has five classes; class E’s vectors are 1.5–1.95 long while the rest sit at 0.9–1.0. For a query at 0°, cosine ranks the class-A sneakers 0.995 / 0.990 / 0.927 at ranks 1–3, while the raw dot product ranks class-E posters 1.494 / 1.492 / 1.115 above them, with class A starting at rank 4.
import torch
import torch.nn.functional as F
# normalise as the encoder's last step, so every consumer agrees
def embed(images, backbone):
with torch.no_grad():
v = backbone(images) # (N, d) raw features
return F.normalize(v, dim=-1) # (N, d), each row has ‖v‖ = 1
q = embed(query, backbone) # (1, d)
g = embed(gallery, backbone) # (N, d)
cos = q @ g.T # == cosine similarity, exact
topk = cos.topk(5, dim=-1) # top-5 by cosine
# the same ranking via L2 on normalised vectors:
# ‖q − g‖² = 2 − 2·cos(q, g) → top-k by cosine == bottom-k by L2
d2 = torch.cdist(q, g).pow(2) # (1, N)
assert torch.equal(cos.topk(5).indices, (-d2).topk(5).indices)
# FAISS has no separate cosine index: normalise, then IndexFlatIP.
# IndexFlatL2 on normalised vectors gives the same order, so either is fine —
# mixing a raw-dot index with unnormalised vectors is the mistake.The explorer lab’s default query is a new photo of sneaker A1, placed at angle 10°, with the catalogue’s A1 at 8°, A2 at 22° and A3 at −6°. Relevance now means two different things, and the metrics disagree — correctly.
Move the query and watch the ranking. Flip the metric between cosine and raw dot product: same points, same query, different top-3 — that is the normalisation lesson with nothing hidden.
The query “new photo of A1” is an instance query: only A1 itself is relevant there, while “same class” counts every sneaker. Try both relevance settings with the metric flipped.
Query q = (1, 0). Candidate P is (0.9, 0.436) — norm 1.0, cosine 0.9. Candidate R is (1.2, 1.6) — norm 2.0, cosine 0.6. Which one wins under each metric?
FAISS is the de-facto nearest-neighbour library, and its index families are a memory/recall/latency menu. The arithmetic that picks one for you starts with four bytes per dimension per vector.
Facebook AI Similarity Search stores vectors and answers “nearest” queries. Four families cover almost every production choice:
nlist cells (a coarse k-means), then search only the nprobe cells closest to the query. Exact within those cells, approximate overall, and it needs a train() pass on representative vectors before anything can be added.m bytes of code rather than 4d bytes of floats. This is the only family that fits 100M vectors in a normal server’s RAM, and the compression caps recall below 1.M neighbours per node). The default in most managed vector databases.The rule of thumb the source states in one line: up to about 1M vectors you probably want IndexFlatIP on normalised vectors; at 10M, IndexIVFFlat; at 100M-plus, IVF combined with product quantisation. Two settings make IVF behave:nlist = 4·√N cells (1M → 4000) and a search that probes 1–5% of them (nprobe = 32–128 at nlist 4000). Fewer probes is faster and less accurate; probing everything turns IVF back into flat search with extra bookkeeping.
Read the last column carefully: PQ does not shrink the vectors, it replaces them. Each 512-d vector becomes 64 bytes — the ids of its nearest centroid in each of 64 eight-dimensional sub-spaces — so distances are computed from reconstructed approximations. Recall is therefore capped by the compression, not by how hard you search: more nprobe finds more candidates, but candidates the codes cannot score correctly are still mis-scored. At 32× compression the cap sits around 0.85 recall@10; the fix when you need the last points is a re-ranking pass that scores the top candidates with the original vectors.
import faiss
import numpy as np
d = 512
gallery = np.random.randn(1_000_000, d).astype("float32")
faiss.normalize_L2(gallery) # in place: cosine == inner product
# 1. exact, no training: the 1M default
index_flat = faiss.IndexFlatIP(d) # IP after normalising == cosine
index_flat.add(gallery)
# 2. partitioned: train on a sample, then add everything
quantiser = faiss.IndexFlatIP(d)
index_ivf = faiss.IndexIVFFlat(quantiser, d, 4000, faiss.METRIC_INNER_PRODUCT)
index_ivf.train(gallery[:200_000]) # train() learns the 4000 coarse centroids
index_ivf.add(gallery)
index_ivf.nprobe = 32 # search 32 of 4000 cells ≈ 0.8% of vectors
# 3. quantised: m = 64 subquantisers → 64 B per vector, 32× compression
index_pq = faiss.IndexIVFPQ(quantiser, d, 4000, 64, 8)
index_pq.train(gallery[:200_000]) # PQ codebooks need training too
index_pq.add(gallery)
index_pq.nprobe = 32
query = np.random.randn(1, d).astype("float32")
faiss.normalize_L2(query)
for name, index in [("flat", index_flat), ("ivf", index_ivf), ("ivfpq", index_pq)]:
scores, ids = index.search(query, 10) # top-10 by cosine
print(name, ids[0][:5], scores[0][:5])
# a text query is the same three lines: encode the text with the image-text
# model (CLIP / SigLIP), normalize_L2, index.search. The index never knows
# which modality the query vector came from — the shared space is the encoder's job.Same catalogue, four FAISS index families. Memory is exact byte arithmetic; recall and latency are an illustrative model — the point is the order of magnitude and the trade, not your benchmark.
IVF plus product quantisation: vectors become m bytes of codes. The only family that fits 100M in RAM.
| Component | Bytes |
|---|---|
| PQ codes · 1000000 × 64 B | 64,000,000 (64.0 MB) |
| inverted-list ids · 1000000 × 8 B | 8,000,000 (8.0 MB) |
| coarse centroids · 4000 × 512 × 4 B | 8,192,000 (8.2 MB) |
| PQ codebooks · 256 × 512 × 4 B | 524,288 (524.3 KB) |
| Total | 80,716,288 (80.7 MB) · 80.7 B per vector |
Change N by one row and watch the exact number: 1M × 512 × 4 B is 2,048,000,000 B, and 10M is simply ten of those. PQ does not shrink the vectors — it throws away the vector and keeps an m-byte code, which is why recall has a ceiling no nprobe can lift.
A 100M-image catalogue at 512 dimensions must be searchable from RAM on a single server. Which index family fits, and what does it cost?
Recall@K is the standard retrieval metric and the easiest one to accidentally inflate. Getting it honest takes two disciplines: a query set the index has never seen, and a relevance definition you wrote down before you looked at the numbers.
The definition fits in one line: recall@K is the fraction of queries with at least one correct match in the top K results. For a single query the value is 0 or 1 — the right item either made the cut or it did not — so the number in a report is always a mean over queries. Run 100 held-out queries; suppose 62 have a correct item at rank 1, 86 somewhere in the top 5 and 95 somewhere in the top 10. Then recall@1 = 0.62, recall@5 = 0.86, recall@10 = 0.95, and each of those is the mean of 100 zeros and ones:
Report @1, @5 and @10 together, because the pattern is a diagnosis. A system at recall@10 = 0.95 but recall@1 = 0.42 has the right structure — the relevant items are nearly always in the shortlist — and a noisy ordering inside it. That is a re-ranking problem: score the top 50–100 candidates with an expensive second-stage model and the ordering sharpens without touching the embedding or the index. Pinterest and Google Photos both run two-stage pipelines for exactly this reason. A system that is low at every K has a different problem: the embedding space itself does not separate your notion of similar.
The two ways to lie are both about the query set. First, the self-match: if the query image is also in the gallery, its own vector is the nearest neighbour with cosine 1.0, so it takes rank 1 and recall@1 = 1.000 for every query, forever. This is a legitimate sanity check — the source’s hard exercise asks for it and expects exactly 1.0 — but it is not a measurement. Query with a disjoint image; if your product really does retrieve from the same set (duplicate detection), then the correct metric is precision@K, because every false positive is a user-visible mistake. Second, querying the training split: an encoder that memorised it will look excellent and mean nothing. The held-out split has to hold out queries, and for instance retrieval it should hold out whole identities or sessions, not single photos.
import torch
def recall_at_k(query_emb, gallery_emb, query_labels, gallery_labels, k=1):
# both sides already L2-normalised: inner product == cosine
sim = query_emb @ gallery_emb.T # (Q, N)
_, top_k = sim.topk(k, dim=-1) # (Q, k) indices
matches = (gallery_labels[top_k] == query_labels[:, None]).any(dim=-1)
return matches.float().mean().item() # mean of Q zeros/ones
for k in (1, 5, 10):
print(f"recall@{k}: {recall_at_k(q_emb, g_emb, qy, gy, k=k):.3f}")
# the split that makes the number honest:
# gallery = the catalogue being searched
# queries = held-out items (disjoint from the gallery), each with a label
# relevance = same class for category retrieval, same instance id for
# instance retrieval — write this definition down firstToggle which results are relevant and read the arithmetic. The self-match switch is the classic evaluation mistake: put the query in the gallery and recall@1 becomes free.
Recall is about presence: did the right thing make the top K at all? Precision@5 asks a different question — how much of the top 5 you would show a user is actually right.
When the number needs to be comparable across teams, fix three things in writing: the query set (which items, held out how), the relevance definition (same class? same instance? same product family?), and the K values reported. Two teams can both honestly report “recall@10 = 0.9” for the same model and disagree by 20 points because one counted same-class items as relevant and the other required the exact instance.
Your retrieval system reports recall@10 = 0.95 and recall@1 = 0.42 on a 500-query held-out set. What is the most useful next step?
Text queries, image queries and instance-level queries need different encoders, and the choice is almost mechanical. The rest of this chapter is the production recipe that turns a pretrained backbone into an instance-retrieval system.
The backbone is the part of the system users never see and the part that decides what “similar” can possibly mean. Three families dominate, and each answers a different query type:
A caveat before the table: all three are trained to agree about categories and captions. None of them is trained to distinguish two units of the same product, or two faces, or two photographs of the same building in different light. When your query means which one rather than what, the frozen backbone underperforms no matter which row you pick — that is the point at which metric learning earns its keep.
| Backbone | Embedding dim | Query side | Wins when |
|---|---|---|---|
| DINOv2 (ViT-S / B / L / g) | 384 / 768 / 1024 / 1536 | image → image | General visual similarity off the shelf: duplicate detection, similar-product search, clustering a catalogue, the default image-side embedding. |
| CLIP (ViT-B/32, ViT-L/14) | 512 / 768 | text → image and image → image | Any query that is text. The shared space comes from contrastive training on ~400M image-caption pairs, so no fine-tune is needed to compare a sentence with an image. |
| SigLIP (Base, So400m) | 768 / 1152 | text → image and image → image | The stronger text-image default when you can afford it: the sigmoid loss removes CLIP's need for a global batch-wise normalisation, and the larger checkpoints lead zero-shot benchmarks. |
| Supervised ImageNet ResNet / EfficientNet | 512–2048 | image → image | A baseline, not a destination: features are biased toward the 1,000 training classes, so “similar” quietly means “shares ImageNet labels”. Useful as a cheap control in an ablation. |
| Fine-tuned DINOv2 + embedding head | usually projected to 128–512 | image → image, instance-level | This exact SKU, this face, this car: the only row that reliably separates near-duplicate instances, and the result of the triplet / InfoNCE recipe below. |
For state-of-the-art instance retrieval the recipe is short and consistent: DINOv2 backbone, add an embedding head, fine-tune with a triplet or InfoNCE loss on instance-labelled pairs, index in FAISS, evaluate recall@K on held-out identities. The head is usually one or two linear layers projecting 768 or 1024 dims down to 128–512; the fine-tune is short (a few epochs) and only viable when instance labels exist. When they do not, the fallback is to mine positives from augmentations or from near-duplicate clusters in the off-the-shelf space — a self-supervised bootstrap of the same recipe.
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.optim import Adam
class Encoder(nn.Module):
def __init__(self, in_dim=128, emb_dim=64):
super().__init__()
self.net = nn.Sequential(
nn.Linear(in_dim, 128), nn.ReLU(),
nn.Linear(128, emb_dim),
)
def forward(self, x):
return F.normalize(self.net(x), dim=-1) # normalise inside the model
torch.manual_seed(0)
num_classes = 6
protos = F.normalize(torch.randn(num_classes, 128), dim=-1)
def sample_batch(bs=32):
labels = torch.randint(0, num_classes, (bs,))
x = protos[labels] + 0.15 * torch.randn(bs, 128)
return x, labels
enc = Encoder()
opt = Adam(enc.parameters(), lr=3e-3)
for step in range(200):
x, y = sample_batch(32)
emb = enc(x) # (32, 64), unit norm
pos_idx, neg_idx = semi_hard_negatives(emb, y)
loss = triplet_loss(emb, emb[pos_idx], emb[neg_idx])
opt.zero_grad(); loss.backward(); opt.step()
enc.eval()
with torch.no_grad():
gx, gy = sample_batch(200) # gallery
g_emb = enc(gx)
qx, qy = sample_batch(50) # held-out queries
q_emb = enc(qx)
for k in (1, 5, 10):
print(f"recall@{k}: {recall_at_k(q_emb, g_emb, qy, gy, k=k):.3f}")
# replace the toy Encoder with a DINOv2 checkpoint, the sampled protos with
# your catalogue, and the labels with instance ids — the loop does not change.Type a query and read the cosine table. This is a 24-dim toy space with 10 hand-named concept axes — real CLIP learns its axes from ~400M pairs — but the arithmetic is the arithmetic.
| Rank | Image | Cosine | Concept loadings |
|---|---|---|---|
| #1 | golden retriever on a beach · img-01 | 0.880 | dog 1.0 · nature 0.4 |
| #2 | mountain lake at dawn · img-04 | 0.647 | nature 1.0 · snow 0.2 |
| #3 | sunflower field in summer · img-09 | 0.303 | flower 1.0 · nature 0.5 |
| #4 | snowboarder mid-jump · img-10 | 0.117 | snow 1.0 · nature 0.2 |
| #5 | city skyline in the rain · img-07 | 0.086 | city 1.0 · nature 0.1 |
| #6 | yellow sneakers on concrete · img-06 | 0.010 | shoe 1.0 |
| #7 | red sports car at night · img-03 | 0.008 | car 1.0 · city 0.3 |
| #8 | vintage camera on a desk · img-08 | 0.007 | camera 1.0 · furniture 0.3 |
| #9 | wooden chair in a studio · img-05 | 0.001 | furniture 1.0 |
| #10 | tabby cat on a sofa · img-02 | -0.006 | cat 1.0 · furniture 0.5 |
Try “sneakers” with the shared space on: the yellow-sneakers image should win by a wide margin. Then flip to separate encoders and watch a different image take rank 1 with a similar-looking score — that is why the backbone choice chapter says text queries need CLIP or SigLIP, not a stronger image-only model.
Answer before you look. The semi-hard question and the recall@10 vs recall@1 question are the two that separate having read the chapter from being able to defend a retrieval system in review.
0 / 5 answered · 0 correct
01A visual-search product takes text queries and returns gallery images. Which backbone do you reach for first?
02What does “semi-hard mining” mean in triplet-loss training?
03Your retrieval system reports recall@10 = 0.95 but recall@1 = 0.42. What should you conclude?
04Cosine similarity computed on unnormalised embeddings is not cosine similarity. What actually goes wrong?
05Between instance-level retrieval (“find this exact car”) and category-level retrieval (“find cars”), which one actually requires metric-learning fine-tuning?
Click a card to swap the lazy description for what it actually means.
Four problems with exact numbers — run the toy training loop, swap the triplet for proxies, build a DINOv2 + FAISS index over ImageNet samples and report honest recall@K, and do the index arithmetic for a 1M × 768 catalogue under a 16 GB / 50 ms budget. Try first; a worked answer is one click away.
The source's recipe: protos = F.normalize(torch.randn(6, 128), dim=-1); sample x = protos[labels] + 0.15 * torch.randn(bs, 128); a 128 → 128 → 64 MLP whose forward returns F.normalize(net(x), dim=-1); Adam at lr = 3e-3 for 200 steps of batch 32–48. Before training the encoder is random, so the six classes overlap in the 2-D PCA plot. After training, intra-class distances shrink toward 0 and the classes separate: the first principal components explain most of the between-class variance, and the six blobs are compact. Print the triplet loss every 40 steps — it starts near the margin value and falls as most triplets become easy and only the mined ones contribute. Then run the recall@K block on 200 gallery and 50 fresh query samples: even this 5-line loss gets recall@1 well above chance, which is the point of the exercise.
Add self.proxies = nn.Parameter(F.normalize(torch.randn(num_classes, emb_dim), dim=-1)) to the model, L2-normalise them each step, then compute logits = emb @ proxies.T / temperature (temperature ≈ 0.1) and use F.cross_entropy(logits, labels). Every sample now compares against all C proxies, so a batch of 32 contributes 32 × 6 comparisons with no triplet search. Expect proxies to reach a useful clustering in noticeably fewer steps: the gradient on each proxy is a softmax-weighted average of the whole batch, so each step uses all the information rather than one mined negative per anchor. The cost is a bias toward whatever the proxies currently encode — with 6 classes and a few hundred samples that is fine, but on a small dataset a proxy can drift and drag its class with it. Run both losses with the same seed and print loss every 20 steps to see the difference.
Embed with a DINOv2 checkpoint (for example facebook/dinov2-base; the [CLS] token is the embedding), F.normalize each vector, and faiss.IndexFlatIP(768) for a 768-d backbone. Querying with the catalogue itself should give recall@1 = 1.000 exactly — the top-1 hit is the query's own vector with cosine 1.0 — which is the sanity check the source asks for and a warning if you report it as a result. For the held-out split: take a disjoint set of query images, define relevance as “same ImageNet class as the query”, exclude the query itself from the gallery, and compute recall@K with the sim.topk code. Expect high recall@5/@10 and much lower recall@1, because ImageNet classes contain visually diverse instances and generic features cluster by scene and texture more than by label. That gap is the lesson in one number: the space has category structure, and label-level ordering is noisy.
Flat: 1,000,000 × 768 × 4 B = 3,072,000,000 B ≈ 3.07 GB of vectors, and a single query scans all of it — at an effective 20 GB/s that is ~154 ms before any other work, so flat misses the latency budget on memory alone. IVF-PQ with m = 96: codes 1,000,000 × 96 B = 96 MB, inverted-list ids 8 MB, coarse centroids (nlist = 4√N = 4000) × 768 × 4 B ≈ 12.3 MB, PQ codebooks 256 × 768 × 4 B ≈ 0.8 MB — about 117 MB total, 26× smaller, and a query at nprobe = 32 scans ~0.8% of the codes, well under 50 ms. What you give up is recall: the compression ratio is 768 × 4 / 96 = 32×, and at 32× the codes are lossy enough to cap recall@10 near 0.85 no matter how large nprobe grows, so the last 10–15% of hits need either more subquantisers (m = 128 costs 128 MB) or a re-ranking pass with the raw vectors for the top candidates. For 100M: flat is 307 GB — impossible — while IVF-PQ m = 96 is roughly 10.5 GB (9.6 GB of codes + 0.8 GB of ids + 0.12 GB of centroids at nlist = 40,000), which fits a 16 GB box with room for the rest of the process. The rule the lesson states in one line: up to ~1M use flat, 10M use IVF, 100M-plus use IVF + PQ.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 04, Lesson 20) and the Math Foundations Notebook reference build. The five labs (the canvas embedding-space explorer, the canvas triplet playground, the canvas recall@K calculator, the index comparator and the text-to-image query board) are original to this page, as are the catalogue arithmetic (1M × 512 × 4 B = 2.048 GB, 512M MACs per brute-force query), the worked triplet checks in both conventions (the same three points → loss 0.085 plain vs 0.060 squared; margin 0.2 as a cosine gap of 0.10 squared vs ≈0.14 plain), the batch-all triplet count (P = 16, K = 4 → 64 anchors and 11,520 triplets), the normalisation flip with exact scores (cosine A3 0.995 vs raw dot E3 1.494, class E norms 1.5–1.95), the 2 − 2·cos identity checked on (3,4) and (4,3), the FAISS memory tables (IVF-Flat 2,064 MB, IVF-PQ 80.7 MB → 25× smaller, HNSW 2,330 MB, 100M → 7.3 GB), the recall@K honesty rules (per-query 0/1, Q beside the number, the self-match leak), the WHAT vs WHICH memory hook, the fourth exercise on index choice under a RAM/latency budget, and the production-recipe framing in the backbone chapter. Every number shown is computed live by the labs or verified by hand in the prose.