A prediction, a target, and one number that says how wrong the guess was. That number — the loss — is the only thing training can see. Its shape decides what the model fears: big misses, confident mistakes, or the rare examples it would otherwise learn to ignore.
The optimizer never sees accuracy, F1 or AUC — it sees one differentiable number and takes its gradient. A model trained on the wrong loss satisfies it the cheapest way possible: predict 0.5 for everything, collapse every embedding to a single point, or ignore the rare class entirely. Choosing a loss is choosing which failure mode you can live with.
if the loss is wrong, the model games itSquared error hunts the loudest miss; absolute error treats every miss alike; cross-entropy charges a fortune for confident mistakes and almost nothing for confident correct answers. Huber splits the difference, focal loss turns the volume down on easy examples, and contrastive loss learns from similarity when there are no labels at all.
the penalty's shape is the policySquared error drives predictions to the conditional mean; absolute error to the median; cross-entropy to calibrated probabilities. Label smoothing caps how confident the model is allowed to be, focal loss re-weights who gets the gradient budget, and KL divergence against a teacher copies its uncertainty. Change the loss and you change the definition of success.
loss = the definition of “good prediction”The loss is the syllabus, not the scoreboard: the optimizer reads nothing else, so the model studies exactly the grading rule you wrote — which is why a squared error becomes a median-chasing regression, cross-entropy becomes a decisive classifier, and a thoughtless contrastive loss produces a model that matches everything to everything.
By the end you will be able to score one batch under MSE, MAE, Huber, binary cross-entropy and categorical cross-entropy by hand; explain why −ln p punishes confident mistakes and why the softmax + cross-entropy pair hands back the gradient p − y; use label smoothing, focal loss and InfoNCE for the jobs plain cross-entropy cannot do; and read a loss curve like a diagnostic — including the times a lower loss means a worse model.
The optimizer never sees accuracy, F1 or your manager’s dashboard. It sees one differentiable number — the loss — and takes its gradient. Whatever that number rewards, the model will learn to do, including finding loopholes you never intended.
Training is a loop with four moves: run the forward pass, score the prediction with the loss function, run the backward pass, nudge every weight downhill. The loss is the only place where “what we want” enters that loop. Accuracy cannot fill that role: nudge a weight by a tiny amount and almost no predicted label flips, so accuracy’s gradient is zero almost everywhere — a staircase, not a slope. A loss is a smooth function of the prediction, which is why every learning signal in deep learning is a loss gradient first.
The consequences show up immediately in a balanced binary task with a model that cannot tell the classes apart. The best constant prediction under MSE is the base rate — 0.5 for every input, for a loss of 0.25, which is the minimum a model with no discriminative power can reach. Cross-entropy charges −ln 0.5 = 0.693 for the same answer and refuses to be satisfied: it keeps asking for a decision. The levels differ, but the slopes differ more, and slopes are what moves weights. Once a sigmoid sits in front, MSE’s signal to the logits is 2(p − y)·p(1 − p) — and the factor p(1 − p) collapses exactly when the model is confidently wrong. Cross-entropy’s logit-gradient is just p − y, still near full strength at the same moment.
This is why choosing a loss is not a formatting decision. Every loss has an equilibrium — the cheapest way to satisfy it — and the model will find that equilibrium whether or not it is useful. MSE settles for the mean. MAE settles for the median and shrugs at outliers. Cross-entropy settles for calibrated probabilities. A naive contrastive loss can settle for collapsing every embedding to a single point: technically zero loss, completely worthless. Picking a loss means picking which failure mode you can live with.
One example, one prediction. Move p and watch the two losses charge different bills — then watch the gradients the optimizer actually receives once a sigmoid sits in front of the loss.
The bars show the size of the signal arriving at the weights. Both losses point in the same direction here — the difference is how much of the signal survives the sigmoid. When the prediction is confidently wrong, MSE’s bar is a sliver (σ′ has flattened and multiplied it away) while cross-entropy’s stays near full strength. That is the whole argument of this chapter in one picture.
Reference point: a model that outputs a constant 0.5 on a balanced binary task pays MSE = 0.25 — the best constant — but cross-entropy charges ln 2 = 0.693 and keeps asking for a decision.
Why can't a network be trained directly on accuracy?
Regression predicts a number, so the error is a number too: the residual r = ŷ − y. A regression loss is a policy that converts each residual into a cost — and the shape of that policy decides which mistakes the model spends its effort on.
Mean squared error averages r² over the batch. Squaring is the whole personality of the loss: an error of 2 costs 4× as much as an error of 1, and an error of 10 costs 100×. The gradient with respect to a prediction is 2r/n — linear in the error, so bigger misses get bigger corrections. For clean regression that is a feature: the worst examples deserve the most attention. For dirty regression it is a liability, because the single loudest voice in the batch is often a typo.
Mean absolute error averages |r|. Twice as wrong is twice as costly, not four times, and the gradient is ±1 everywhere — an outlier cannot shout louder than an ordinary miss. MAE pays for that calm with a kink at zero (the derivative changes sign abruptly) and a thinner signal near the target. Huber loss negotiates: quadratic while the miss is small (|r| ≤ δ), linear once it is large. It keeps MSE’s responsiveness near the target and MAE’s immunity far from it, with δ deciding where the switch happens.
That last pair of lines is a quiet but important fact: the minimum of the loss is the definition of a good prediction. Squared error drives predictions toward the conditional mean of the targets; absolute error drives them toward the conditional median; with the four targets above the mean is 2.25 and the median is 0, and each loss prefers its own answer. Swap the loss and you have changed what the model is trying to learn.
The source’s Step-1 batch: predictions [0.9, 0.1, 0.7, 0.4], targets [1, 0, 1, 0], residuals [−0.1, +0.1, −0.3, +0.4]. Score the same four numbers under every regression loss in this chapter.
Read the MSE gradient row and then the mansion arithmetic: both are the same fact. A quadratic amplifier hands the gradient budget to the largest residual, and a single bad label can therefore own most of the update. Huber caps the amplifier at δ; MAE removes it entirely.
Drag the residual and watch MSE, MAE and Huber disagree about what the same miss is worth — and how loudly each one pulls the prediction back.
MSE’s gradient grows without bound, so one wild residual can dominate an entire batch. MAE’s gradient is ±1 everywhere: it never panics, but it also cannot tell a small miss from a moderate one — and it has a kink at zero. Huber is the negotiated settlement.
Your regression dataset has one label typo with a residual of +50 while genuinely difficult examples sit around ±1. Which loss is most distorted by the typo, and by how much?
A classifier does not output a label — it outputs a probability distribution. Cross-entropy scores that distribution by asking one question: how surprised should you have been by the truth? The answer is measured in nats, and the bill grows without bound when confidence and correctness disagree.
For a two-class problem the formula is BCE = −(y·ln p + (1 − y)·ln(1 − p)). Read it in plain English: the true label chooses which logarithm you pay. If the answer is 1 you pay −ln p, the negative log of the probability you gave the truth; if the answer is 0 you pay −ln(1 − p). One of the two terms always vanishes. The log is what makes the loss information-theoretic: −ln p is the surprise of an event you assigned probability p, in nats — the same quantity you met as entropy in Phase 1, Lesson 09. Assigning 0.99 to the truth costs almost nothing; assigning 0.01 costs 4.6.
The asymmetry is the entire point. Watch what happens as the model becomes more confident in the right answer: the bill shrinks 0.693 → 0.105 → 0.010, with rapidly diminishing rewards. Now watch the same sweep in the wrong direction: 0.693 → 2.303 → 4.605, with no ceiling. A classifier that says 0.01 on the true class pays 458× what it would have paid for saying 0.99. That ratio is why cross-entropy produces classifiers that are decisive rather than merely correct — and it is exactly what MSE lacks, since MSE’s worst possible binary charge is (0 − 1)² = 1 and its probability gradient never exceeds 2.
For multi-class problems, categorical cross-entropy keeps only one term: CCE = −ln p_true. Random guessing among ten classes costs ln 10 = 2.303; getting the right class with 0.9 probability costs 0.105. The other nine probabilities never appear in the loss’s value — but they do appear in its gradient, which is the subject of the next chapter. The loss wants the model to concentrate probability mass on the right answer, and it pays for that concentration in log scale, where being a little wrong is cheap and being confidently wrong is catastrophic.
The numbers in the table above come from −ln p directly; here is the arithmetic for the two headline comparisons, plus a numerical check that the gradient formula is the real derivative of the loss (not just a slogan).
This is also where the practical warning lives: ln 0 is negative infinity, so a model that outputs exactly 0 for a positive example would receive an infinite loss. The source clips predictions to [1e-15, 1 − 1e-15]; production frameworks go further and fuse the sigmoid or softmax into the loss so the computation never materializes an exact 0.
Cross-entropy is not just a different scale — it has a different shape. Push the prediction toward zero and CE climbs without bound while MSE flattens out at 1. The gradient chart shows the same split on a log scale: CE’s pull grows as 1/p, MSE’s shrinks.
The gradient ratio is the same at both levels — the sigmoid slope multiplies both losses equally. The difference that matters is the shape: when the prediction is confidently wrong, CE’s signal to the weights is ≈ 1 while MSE’s has faded to ≈ 0.02. Fifty times weaker exactly where the mistake is worst.
Confidence in the true class improves from 0.9 to 0.99. Why does cross-entropy pay so much less for that jump than it charges for the reverse move from 0.1 down to 0.01?
Networks do not output probabilities — they output raw scores called logits. A softmax turns them into a distribution, cross-entropy scores that distribution, and the pair gives back the cleanest gradient in machine learning: predicted minus actual.
Softmax exponentiates every logit and divides by the sum, so the outputs are positive and add to 1: softmax(z)_i = e^zᵢ / Σⱼ e^zⱼ. It is a competition — raising one logit necessarily lowers everyone else’s probability. Categorical cross-entropy then reads the probability the model gave the true class: −ln p_true. Frameworks fuse the two into one operation (F.cross_entropy takes logits), and the reason is a small piece of calculus worth seeing once: the derivative of −ln softmax(z)ᵢ* with respect to a logit simplifies to
That gradient is the payoff of the marriage. There is no saturation factor hiding in it: when the model is confidently wrong, the target logit receives a gradient of nearly −1 and every wrong logit a gradient of nearly its full probability. When the model is confidently right, the gradient fades to 0 and training settles — precisely the behaviour you want, and precisely what a sigmoid + MSE pairing cannot give you, because there p(1 − p) multiplies the signal away in the moments it is needed most. The gradient also sums to zero, which is the mathematical version of “move probability mass, don’t invent it.”
Numerics deserve a sentence too. e^1000 overflows, so every real implementation subtracts the largest logit before exponentiating — softmax(z) = softmax(z − max z), which changes nothing mathematically and everything numerically — and combines log-softmax with the negative log-likelihood in one stable kernel. The source’s own code clips probabilities to 1e-15 for the same reason: ln 0 is not a number you can backpropagate.
The source’s Step-3 example: five classes, logits [2.0, 1.0, 0.1, −1.0, 3.0], true class 5 (index 4). Run softmax, read the loss, take the gradient — and then soften the targets.
Label smoothing is a regularizer, not a lie: it says “be very confident, but not infinitely confident.” The model still puts the true class far ahead — it just stops spending capacity pushing logits toward infinity, which improves calibration and makes the network less brittle when the test distribution shifts. It also explains why the smoothed loss never reaches zero: the floor is the entropy of the softened target itself (0.50 in the ten-class case).
Five raw scores become a probability distribution, the loss reads off the target class, and the gradient is exactly p − y — one number per logit saying how wrong its probability is. Turn on label smoothing and watch the target soften.
Watch the gradient row: it always sums to zero. The softmax and cross-entropy pair moves probability mass between classes instead of inflating it — and it does so without ever computing the softmax Jacobian, which is the memory hook of this chapter.
PyTorch's F.cross_entropy expects raw logits, not probabilities. Why does that matter beyond convenience?
Plain cross-entropy assumes the classes are balanced and the labels are honest. Neither is usually true. Focal loss re-weights the examples that still hurt, and contrastive loss learns from similarity when there are no labels at all.
Label noise. Hard targets turn a flipped label into a catastrophe. Take a binary model that puts 0.9 on the true class for 95% of examples; those cost −ln 0.9 = 0.105 each. The 5% of labels that were flipped now sit on the wrong side of the model’s confidence: probability 0.1 on the flipped truth, a charge of −ln 0.1 = 2.303 — 21.9× an honest example. The average loss more than doubles (0.105 → 0.215), and those 5% of rows carry 53.5% of the total loss. The gradient skew is just as bad: each flipped row pulls 9× as hard as an honest one (1/p = 10 versus 1.11), so after averaging they still supply 32% of the gradient magnitude while being 5% of the data. The model will gladly memorize the lies. Label smoothing softens this a little (the ceiling on any single example drops), and robust losses such as symmetric cross-entropy exist for the rest; but the cheapest fix is still to audit your labels.
Class imbalance. The source’s object-detection framing is the canonical one: 99% of candidate regions are background, and standard cross-entropy drowns in easy negatives that are already scored correctly. Focal loss multiplies each example’s cross-entropy by (1 − p_t)^γ, a weight that is near zero for easy examples and near one for hard ones; α adds a class weight for the rare class. With γ = 2 an easy example at p_t = 0.9 is scaled by 0.01 while a hard one at p_t = 0.1 keeps 0.81 — an 81× difference in per-example influence. In the lesson’s 990-easy / 10-hard population the easy group’s share of the loss falls from 81.9% to 5.3% under γ = 2, and to 14.4% once the RetinaNet class weight α = 0.25 is added.
Embeddings without labels. Contrastive learning asks a different question: are these two things similar or different? Each image is augmented twice to make a positive pair; every other image in the batch is a negative. InfoNCE treats the similarities as logits and runs a softmax: the positive must win among all candidates, and the loss is the negative log of its share. Temperature τ divides the similarities first, so it controls how sharply the competition is judged — SimCLR uses τ = 0.07 with a batch of 256, meaning 255 negatives per positive. The failure mode is instructive: get contrastive loss wrong and every embedding collapses to the same point. Zero loss, zero information.
Two computations, both from the source’s own setup: the focal arithmetic for the 990/10 population, and what temperature does to one positive among 255 negatives.
This is the honest version of “lower temperature = harder separation”: τ scales the margin before the softmax, so a positive margin becomes a rout and a negative margin becomes a disaster. Self-supervised training alternates between those two regimes batch by batch, which is why temperature is one of the few hyperparameters every contrastive paper reports.
990 easy negatives and 10 hard positives. Standard cross-entropy lets the solved examples dominate the loss; focal loss multiplies each example by (1 − p_t)^γ so the budget moves to the examples that still hurt. α additionally weights the rare positive class — this board uses the binary convention from the paper, positives × α and negatives × (1 − α), so α = 0.5 isolates the focusing effect of γ.
STANDARD CROSS-ENTROPY — share of total loss
easy negatives 81.9% · hard positives 18.1%
FOCAL LOSS — share of total loss
easy negatives 14.4% · hard positives 85.6%
■ easy negatives · ■ hard positives. With γ = 2 and α = 0.25 the group that was 81.9% of the loss becomes 14.4%; with α = 0.5 it drops to 5.3%, isolating the focusing effect. This is exactly the RetinaNet setting: 99% of candidate regions are easy background, and focal loss is what stops the detector from spending its whole gradient budget saying “still background”.
The board is a toy population — 1,000 examples in two groups, all with the same p_t within a group — so the shares are exact and easy to check by hand. Real datasets have a continuum of difficulties; the mechanism is identical.
No labels — just an anchor, one positive, and negatives competing for attention. Rotate the positive and shrink the temperature; the loss is the negative log of the positive’s share of the similarity softmax.
The SimCLR recipe uses τ = 0.07 and a batch of 256 — that is 255 negatives for every positive, all inside the same softmax. Turn the angle up and shrink τ to feel why “hard negatives” are where representation learning does its work.
def contrastive_loss(anchor, positive, negatives, temperature=0.07):
sim_pos = cosine_similarity(anchor, positive) / temperature
sim_negs = [cosine_similarity(anchor, neg) / temperature for neg in negatives]
# the positive competes with every negative inside one softmax
max_sim = max(sim_pos, max(sim_negs))
exp_pos = math.exp(sim_pos - max_sim)
total = exp_pos + sum(math.exp(s - max_sim) for s in sim_negs)
return -math.log(max(1e-15, exp_pos / total))
def focal_loss(p_true, gamma=2.0, alpha=1.0):
# easy examples (p_true near 1) are down-weighted to almost nothing
weight = (1.0 - p_true) ** gamma
return -alpha * weight * math.log(max(1e-15, p_true))Every loss is a surface stretched over all of the model’s weights. Training is local descent on that surface — no map, no destination — so its shape decides whether the journey is a gentle bowl or a mountain range of basins, plateaus and saddles.
For linear regression with MSE the surface is convex: one bowl, one minimum, and gradient descent cannot get stuck. Neural networks give that up the moment a hidden layer appears. The surface becomes non-convex, with many valleys at different depths, long flat plateaus where the gradient nearly vanishes, and saddle points — places where the gradient is exactly zero but the surface bends down in some directions and up in others. Saddles, not local minima, are the obstacle that dominates in high dimensions: at a random critical point of a million-parameter loss, the odds strongly favour a mix of up and down directions over a true minimum.
Gradient descent is a local procedure, so where you start decides which basin you reach. The lab makes this visible on a deliberately small teaching surface, L(a, b) = (a² − 1)² + b² + 0.3ab + 0.1: two minima near (+1.006, −0.151) and (−1.006, +0.151), both at L ≈ 0.0774, separated by a saddle at the origin with L = 1.1. Start at (0.2, 0.9) and descent lands in the right-hand basin; start at (−0.15, 0.85) and the same rule lands in the left. Nothing about the algorithm changed — only the starting point.
Two more features of real landscapes are worth naming. First, the surface you descend is not the true loss: each mini-batch defines its own slightly different surface, and the noise is a feature — it is what kicks the model off plateaus and out of shallow basins that a full-batch gradient might never leave. Second, some losses redraw the landscape every step: contrastive loss depends on which negatives happen to be in the batch, so the surface itself changes as the batch composition changes. And the deepest point of the loss is the definition of success: squared error’s minimum is the conditional mean, absolute error’s minimum is the conditional median, cross-entropy’s minimum is calibrated probabilities, and a KL-to-a-teacher minimum is a student that copies the teacher’s uncertainty. Whatever the loss, the model becomes what its lowest point means.
Two parameters, one surface. Place the ball, then let gradient descent roll. The same loss has two answers here — which one you reach depends on where you start, which is what “non-convex” means in practice.
The surface is a two-parameter teaching model, chosen so the basins are visible — a real network’s loss lives in millions of dimensions with many more flat directions. The lesson it teaches is the same: gradient descent is a local procedure, and the landscape it can see is the one the loss function defines.
The choice follows from what the model outputs: a number, a probability, a distribution, or an embedding. After that, the loss curve is your first diagnostic instrument — a small set of shapes covers most of what goes wrong.
The source’s decision tree, flattened into a table. Start from the last layer’s output — that is what the loss reads — then adjust for data conditions: imbalance, outliers, overconfidence, soft targets.
| task | default loss | adjust when… |
|---|---|---|
| Regression — predict a number | MSE | Switch to MAE or Huber when outliers, glitches or label typos are real. Never use MSE as a classification loss. |
| Binary classification — one yes/no | Binary cross-entropy | Pair it with a sigmoid, and prefer the fused logits version. Class weights or focal loss when the classes are imbalanced. |
| Multi-class — one label out of C | Categorical cross-entropy over softmax | Fuse log-softmax and NLL (F.cross_entropy). Add label smoothing (α ≈ 0.1) when predictions are overconfident. |
| Multi-label — many independent yes/no | One BCE per label | Labels are not mutually exclusive, so there is no softmax competition; each label gets its own sigmoid. |
| Embeddings / no labels | InfoNCE (contrastive) or triplet | Positives are augmentations or known pairs; negatives come from the batch. Batch size and temperature are part of the loss. |
| Soft targets — distillation | KL divergence | The teacher provides a full distribution; KL(t ‖ p) = CE(t, p) − H(t), so with the teacher fixed it is cross-entropy against soft labels. |
In PyTorch, every default lives as a function, and two of them are fused for stability. The lesson’s Use It section done honestly looks like this:
import torch
import torch.nn.functional as F
predictions = torch.tensor([0.9, 0.1, 0.7], requires_grad=True)
targets = torch.tensor([1.0, 0.0, 1.0])
mse_loss = F.mse_loss(predictions, targets) # regression default
l1_loss = F.l1_loss(predictions, targets) # robust to outliers
huber = F.smooth_l1_loss(predictions, targets) # Huber with beta = 1
# binary classification: prefer the fused logits version
logits = torch.randn(4)
labels = torch.tensor([0.0, 1.0, 1.0, 0.0])
bce_loss = F.binary_cross_entropy_with_logits(logits, labels)
# multi-class: logits + integer labels; label smoothing is a keyword
logits = torch.randn(4, 10)
labels = torch.tensor([3, 7, 1, 9])
ce_loss = F.cross_entropy(logits, labels)
ce_smooth = F.cross_entropy(logits, labels, label_smoothing=0.1)Then watch the curve. A loss that will not move is information, and the number it sticks at usually names the problem. A binary model that sits at 0.693 = ln 2 is outputting 0.5 for everything; a ten-class model at 2.303 = ln 10 is outputting a uniform distribution; a loss that falls all the way to zero within an epoch has found a shortcut or a leak. A loss that drops while your metric refuses to move is the chapter-01 warning arriving in production: the model is optimizing exactly what you asked it to, and it is not what you meant.
Six situations, four candidate losses each. Decide before you click — the feedback names the mechanism, not just the answer.
Hints: is the output a number, a probability, a distribution, or an embedding? Are the classes balanced? Are the labels hard or soft? Does the loss need to be robust to outliers?
Answer before you look. The MSE-on-classification question, the label-smoothing question and the focal-weight question are the three that separate “I can name the losses” from “I know which one to reach for, and why.”
0 / 6 answered · 0 correct
01What does the loss function represent in neural network training?
02Why is cross-entropy preferred over MSE for classification tasks?
03What happens if you use MSE loss for binary classification?
04What does label smoothing do and why is it useful?
05In contrastive loss (InfoNCE), what role does the temperature parameter play?
06In focal loss FL = −α(1 − p_t)^γ log(p_t) with γ = 2, how much do an easy example (p_t = 0.9) and a hard example (p_t = 0.1) weigh?
Click a card to swap the lazy description for what it actually means.
Four problems with exact numbers — implement Huber and watch it ignore a typo, add focal loss to an imbalanced loop, mine semi-hard triplets, and verify that KL divergence against a soft teacher is cross-entropy in disguise. Try first; a worked answer is one click away.
Huber: L(r) = 0.5r² if |r| ≤ δ else δ(|r| − 0.5δ), with gradient r inside and δ·sign(r) outside. Take δ = 1 and one corrupted label that is off by r = 50: MSE charges 2500 and sends a gradient of 100, while Huber charges 1 × (50 − 0.5) = 49.5 and sends a gradient of exactly 1 — the same size as an ordinary miss. So MSE spends the batch bending the curve toward five-percent garbage, while Huber treats corruption as a normal-sized error to reduce. Expect Huber to win clearly on clean test data (MSE's test error is inflated because the fit is pulled toward the corrupted targets) with the gap depending on the run; also expect MSE to fit the clean points slightly more tightly, since quadratic-inside-δ behaviour is preserved exactly. The experiment is a controlled demonstration of chapter 02: the loss's shape is the policy for who gets heard, and with a leaky quadratic amplifier the loudest voice is usually a typo.
Focal loss multiplies each example's cross-entropy by (1 − p_t)^γ. Take the lesson's extreme version: 990 easy negatives at p_t = 0.9 and 10 hard positives at p_t = 0.1. Standard BCE totals 990 × 0.1054 + 10 × 2.3026 = 127.3, of which the easy negatives supply 81.9% — the gradients mostly say 'you are already right' about background. Focal (γ = 2) weights those examples by 0.01 and the positives by 0.81, giving 990 × 0.01 × 0.1054 + 10 × 0.81 × 2.3026 = 19.7, with the easy negatives down to 5.3% of the loss. Adding α = 0.25 for the minority pushes the positive group to 85.6% of the total. The expected result: standard BCE plateaus with the minority recall stuck at the level the majority prior allows, while focal reaches noticeably higher minority recall because the few positive gradients are no longer drowned out. Report both recall and the per-group loss share — the share is the mechanism, recall is the outcome. Caveat: push γ too high and the loss ignores everything easy, which can add noise; γ = 2 is the RetinaNet default for a reason.
Triplet loss is L = max(0, d(a, p) − d(a, n) + margin). With d(a, p) = 0.5, d(a, n) = 0.7 and margin 0.2 the loss is exactly 0 — already separated, no gradient. With d(a, n) = 0.6 the loss is 0.1 and the pair is still being pushed. Random triplets spend most steps in the first regime because most negatives are far away; semi-hard mining deliberately picks n with d(a, p) < d(a, n) < d(a, p) + margin so every batch carries live gradients, and it avoids the hardest negatives, which are often same-class points (false negatives) or noise and can collapse the embedding space. Expect semi-hard runs to show a larger fraction of nonzero triplet losses per epoch and a faster drop in the mean loss at equal step count; the comparison statistic to record is the average fraction of active triplets and the mean d(a, p) versus d(a, n) gap across epochs. Caveat: mining makes the batch's loss distribution shift as the model improves — recalculate the mining set every epoch.
KL(t ‖ p) = Σ t_i·ln(t_i/p_i) = Σ t_i·ln t_i − Σ t_i·ln p_i = H(t) − CE(t, p), so with a fixed target, minimizing KL and minimizing cross-entropy are the same optimization; H(t) is a constant term. Numeric check with t = [0.7, 0.3] and p = [0.6, 0.4]: KL = 0.7·ln(0.7/0.6) + 0.3·ln(0.3/0.4) = 0.7(0.1542) + 0.3(−0.2877) = 0.0216 nats; CE = −(0.7·ln 0.6 + 0.3·ln 0.4) = 0.6325; H(t) = −(0.7·ln 0.7 + 0.3·ln 0.3) = 0.6109, and indeed 0.6109 + 0.0216 = 0.6325. The one-hot case is the limit where t has a single 1: KL = −ln p_true = CE. With soft teacher targets the gradient at the logits is still p − t, so every class with nonzero teacher probability receives a gradient — that is the extra information distillation uses: 'class 4 is somewhat similar to class 3' instead of 'everything except class 3 is exactly wrong'. Raising the temperature T both in the teacher and student softmax divides logits by T; the distilled gradients carry a 1/T² factor, which is why implementations multiply the KL term by T² to keep the loss scale comparable.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 03, Lesson 05) and the Math Foundations Notebook reference build. The eight labs (loss-gradient switchboard, residual penalty explorer, confidence-price chart, softmax console with label smoothing, imbalance board, InfoNCE similarity lab, two-basin loss-surface explorer, and loss-decision board), worked examples A–D (one batch under four penalties; the −log price list with its finite-difference check; the source's logits end to end with label smoothing; the focal and temperature arithmetic), the mean-versus-median constant-prediction check, the 5%-label-noise loss-budget calculation, the 255-negative InfoNCE numbers, and the two-basin teaching surface with its saddle are original to this page. Every number shown is computed live by the labs or verified by hand in the prose. The loss surface and the 990/10 imbalance population are simplified teaching models, labelled as such in the labs.