d(a, b) = ‖a − b‖ A norm measures length; a distance is the norm of a difference. Pick L1, L2, cosine or Mahalanobis and you have picked what “similar” means — for your neighbours, your clusters and your search index alike.
A norm is a length that keeps three promises: never negative, scales by |c|, and never lets a detour beat the direct route. Every norm hands you a distance for free — measure the arrow between two points.
d(a, b) = ‖a − b‖Raise absolute coordinates to the p-th power, add, take the p-th root. p = 1 is the Manhattan diamond, p = 2 the Euclidean circle, p → ∞ the Chebyshev square. The set of length-1 vectors is the norm's fingerprint.
(Σ|xᵢ|ᵖ)^(1/p)Cosine keeps direction and drops length; L2 keeps the straight-line gap; L1 is robust to a single large coordinate; Mahalanobis divides by the covariance so units and correlation stop deciding for you. kNN, clustering and search all inherit the choice.
cos = 1.00 vs L2 = 7.07A norm is a way of measuring how long a vector is, and every distance is the norm of a difference: d(a, b) = ‖a − b‖. Choosing L1, L2, cosine or Mahalanobis is choosing what “similar” means — and every nearest neighbour, cluster, loss and regularizer you build inherits that choice.
By the end you will be able to compute L1, L2 and L∞ by hand, read a unit ball on sight, explain why Lasso snaps weights to zero while Ridge only shrinks them, know when cosine beats Euclidean, whiten a distance with the covariance matrix, and see why high dimensions quietly flatten every distance ratio.
Before you can ask how far apart two points are, you need to say how long one vector is. A norm is that answer — and it comes with three non-negotiable rules.
A norm is a function that assigns every vector a non-negative number, its length, written ‖x‖. Familiar Euclidean length is one norm; it is not the only one. What makes something a norm rather than an arbitrary score is that it keeps three promises.
‖x‖ ≥ 0, and ‖x‖ = 0 exactly when x is the zero vector. Plain English: no vector has a negative length, and nothing except “no movement at all” has no length. Check: ‖(0, 0)‖ = 0, while ‖(0.001, 0)‖ = 0.001 > 0.‖c · x‖ = |c| · ‖x‖ for any number c. Double an arrow and its length doubles; flip it and the length is unchanged. Check: ‖(6, 8)‖ = 10 and ‖(−6, −8)‖ = 10, while ‖2 · (3, 4)‖ = ‖(6, 8)‖ = 10 = 2 · 5.‖a + b‖ ≤ ‖a‖ + ‖b‖. Walk a and then b; you cannot arrive closer to the origin than by walking the single vector a + b. Check: a = (1, 0), b = (0, 1) gives ‖a + b‖ = √2 ≈ 1.414 ≤ 2.That is the whole definition. Anything that keeps these three promises behaves like a length, and anything that behaves like a length gives you a distance by subtraction: d(a, b) = ‖a − b‖. The distance between two points is just the length of the arrow from one to the other.
A distance (a metric) must satisfy three matching rules — symmetry, zero only for identical points, and the triangle inequality. Each one follows from the corresponding norm axiom:
Notice that the proof used only the three norm rules — nothing about squares, roots or coordinates. That is why the L1, L2 and L∞ norms you meet next all produce valid distances: each one is a length that keeps the promises.
L1, L2 and L∞ are not three unrelated rules — they are three settings of the same dial, p. Change p and the set of vectors of length 1 changes shape.
The Lp norm raises every absolute coordinate to the power p, adds the results, and takes the p-th root. In plain English: it measures the total size of the coordinates, but the dial p decides how much credit a single large coordinate gets compared with many small ones.
Read the last two lines as three different opinions about the same pair of points. L1 charges by total travel, L2 by the straight-line gap, and L∞ by the single worst disagreement — and each opinion is a legitimate definition of “far”. The unit ball makes the differences visible.
Slide p to morph the set of vectors of length 1, then drag A and B. All four distances are recomputed live: the same two points, four answers.
The ball's shape is the norm's fingerprint: it tells you which directions the norm considers expensive.
The limit. Let M be the largest absolute component. Factor it out of the sum:
The ordering. L∞ ≤ L2 because M² is one of the terms of Σxᵢ², so M² ≤ Σxᵢ² and taking roots gives M ≤ √Σxᵢ². And L2 ≤ L1 because (Σ|xᵢ|)² = Σxᵢ² + 2Σᵢ<ⱼ|xᵢ||xⱼ| — the cross terms are non-negative, so Σxᵢ² ≤ (Σ|xᵢ|)². Take square roots again. Numeric check on the difference (3, −2, 3): 3 ≤ 4.690 ≤ 8 ✓. The three norms agree only when one coordinate does all the work (or nothing moves).
The same dial, a second job. In training, a norm is added to the loss as a penalty on the weights: loss + λ‖w‖. This is where L1 and L2 stop being interchangeable. L1 (“Lasso”) pushes some weights to exactly zero, selecting features. L2 (“Ridge” or weight decay) shrinks every weight smoothly, selecting nothing. The reason is geometric: the L1 ball has corners on the axes, the L2 ball does not.
A separable teaching model: the loss wants each weight at its best value a and curves like ½(w − a)²; add the penalty and minimize one weight at a time.
The absolute value has a corner at zero, and a penalty with a corner can pin a weight to the corner. The square is smooth everywhere, so it can only pull a weight toward zero, never exactly onto it.
For x = (3, 4), which ordering of L1, L2 and L∞ is guaranteed?
L1 and L2 are two models of travel. One is only allowed to move along the axes; the other cuts straight across. On a city grid the difference is exactly the diagonal.
Put two points on a grid: A = (1, 1) and B = (4, 5). A taxi can move only along the streets, so it drives 3 blocks east and 4 blocks north — length 3 + 4 = 7, the L1 distance. A bird ignores the streets and flies the straight line: a 3-4-5 right triangle, so √(3² + 4²) = 5, the L2 distance. The L∞ distance keeps only the worst single leg, max(3, 4) = 4, which is how a gantry robot that can move one axis at a time experiences the trip: the slow axis sets the time.
The three numbers are not rival answers to one question. They are answers to three different questions — total travel, straight-line gap, worst coordinate — and the point of this chapter is that all three are the correct answer once you have decided what you are measuring.
Drag A and B and switch rulers. Every metric agrees the points are two different places — they disagree about how far apart.
All three norms come from the same two legs: L1 adds them, L2 squares them and takes a root, L∞ keeps the larger one. Try the “Tiny offset” preset — small Euclidean gap, large cosine angle.
Why L2 ≤ L1. Square the two legs. The cross term is never negative, so the straight line can never be longer:
Counting routes. Every shortest grid route uses exactly 3 east steps and 4 north steps — 7 steps in all. Choosing a route is choosing which 3 of the 7 positions are the east steps, so the count is the binomial coefficient:
That is the quiet lesson of the staircase: many different paths can share one distance. A distance function does not care which route you took, only where you started and where you ended.
Cosine similarity asks how aligned two vectors are and deliberately forgets how long they are. That one decision makes it the default for text — and makes its rankings disagree with L2.
Cosine similarity is the dot product divided by both lengths: cos(a, b) = a·b / (‖a‖‖b‖). In plain English, it is the cosine of the angle between the arrows. It runs from +1 (same direction) through 0 (perpendicular) to −1 (opposite), and it does not change if you stretch either arrow. Cosine distance is simply 1 − cos, running from 0 to 2.
One query, six items. Switch the ruler and watch the order change: cosine sorts by direction, L2 sorts by straight-line gap.
| Rank | Item | vector | L2 distance | cosine |
|---|---|---|---|---|
| 1 | 1. same topic, short | (2, 0.3) | 1.005 | 0.9988 |
| 2 | 2. same topic, long | (6, 0.9) | 5.049 | 0.9988 |
| 3 | 5. close but angled | (0.9, 0.9) | 0.707 | 0.8321 |
| 4 | 4. off-topic, tiny | (0.05, 0.05) | 0.962 | 0.8321 |
| 5 | 3. keyword overlap | (0.5, 1.2) | 1.118 | 0.5582 |
| 6 | 6. odd angle, close | (0.2, 1) | 1.131 | 0.3846 |
When magnitude is noise and direction is meaning — text, embeddings — rank by cosine. When magnitude is the message, keep it: use L2 or a raw dot product.
A second worked example, with all four rulers. Take a = (1, 0, 3) and b = (4, 2, 0). The difference is (3, 2, −3), so L1 = 8, L2 = √22 ≈ 4.690 and L∞ = 3 — the points look reasonably close. But the dot product is 1·4 + 0·2 + 3·0 = 4, with lengths ‖a‖ = √10 ≈ 3.162 and ‖b‖ = √20 ≈ 4.472, so cos = 4 / 14.142 ≈ 0.283: nearly perpendicular. Close in position, far in direction. Which one is “more similar” depends entirely on what the vectors represent.
For unit vectors, the dot product is the cosine. For general vectors it is the cosine multiplied by both lengths:
This is also the reason a vector database can offer “cosine or dot product” as a choice: if the stored embeddings are already L2-normalized, each has length 1 and the two formulas compute exactly the same number.
For q = (1, 1), u = (6, 6) and v = (1.5, 0), which ruler ranks u as the nearest neighbour?
Euclidean distance treats every feature as equally important and independent. Real features have units, spread and correlations — Mahalanobis distance measures a gap in units of how unusual it is.
Someone who is 188 cm tall and weighs 82 kg is ordinary. Someone who is 152 cm tall and weighs 82 kg is unusual — but both people are the same Euclidean distance from the average, because Euclidean distance cannot see that weight is a much noisier feature than height, or that the two move together. Mahalanobis distance fixes this by measuring the difference in standard deviations instead of centimetres and kilograms.
A correlated cloud of points. Drag the scored point: L2 measures the straight-line gap to the mean, Mahalanobis measures it in standard deviations along the cloud’s own axes.
A point with a big L2 distance can have a small Mahalanobis distance if it follows the cloud’s correlation — and a small L2 distance can be very unusual if it crosses it. The presets show both.
Case 1 — independent features. If features do not correlate, S is diagonal: S = diag(σ₁², …, σₙ²) and S⁻¹ = diag(1/σ₁², …, 1/σₙ²), so the formula collapses to a sum of z-scores:
Case 2 — correlated features. Any covariance matrix factors as S = L·Lᵀ (Cholesky). Then d_M = ‖L⁻¹(x − y)‖₂: multiply the difference by L⁻¹ and take an ordinary Euclidean length. Numeric check with S = [[4, 2], [2, 3]] and d = (2, 1):
So the units cancel, the correlation is undone, and what is left is a gap measured in standard deviations — comparable across features and across datasets.
Heights spread by 7 cm and weights by 10 kg, and the two are uncorrelated. Person P is 14 cm above average; person Q is 30 kg above average. Under Mahalanobis distance with S = diag(49, 100), who is more unusual?
Add enough independent features and the nearest and farthest points start reporting almost the same distance. The ruler is still accurate — the contrast it measures has drained away.
In two dimensions, a near neighbour and a far point are obviously different. In 500 dimensions, every pair of random points sits at almost the same distance from every other pair. This is distance concentration, the sharpest edge of the curse of dimensionality: as dimension grows, the ratio of farthest to nearest distance for unrelated data approaches 1.
The simulation below is small enough to check by hand: 200 points drawn uniformly in [0, 1]ᵈ, every one of the 19,900 pairwise distances computed, then bucketed. At d = 2 the histogram is wide — some pairs are nearly touching, others are a third of the square apart. By d = 500 the whole distribution has collapsed into a narrow spike.
| dimension d | mean distance | min | max | max / min |
|---|---|---|---|---|
| 2 | 0.507 | 0.003 | 1.272 | 375× |
| 10 | 1.269 | 0.381 | 2.228 | 5.8× |
| 100 | 4.063 | 3.221 | 4.949 | 1.54× |
| 500 | 9.111 | 8.141 | 10.068 | 1.24× |
| 2000 | 18.254 | 17.384 | 19.180 | 1.10× |
Take two random points in [0, 1]ᵈ, coordinate by coordinate. For a single coordinate, the squared gap (u − v)² has mean 1/6 and variance 7/180 — both facts you can verify by integrating over the unit square. Square distances add across coordinates, so:
The average distance still grows — like √d, not d — but the spread grows more slowly than the mean, so the ratio narrows. Distances are not wrong in 500 dimensions; they are all crowded together.
You keep adding independent random features to a dataset, and nothing else changes. What happens to the ratio of the farthest pair's distance to the nearest pair's distance?
kNN, clustering, search indexes, losses and regularizers all inherit one decision made before training starts: what counts as close. Two labs below make the inheritance visible.
A nearest-neighbour classifier is nothing but a distance function plus a vote. A clustering algorithm is a distance function plus a rule for grouping. A vector database is a distance function plus an index. Change the distance and you have changed the model — before a single weight is trained. The heatmap shows six points under four rulers; the kNN lab shows the decision boundary moving.
Six points, all pairwise distances. Darker cells are closer. The ring marks each row’s nearest neighbour — switch rulers and watch the rings move.
| A | B | C | D | E | F | nearest | |
|---|---|---|---|---|---|---|---|
| – | 0.01 | 0.13 | 0.13 | 0.00 | 0.29 | E | |
| 0.01 | – | 0.08 | 0.18 | 0.01 | 0.22 | A | |
| 0.13 | 0.08 | – | 0.47 | 0.13 | 0.04 | F | |
| 0.13 | 0.18 | 0.47 | – | 0.13 | 0.73 | A | |
| 0.00 | 0.01 | 0.13 | 0.13 | – | 0.29 | A | |
| 0.29 | 0.22 | 0.04 | 0.73 | 0.29 | – | C |
Cosine picks a different nearest neighbour than L1 in 6 of 6 rows: A and E point in exactly the same direction (cosine distance 0), so they are each other's nearest.
Click a row label to focus that point. In high dimensions these differences matter even more, because the distances themselves lose contrast — the next chapter.
Drag the query and change k or the ruler. The shaded regions are where each class would win the vote; the boundary itself moves when the metric does.
kNN inherits whatever “close” means. The same points, a different ruler, and a different decision boundary — before any model is trained.
Losses are distances too: they score the gap between a prediction and a target. Regularizers are norms on the weights. The whole training objective is therefore a statement about which errors you care about and which weights you are willing to pay for.
| Loss | Distance it uses | Behavior |
|---|---|---|
| Mean squared error (MSE) | L2 squared | Large errors punished hard; sensitive to outliers |
| Mean absolute error (MAE) | L1 | Every error counts linearly; robust to outliers |
| Huber loss | L2 near zero, L1 far away | Smooth gradient at zero, robust tails |
| Cross-entropy | KL divergence | Distribution mismatch; the classification default |
| Task | Distance | Why |
|---|---|---|
| Text and embeddings | Cosine (or dot product) | Length is noise, direction is meaning; embeddings are trained that way |
| Pixel-level image comparison | L2 | Comparable scales, spatial meaning |
| Sparse high-dimensional features | L1 | Robust; one huge difference does not dominate |
| Tags, categories, masks | Jaccard | The data are sets, not vectors |
| Strings and DNA | Edit distance | The operations match how text changes |
| Outlier detection | Mahalanobis | Accounts for feature scale and correlation |
| Comparing distributions | KL, or Wasserstein if they may not overlap | Information lost, or earth actually moved |
| Manufacturing tolerance, chess king moves | L∞ | Only the worst coordinate matters |
Answer before you look. The Lasso question and the Mahalanobis question are the two that decide whether the ideas have actually landed.
0 / 5 answered · 0 correct
01What does cosine similarity measure between two vectors?
02Why is L1 distance called “Manhattan distance”?
03Why does L1 regularization (Lasso) produce sparse weights while L2 regularization (Ridge) does not?
04What advantage does Wasserstein distance have over KL divergence when comparing probability distributions?
05When would you use Mahalanobis distance instead of Euclidean distance?
Click a card to swap the lazy description for what it actually means.
Four problems, all worked. Try first; the answer is one click away.
Difference = (4−1, 0−2, 6−3) = (3, −2, 3). L1 = 3 + 2 + 3 = 8. L2 = √(9 + 4 + 9) = √22 ≈ 4.690. L∞ = max(3, 2, 3) = 3. So 3 ≤ 4.690 ≤ 8 ✓. Why it must hold: let M = max|xᵢ|. Then M² is one term of Σxᵢ², so M² ≤ Σxᵢ² and M ≤ √Σxᵢ² — that is L∞ ≤ L2. For L2 ≤ L1, expand (Σ|xᵢ|)² = Σxᵢ² + 2Σᵢ<ⱼ|xᵢ||xⱼ|. The cross terms are non-negative, so Σxᵢ² ≤ (Σ|xᵢ|)²; take square roots. Equality holds only when all non-zero components sit in one coordinate (or the vector is zero).
Same direction, different length: a = (1, 1) and b = (100, 100). cos = 1 > 0.9, but L2 = √(99² + 99²) = 99√2 ≈ 140.0 > 10. The arrows point the same way; only their lengths differ, and cosine deliberately ignores length. Different directions, both tiny: a = (0.1, 0) and b = (0, 0.1). cos = 0 < 0.3, while L2 = √(0.01 + 0.01) ≈ 0.141 < 0.5. They are perpendicular — maximum directional disagreement — but they both sit near the origin, so the straight-line gap is small. One more check on the same idea: (1, 0) and (2, 0) have cosine 1 but L2 = 1.
Query q = (0, 0). Candidates A = (2, 0.1), B = (0.42, 0.42), C = (0.05, 0.63), D = (0.75, 0.05). Suppose the two features have spreads σx = 4 and σy = 1 with no correlation, so S = diag(16, 1) and dM = √(x²/16 + y²). Scores: A — L1 2.100, L2 2.003, cos 0.9988, dM 0.510. B — L1 0.840, L2 0.594, cos 0.7071, dM 0.433. C — L1 0.680, L2 0.632, cos 0.0791, dM 0.630. D — L1 0.800, L2 0.752, cos 0.9978, dM 0.194. Each ruler picks a different winner: L1 → C (0.680), L2 → B (0.594), cosine → A (0.9988), Mahalanobis → D (0.194). Nothing about the data changed; only the definition of “close” did.
In one dimension W₁ = Σ |CDF_P(i) − CDF_Q(i)| over the bin edges. For P: CDF = (0.5, 1, 1, 1); for Q: CDF = (0, 0, 0.5, 1). Gaps: |0.5−0| = 0.5, |1−0| = 1, |1−0.5| = 0.5, |1−1| = 0 → W₁ = 2. For the uniform P: CDF = (0.25, 0.5, 0.75, 1); same Q. Gaps: 0.25, 0.5, 0.25, 0 → W₁ = 1. The first is larger: half of P's mass must travel from bins 1–2 to bins 3–4, crossing 2 bin widths. The uniform distribution already has half its mass on the right, so less earth moves. KL divergence would instead report ∞ in both directions, because Q puts zero probability where P is non-zero — a true metric keeps giving a useful number.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 01, Lesson 14) and the Math Foundations Notebook reference build. The unit-ball explorer, distance playground, cosine/L2 ranking board, Mahalanobis cloud, distance-matrix heatmap, kNN metric lab, regularization figure and simulation histograms are original to this page. Every numeric check and worked answer was recomputed. All labs run in your browser.