inertia = Σ ‖xᵢ − μ_c(i)‖² is a clustering algorithm’s report card. K-means lowers it with two moves: every point claims its nearest centroid, then every centroid moves to the mean of its claimants.
Clustering partitions points so that within-group similarity beats between-group similarity — and similarity is a distance metric you pick. Euclidean distance over standardized features, cosine over embeddings: the same rows, a different map. There is no accuracy to print, only internal evidence.
similar = small distanceK-means needs one number, k. Assign every point to its nearest centroid; move every centroid to the mean of its points. Inertia — the total squared distance to the assigned centroid — falls on both moves, so the loop always ends. A different start can end somewhere else.
inertia = Σ ‖xᵢ − μ‖²Inertia always falls as k grows, so compare the shape of the fall: the elbow where extra clusters stop paying. The silhouette scores each point, (b − a)/max(a, b), against its nearest rival cluster. Above zero the grouping is defensible; below zero the point is in the wrong group.
elbow + silhouette, never raw inertiaClustering never tells you the right answer — it tells you the structure implied by one definition of similar (a distance), one number of groups (k, eps, or a cut height), and one dataset; change any of the three and the clusters change with it.
By the end you will be able to run k-means by hand and by stepper, choose k from an elbow and a silhouette, explain why moons and rings break k-means, read a dendrogram, classify points as core, border or noise in DBSCAN, and pick a sensible first method for an unlabeled table.
Every lesson so far had an answer key. Unsupervised learning does not: it groups similar data, compresses it, or flags what refuses to fit — without ever being told what is right.
Labels are expensive. A hospital has millions of patient records and nobody has hand-tagged each one with a condition; an e-commerce site has millions of sessions and no one has written down which ones belong to the same kind of shopper; a security team has network logs and no complete list of past intrusions. Unsupervised learning is what you do when the data is plentiful and the answers are not.
Clustering assigns each point to a group so that points in the same group are more similar to each other than to points in other groups. The loaded word is similar. It is not a fact about the data — it is a distance metric you choose. Euclidean distance over standardized features, cosine similarity over embedding vectors, edit distance over strings: each one redraws the groups. Everything else in this lesson is machinery for optimizing or judging a choice that starts right there.
The catch the source states plainly: without labels you cannot measure “right” or “wrong”. There is no accuracy to print. Instead you get internal evidence — how tight the clusters are, how separated they are, whether the result survives a change of initialization — and you stay honest about what that evidence can and cannot prove.
You have 10,000 customer sessions and no labels. What does a clustering algorithm need before it can group them?
K-Means is the workhorse: give it a number of clusters k and it alternates two moves until nothing changes. No gradients, no labels — just means and distances.
The algorithm starts with k centroids — one guess per cluster. Then it repeats two moves:
The loop stops — provably, in a finite number of steps — when the assignments stop changing. Both moves only ever decrease the same objective, the total spread of the clusters:
Assigning a point to its nearest centroid can only lower its distance to its centroid; moving a centroid to the mean of its members can only lower the total squared distance of those members. So inertia slides downhill on both moves — which also guarantees it cannot cycle forever. What it does not guarantee is the global minimum: different starting centroids can settle in different valleys.
Two things to notice. First, the move nearly halved the inertia before any re-assignment happened — the “tightening” is real arithmetic, not vibes. Second, the six points split into the two obvious triples even though the starting centroids were data points, not cluster means. K-Means does not need a clever start here because the blobs are far apart; make them overlap and the start matters (see the trap below).
Six points, two centroids — the same data as the hand-worked iteration above. Click the plot to move the nearest centroid, then step.
The inertia never needs to say “converged” on its own — it decreases at assign, decreases again at move, and the first time the colors stop changing you are done.
After the assign step, every point has a cluster. What does the update step do?
K-Means answers “what are the best k clusters?” but never “how many should there be?”. Two honest tools for that question: where inertia bends, and how separated the groups are.
Inertia is not a score you can optimize, because it always falls as k grows. With one centroid per point it reaches exactly 0 — a perfect “score” and a useless clustering. The signal is not the height but the shape of the curve: the elbow method plots inertia against k and looks for the point where each extra cluster stops paying for itself. Before the elbow, splitting a merged group removes a lot of spread; after it, the algorithm is just carving up real clusters.
The silhouette score attacks the same question from the other side — not “how tight?” but “how much better is this point’s own cluster than the next best one?” For a single point:
Average s over all points and you have one number for a whole clustering, comparable across values of k. (Points in single-member clusters are conventionally scored 0, since there is no “own” group to compare against.)
Three blobs, k = 1 through 6. Move k: the scatter recolors, and the two curves on the right come from the same nearest-centroid math.
Inertia always falls as k grows — with one centroid per point it reaches 0. That is why the choice is made by the size of the drop (elbow) or by a separation score (silhouette), never by raw inertia.
The console above prints every row of this table for whichever candidate you pick; P1 is the first line. On real data you rarely get such a clean answer, but the direction is always the same: compare the mean s of several candidate clusterings and distrust the negative points first.
A point's silhouette is −0.4. What does that mean?
K-Means is fast, simple and deterministic once started — and wrong the moment its geometry does not match your data. Recognizing the four assumptions tells you when to reach for a different method.
1 · Clusters are convex and roughly equal in size. Nearest-centroid boundaries are straight lines (in higher dimensions, flat planes). That partition carves space into convex cells, so it cannot wrap a crescent, a ring, or any shape with a bend. When one true cluster is twice as large as another, the small one often gets split in two because both centroids find room inside it.
2 · Every point belongs to exactly one cluster. Assignments are hard: a point on the boundary is flung to one side or the other, even when it genuinely sits between two groups. There is no “60% A, 40% B” in k-means.
3 · Distance is meaningful at the scale you measured. Change a column from metres to millimetres and the clusters change. Feature scaling is not optional here — it is part of the model.
4 · No point deserves to be ignored. Because the center is a mean, a single far-away point drags it:
And one assumption that is not about geometry at all: k must be chosen before the algorithm runs. The next two methods drop that requirement in two different ways.
Pick a shape and a method. K-means can only ask “which centroid is nearest?”; DBSCAN asks “who is crowded around whom?”
Neither answer is wrong on its own terms — they are answers to different questions. Match the method to the shape you expect, or just to the shape you can see.
Hierarchical clustering does not commit to k upfront. It builds a tree of every possible merge, and choosing k becomes choosing where to cut the tree.
Agglomerative (bottom-up) clustering starts with every point as its own cluster and repeatedly merges the two closest clusters. The result is a dendrogram: a tree whose branches merge at a height equal to the distance between the two clusters at that moment. Cut the tree with a horizontal line and its intersections are your clusters; slide the line down and clusters split, slide it up and they merge. One run gives you every k at once.
The one design decision is what “closest clusters” means. The source lists four standard answers:
The cost is the trade-off. Comparing every pair at every merge is O(n²) time and memory, so this is a small-data tool: a few thousand points at most. In exchange you get a complete picture of structure — which is why it is the standard method in phylogenetics, document taxonomies, and any setting where the hierarchy itself is the answer.
Both linkages produce the same merge order here, but the heights tell different stories. Single linkage calls the last merge 4 — the distance between the two closest survivors. Ward calls it 24.08, because welding two compact groups into one bag costs an enormous amount of variance. That is how a dendrogram encodes not justwhat merged but how much it hurt.
Six points on a line. Merge the closest pair again and again — then slide the cut to choose how many clusters survive.
Changing the linkage changes the heights, not the points. Ward merges compact groups first and makes the big gaps look huge; single linkage chains through the nearest neighbour and makes the tree look deceptively shallow.
DBSCAN asks a different question: not “which centroid is nearest?” but “is there a dense enough path from here to there?” The answer can be any shape, and some points get no cluster at all.
DBSCAN has exactly two dials. eps is the radius of a point’s neighborhood. minPts is how many neighbors (counting the point itself, as scikit-learn does) make that neighborhood crowded. Those two numbers sort every point into one of three kinds:
Then clusters grow by connectivity: start from any unvisited core, absorb every core within eps, absorb their borders, and repeat until the crowd runs out. Because the cluster is a connected chain of dense neighborhoods, it can curve, spiral or ring — shapes that no straight boundary can capture. And because you never told it how many clusters to find, it reports however many the density supports, which may be zero.
The weakness is the flip side of the two dials: a single eps has to serve every cluster, so DBSCAN struggles when one cluster is dense and another is sparse — the same eps is too small for one and too large for the other. Very high dimensions hurt too, for the same reason distance-based methods do: distances concentrate and “near” stops meaning much.
Ten points with one straggler and one far outlier. Turn the two dials and watch core, border and noise change labels.
DBSCAN never asks for k. It finds as many clusters as the density supports — and it is allowed to call a point noise.
Now leave the points alone and turn one dial:
Both moves make the same point: DBSCAN’s output is a function of the dataset and the two dials, and the dials have no universal right answer. Plot eps against the number of clusters and noise points, pick the plateau, and let the domain — not just the chart — break the tie.
In this lesson's DBSCAN lab (eps = 1.5, minPts = 3, neighborhoods count the point itself), the point P5 at (3.2, 1) has exactly one other point within eps: the core point P2 at (2, 1). What is P5?
The same “no labels” setting supports soft clustering, anomaly detection, association rules and dimensionality reduction. Knowing which family answers which question is most of the skill.
GMM — soft clustering. K-means gives each point one hard label. A Gaussian Mixture Model assumes the data came from k overlapping bell curves, each with its own center, spread and weight, and fits them with the EM algorithm: the E-step computes each point’s probability of belonging to each component, the M-step updates the components from those probabilities. The result is a probability vector per point and — unlike k-means — elliptical clusters that may overlap.
Anomaly detection for free. Each method has a built-in “this does not fit” signal: the distance from a point to its nearest k-means centroid, a DBSCAN noise label, or a low probability under every GMM component. Combine the signals and keep the agreements — a point that all three methods single out is a much safer alert than one that only one of them dislikes.
Association rules. Instead of grouping rows, market basket analysis looks for items that co-occur. For itemset A → B in n baskets: support = P(A and B), confidence = P(B | A), and lift = confidence / P(B) — how much more often B appears with A than it would alone. Lift above 1 means an association worth investigating; below 1 means the two items avoid each other. Same unsupervised setting, rows and columns swapped.
Dimensionality reduction. With hundreds of noisy features, distances concentrate and every cluster looks equally far away (the curse of dimensionality). PCA and t-SNE find a smaller set of directions that preserve most of the structure, and clustering then runs on two or ten dimensions instead of a thousand. That is a whole lesson of its own — Phase 1, Lesson 10 — and the natural next step after this one.
| Method | Best for | Avoid when |
|---|---|---|
| K-Means | Large datasets, compact spherical clusters, a k you can defend | Irregular shapes, outliers, features on different scales |
| DBSCAN | Unknown k, arbitrary shapes, outlier detection | Varying densities, very high dimensions |
| Hierarchical | Small datasets, a dendrogram, exploring many k at once | Large datasets (O(n²) time and memory) |
| GMM | Overlapping clusters, soft assignments, elliptical shapes | Very large datasets, too many dimensions |
Answer before you look. The moons question and the DBSCAN classification question are exactly the ones that separate a memorized definition from a working model.
0 / 6 answered · 0 correct
01What distinguishes unsupervised learning from supervised learning?
02What does K-Means require you to specify before training?
03K-Means fails on two interlocking half-moon shapes but DBSCAN succeeds. Why?
04What is the silhouette score measuring?
05How does a Gaussian Mixture Model differ from K-Means in its cluster assignments?
06In this lesson's DBSCAN lab (eps = 1.5, minPts = 3, neighborhoods count the point itself), the point P5 at (3.2, 1) has exactly one other point within eps: the core point P2 at (2, 1). What is P5?
Click a card to swap the lazy description for what it actually means.
Four short problems with exact numbers. Try first; a worked answer is one click away.
Squared distances to (0,0): (1,0) → 1; (0,1) → 1; (10,0) → 100; (10,1) → 101; (11,0) → 121. Total = 324. K-Means++ picks the next centroid with probabilities 1/324 ≈ 0.003, 1/324 ≈ 0.003, 100/324 ≈ 0.309, 101/324 ≈ 0.312, 121/324 ≈ 0.373 — a 99.4% chance of landing in the right-hand blob, so the two centroids nearly always start in different clusters. Uniform random initialization draws two distinct points from all six: the chance both land in the right blob is (3/6)·(2/5) = 0.2, and the chance both land in the left blob is also 0.2. In those 40% of runs the first assign step puts nothing on one side, an entire iteration is wasted, and the final inertia can be worse. Same algorithm, better start.
Ward's merge cost is Δ = (n₁n₂ / (n₁ + n₂)) · ‖c₁ − c₂‖², where c is each cluster's centroid. Merge (1, 2): cost = (1·1/2)(1 − 2)² = 0.5, centroid 1.5. Merge ({1,2}, {4}): cost = (2·1/3)(4 − 1.5)² = 4.1667, centroid 7/3 ≈ 2.3333. Merge ({1,2,4}, {8}): cost = (3·1/4)(8 − 7/3)² = 0.75 × 32.1111 = 24.0833. Heights: 0.5, 4.17, 24.08. Cut at 5: the first two merges are below 5, the third is above, so the clusters are {1, 2, 4} and {8}. Cut at 30: all three merges are below, one cluster {1, 2, 4, 8}. For comparison, K-Means with k = 2 and farthest-first initialization also returns {1, 2, 4} and {8}, with the split falling inside the 4-wide gap — exactly where the tallest dendrogram bar says it should.
Gaussian density: p(x) = (1/√(2π))·e^(−(x−μ)²/2), so the shared constant is 0.3989. At x = 3: component 1 gives 0.3989·e^(−4.5) = 0.00443; component 2 gives 0.3989·e^(−2) = 0.05399; the weighted responsibilities are 0.5·0.00443 / (0.5·0.00443 + 0.5·0.05399) = 0.0758 and 0.9242. The point clearly belongs to component 2, so it is not an anomaly. At x = −4: component 1 gives 0.3989·e^(−8) = 1.338×10⁻⁴ and component 2 gives 0.3989·e^(−40.5) ≈ 1.0×10⁻¹⁸; the best (largest) weighted density is 6.69×10⁻⁵, far below any sensible threshold such as 10⁻³, so GMM flags it. DBSCAN flags a point as noise when its eps-neighborhood has fewer than minPts members — for x = −4 with eps = 1.5 and minPts = 3, it has no neighbours at all, so both methods agree. Where they disagree is the interesting case: a point between two Gaussians can have a decent likelihood (not an anomaly statistically) but still be sparse in DBSCAN's local sense, or vice versa for a tight far-away cluster of two points. Agreement-based pipelines keep the confident flags and hand the disagreements to a human.
Assign by squared distance. (1,1): to (2,1) is 1, to (9,9) is 128 → cluster 1. (2,1): 0 vs 113 → cluster 1. (1,2): 2 vs 113 → cluster 1. (8,8): 85 vs 2 → cluster 2. (9,8): 98 vs 1 → cluster 2. (8,9): 100 vs 1 → cluster 2. Assignment clusters keep the two triples, and inertia = 1 + 0 + 2 + 2 + 1 + 1 = 7. Move: cluster 1 mean is ((1+2+1)/3, (1+1+2)/3) = (4/3, 4/3); cluster 2 mean is ((8+9+8)/3, (8+8+9)/3) = (25/3, 25/3) ≈ (8.333, 8.333). Re-assign: (1,1) is 2/9 from the new first centroid; (2,1) and (1,2) are 5/9; (8,8) is 2/9 from the second; (9,8) and (8,9) are 5/9. Same triples, so inertia becomes 2/9 + 5/9 + 5/9 + 2/9 + 5/9 + 5/9 = 24/9 = 8/3 ≈ 2.667 and the algorithm settles. The init (2,1) is not a data point's centroid convention — any point works — and the iteration still lands on the same partition because the two blobs are far apart relative to their spread.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 02, Lesson 07) and the Math Foundations Notebook reference build. The k-means stepper, choose-k chart, shape comparison, dendrogram builder, DBSCAN explorer, silhouette console, both worked iterations, the DBSCAN classification walkthrough and every numeric check are original to this page. Every lab runs in your browser.