maximize wᵀΣw subject to ‖w‖ = 1 looks like calculus, but its answer is an eigenvector: the direction the data stretches most. Find it, project onto it, and 784 numbers become a handful without losing the pattern.
In a d-dimensional box, pairwise distances divide by roughly the same mean and grow only √d apart: max/min falls from ~5 at d = 2 to ~1.02 at d = 1000. At the same time volume flees to the corners and the outer shell, and keeping sample density constant costs 10× more data per added dimension.
max/min → 1.02 at d = 1000 · outer shell 1 − 0.9ᵈCenter the data, compute C = XᵀX/(n − 1), and eigendecompose. The eigenvectors are the perpendicular directions of the cloud, and each eigenvalue is the variance along its direction. Keep the top k and every sample becomes k coordinates instead of d.
Cw = λw · explained ratio = λₖ / ΣλᵢPCA is deterministic, invertible and fast: use it as preprocessing, compression and a quick look at variance. t-SNE and UMAP preserve neighborhoods only, shuffle global positions between seeds, and must never be fed to a classifier. Kernel PCA and autoencoders handle curved structure when a flat subspace is not enough.
PCA → model · t-SNE / UMAP → eyes onlyA cloud of data points is usually flat-ish: it spreads a lot in a few directions and barely at all in the rest. PCA rotates your axes so the first points along the biggest spread, then keeps the first k axes and throws the rest away — and those axes are exactly the eigenvectors from Lesson 03.
By the end you will be able to explain the curse of dimensionality with numbers, run and read the explained-variance curve, choose k from the elbow or from downstream accuracy, tell which method is for looking and which is for training, and recognize when reducing dimensions is the wrong move.
Four features is already hard to picture; 784 is impossible to picture and mathematically strange. Three everyday intuitions — distance, volume and density — quietly break as dimension grows.
Distances stop meaning anything. Take two random points in a d-dimensional box. As d grows, the distance between them converges to one predictable value: every pair sits at almost the same distance. If the nearest and farthest neighbours are equally far, then “closest” is a coin flip, and nearest-neighbour search, clustering and similarity ranking all degrade together.
Volume hides in the corners and the shell. A d-cube has 2ᵈ corners, and the fraction of a ball’s volume in its thin outer shell, 1 − 0.9ᵈ, climbs to 1: in high dimensions almost all the volume is at the edge, far from the center where your samples cluster. Data points spread outward, and models starve in the middle.
Density collapses exponentially. To keep the same number of samples per unit volume, going from 2-D to 20-D multiplies the data you need by 10¹⁸ — ten per new dimension. You never have enough data; reduction brings the density back to something workable.
Raise d and watch two things happen to a cloud of 90 uniform random points: the distance histogram packs into a spike around its mean, and the volume of the unit ball migrates into the thin outer shell.
The generated points above also drive the distance histogram: the same seed gives the same cloud, so raising d extends the same data instead of rolling new dice.
Take two random points whose d coordinates are each uniform on [0, 1]. Their squared distance is a sum of d independent terms (xᵢ − yᵢ)², each with the same mean and variance. For uniforms, that single term has mean μ = 1/6 and variance σ² = 7/180:
The mean grows faster than the spread, so the histogram of distances becomes a narrower and narrower spike relative to its position — the lab draws exactly that. A similar calculation explains the shell: the volume of a d-ball of radius r is proportional to rᵈ, so the share inside radius 0.9 is 0.9ᵈ and the share in the outer shell is 1 − 0.9ᵈ. Numeric check: 1 − 0.9² = 0.19 (19% of a disk is in the outer shell), while 1 − 0.9¹⁰⁰ = 0.99997 — essentially all of it.
You index 100-dimensional embeddings and find that the closest and farthest of almost any point's neighbours are nearly the same distance away. What is the most accurate conclusion?
Real high-dimensional data is almost never spread evenly through every direction. It lives on a much smaller surface, and most of the extra coordinates are redundancy or noise.
A handwritten “7” is a 28 × 28 grid of pixel values — 784 numbers. But it does not take 784 independent choices to draw a seven. It takes a few: the angle of the stroke, the length of the crossbar, how much the digit leans, how thick the pen is. Everything else about those pixels is a consequence. The information lives on a low-dimensional manifold: a small surface folded inside the big space.
Finding that surface buys three things. Compression: store or transmit fewer numbers. Speed: models train on shorter vectors. Clarity: two or three directions can be plotted, and plots are how humans spot structure. It also acts as a gentle denoiser, because directions that carry almost no variance are often exactly where the measurement noise lives.
The catch is the word “almost”. Reduction always throws something away. The whole craft is choosing a surface that keeps what matters for your task and drops what does not.
Eighteen 6 × 6 patterns built from a small vocabulary of smooth shapes, squeezed to k components. Blue tiles are the components themselves — the shapes the data is made of.
This is why PCA is compression: the components are a basis, and each pattern is stored as k numbers plus the shared basis. Lose the tail of the eigenvalue list and the blurry parts of the picture go first.
Suppose you have 5,000 digit images at 784 pixels each, and 50 principal components capture the digit shapes. What does storing the compressed version actually cost?
Two honest footnotes. The 50 codes are not automatically smaller than the raw pixels in a byte-for-byte sense — a float is a float — so the saving is real only when k is small relative to d and the decoder is shared. And if 50 components capture 95% of the variance, the missing 5% is still in the file: reconstruction is a trade, not a magic trick.
Principal Component Analysis finds the directions along which the cloud actually stretches, ranks them by how much variance they hold, and lets you keep the top few. The eigenvectors of Lesson 03 are those directions.
The recipe is five steps, and every one has a plain-English job:
C = XᵀX / (n − 1) records how each feature varies and how pairs of features move together.Drag the cloud into a shape, then project it onto PC1. The axes are the eigenvectors of the covariance matrix, drawn at one standard deviation; the pink dots are the compressed data.
When the cloud is almost a line, λ₂ ≈ 0 and the projection loses almost nothing. When the cloud is round, λ₁ ≈ λ₂ and losing PC2 throws away half the spread — no rotation can fix that.
Edit the symmetric covariance matrix and a sample point; the console runs the whole PCA derivation: trace, determinant, eigenvalues, eigenvectors, explained variance and the projection arithmetic.
The last line is the whole promise in one number: for this point and this covariance, keeping the single best direction loses exactly the 10% that PC2 accounted for — and no other direction could lose less.
Forming the covariance matrix squares every number, and squaring is also what happens to numerical error: the condition number of XᵀX is the square of X’s. Libraries therefore compute PCA from the singular value decomposition of the centered data directly: X = UΣVᵀ. The columns of V are exactly the principal directions, the squared singular values divided by (n − 1) are the covariance eigenvalues, and the scores are UΣ.
import numpy as np
Xc = X - X.mean(axis=0) # center first, always
U, s, Vt = np.linalg.svd(Xc, full_matrices=False)
components = Vt[:k] # k x d principal directions
Z = Xc @ components.T # n x k scores
explained = s[:k]**2 / (s**2).sum() # ratios, no (n-1) needed
# sklearn does this for you:
# from sklearn.decomposition import PCA
# pca = PCA(n_components=k).fit(X)A 2-D covariance matrix has eigenvalues 6 and 2 along perpendicular directions. Which statement is correct?
PCA hands you a ranked list of directions. Choosing k turns that list into a decision — and there are three honest ways to make it.
Each principal component captures a fraction of the total variance: explained ratio for component k = λₖ / Σλᵢ. Sum the top-k ratios and you know how much of the original spread k coordinates preserve. The rest is exactly what reconstruction error measures: run the data down to k components and back up, and the mean squared difference from the original is the share of eigenvalues you dropped.
Three strategies, in increasing order of honesty about your actual task:
| Component | Eigenvalue | Explained ratio | Cumulative |
|---|---|---|---|
| PC1 | 4.73 | 0.473 | 0.473 |
| PC2 | 2.51 | 0.251 | 0.724 |
| PC3 | 1.12 | 0.112 | 0.836 |
| PC4 | 0.89 | 0.089 | 0.925 |
| PC5–PC10 | 0.75 total | 0.075 total | 1.000 |
The bars are the eigenvalues of a synthetic 10-feature dataset; the dashed line is cumulative explained variance. Slide k and watch one sample’s 10 numbers rebuilt from its first k coordinates.
The MSE counts squared error per coordinate, so it is tiny next to the raw eigenvalues: the total squared error equals (n − 1) × the dropped eigenvalues, spread over n × d entries.
The total variance of the dataset is the trace of the covariance matrix, which equals Σλᵢ. Keeping k components preserves Σᵏλᵢ of it; the reconstruction error — the average squared distance between each original point and its rebuilt version — is the rest:
Because the relationship is exact — not approximate — reconstruction error and explained variance are two readings of the same number. That is also why reconstruction error doubles as an anomaly score: a point the kept subspace cannot rebuild is a point that does not fit the pattern the rest of the data learned.
A classifier's validation accuracy is 0.91 with k = 30 components and 0.911 with k = 300. Which k would you ship, and why?
PCA finds flat subspaces. Circles, spirals and Swiss rolls are not flat — so the toolbox grows three more tools: kernel PCA for curved structure, and t-SNE and UMAP for looking at it.
Two concentric rings of points cannot be separated by any line. Project them onto the best straight direction and both rings collapse onto the same span of numbers. Kernel PCA first maps the data into a higher-dimensional space — where the rings become a flat, separable problem — and runs PCA there, without ever computing the coordinates. The kernel trick is the shortcut: a kernel function k(x, y) returns dot products between mapped points directly. The most common is the RBF (Gaussian) kernel, exp(−γ‖x − y‖²).
t-SNE and UMAP take a different bargain. Instead of preserving variance, they preserve who is near whom: t-SNE turns pairwise distances into neighbor probabilities and searches for a 2-D layout with the same probabilities; UMAP builds a weighted neighbor graph and lays it out. Both produce gorgeous cluster pictures. Both distort global distances — t-SNE severely, UMAP less so — and both are stochastic.
A teaching picture, not a literal rendering: kernel PCA does not draw points on a line, it finds their principal directions in a lifted feature space.
Forty-five points in 6-D arranged in three groups, laid out by a small t-SNE-style optimizer: neighbor affinities in 6-D, attraction and repulsion in 2-D. Change the seed and watch the global arrangement move while the neighbors stay together.
A teaching implementation: exact O(n²) affinities and gradient, no early exaggeration, no Barnes-Hut. Real t-SNE adds those for speed and stability — and still carries the same warning about global distances.
The RBF kernel measures closeness in the lifted space. With x − y the distance between two points:
γ is the bandwidth knob: large γ makes each point care only about its immediate neighbours (risking overfit), small γ makes everything look similar (losing structure). It plays the same role perplexity does in t-SNE.
| Method | Use it for | Preserves | Speed |
|---|---|---|---|
| PCA | Preprocessing, compression, quick plots | Global variance, linear structure | Fast, exact, millions of samples |
| Kernel PCA | Curved structure before a model | Variance in a lifted space | Slow — n × n kernel matrix |
| t-SNE | Publication-quality 2-D cluster plots | Local neighborhoods only | Slow (under ~10k samples) |
| UMAP | 2-D plots at scale | Local plus some global structure | Medium (handles millions) |
You want to feed 2-D coordinates into a fraud classifier to make it faster. A colleague suggests running t-SNE and using its output. What is wrong with the plan?
PCA compresses by rotating and dropping axes. An autoencoder compresses by squeezing data through a narrow layer and learning to rebuild it — a generalization of PCA whose surface can curve.
An autoencoder is two networks in a row. The encoder maps a 784-pixel image down to a small code, say 32 numbers. The decoder maps that code back to 784 pixels. Training minimizes reconstruction error — mean squared difference between input and output — so the only way to do well is to keep whatever information the decoder needs. The narrow layer in the middle is the bottleneck, and its 32 activations are the compressed representation.
Strip out the nonlinearities and the autoencoder is a matrix product: code z = W₁x, reconstruction x̂ = W₂z. Minimizing mean squared error over all data has a known optimum: the column space of W₂ (and the row space of W₁) spans exactly the top-k principal subspace. So PCA is the linear special case — same answer, found by gradient descent instead of an eigendecomposition.
Now add a nonlinearity between encoder and decoder and the optimal surface can bend. A linear method cannot represent a circle with one coordinate; a nonlinear autoencoder can — it learns a curved manifold just as kernel PCA finds one, but as a trained network rather than a fixed kernel. The cost is what PCA never had: a training loop, a learning rate, and no guarantee of a global optimum.
PCA is powerful, unsupervised and blind to your labels. There are real situations where running it is the wrong first move — and one where it throws away exactly the direction you needed.
Run PCA because the data is redundant, not because it is high-dimensional. If a model is already fast enough, if features are individually meaningful, or if a regularizer can handle the redundant directions, reduction adds a preprocessing pipeline without buying anything. The checklist before reaching for PCA or t-SNE:
Suppose the x coordinate is uniform on [−3, 3] in both classes, and the only difference is a small shift in y: class A sits at y = +0.05, class B at y = −0.05.
This is not a pathology of the numbers; it is a property of unsupervised methods. If the labels exist, use them: linear discriminant analysis, partial least squares or a supervised feature selection step can rank directions by how much they separate classes. PCA has no idea the labels exist.
Answer before you look. The PCA question and the t-SNE question are the two that come up most in real debugging sessions.
0 / 5 answered · 0 correct
01What is the “curse of dimensionality”?
02What does PCA find?
03After running PCA on 784-dimensional MNIST data with k = 50 components, you find 95% of the variance is captured. What does this tell you?
04Why should you NOT use t-SNE as preprocessing before training a classifier?
05When would you choose kernel PCA over standard PCA?
Click a card to swap the lazy description for what it actually means.
Four problems, including a hand-computed PCA from scratch. Try first; a worked answer is one click away.
inverse_transform(Z) = Z @ components + mean undoes transform because components has orthonormal rows. The total squared reconstruction error over the dataset equals (n − 1) × (sum of eigenvalues not kept), so the MSE per entry is (n − 1)·Σ_dropped / (n·d). Concretely: with n = 5000, d = 784 and dropped eigenvalues summing to 0.31, total squared error = 4999 × 0.31 = 1549.7, and per-element MSE = 1549.7 / 3,920,000 ≈ 0.000395. Which components you drop matters more than how many: at 10 components the dropped sum is large; by 200 it is a rounding error, and the per-pixel error is dominated by genuine noise the subspace never captured.
Perplexity is the effective number of neighbors each point uses. At 5, each point listens only to its closest handful: the picture fragments into many tiny tight clumps, and large-scale relationships wash out. At 30, local clusters themselves become clear. At 100, neighborhoods are large enough that clusters bleed into one another into a smoother, less separated map — small clusters can dissolve. The tightness changes because perplexity sets the width σᵢ of each point's Gaussian affinity in high-dimensional space; a narrow Gaussian makes almost everything a near-zero neighbor, so the optimizer is pushed to fold close friends tightly together. Standard advice is to try several values and distrust conclusions that only appear at one setting.
The curve climbs steeply for the first 5 components and then flattens: those 5 directions capture the class structure, and the remaining 45 are pure noise with roughly equal, tiny eigenvalues. Numeric sanity check: if the 5 informative eigenvalues are around 3.0 each and each of the 45 noise directions contributes about 1.0, the informative share is 15/(15 + 45) = 25% spread over 5 components, so the elbow sits near 25% cumulative — not 90–95%. That is the point: a flat tail of eigenvalue-1 directions is the signature of pure noise, and a 95% threshold would keep far more components than the data's true dimension. The elbow often identifies the dimension better than a fixed threshold.
C = XᵀX/(n − 1) with n − 1 = 3. Σx² = 2.25 + 0.25 + 0.25 + 2.25 = 5, Σy² = 5, Σxy = 2.25 − 0.25 − 0.25 + 2.25 = 4, so C = [[5/3, 4/3], [4/3, 5/3]]. This is 1/3 × [[5, 4], [4, 5]], whose eigenvalues are (5 ± 4)/3 = 3 and 1/3. Eigenvectors: [1, 1]/√2 for eigenvalue 3, and [1, −1]/√2 for 1/3. Explained ratios: 3/(10/3) = 90% and (1/3)/(10/3) = 10%. Projecting with PC1 = [0.707, 0.707]: the first point gives z = (−1.5 − 1.5)/√2 = −2.121 and rebuilds exactly to (−1.5, −1.5); the second gives z = 0 and rebuilds to (0, 0), leaving residual (−0.5, 0.5) with squared length 0.5. By symmetry point three loses 0.5 and points one and four lose nothing. Total squared error = 0 + 0.5 + 0.5 + 0 = 1.000, which equals (n − 1) × λ₂ = 3 × (1/3) = 1.000 ✓.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 01, Lesson 10) and the Math Foundations Notebook reference build. Interactive figures, the animated projection hero, the hand-computed PCA example, the when-not-to-reduce counterexample and the curse, PCA, scree, compression, t-SNE and eigen labs are original to this page. Every lab runs in your browser.