Any matrix is just
rotate, stretch, rotate.

The singular value decomposition of an m × n matrix A is A = U Σ Vᵀ, where U is m × m orthogonal, Σ is m × n with non-negative numbers on the diagonal, and V is n × n orthogonal. “Orthogonal” means the columns are perpendicular unit vectors, so the matrix is a pure rotation (possibly with a flip), and its transpose undoes it.

Plain English: read the product right to left. Vᵀ rotates the input space so the directions the matrix cares about line up with the coordinate axes. Σ then stretches or shrinks each axis by its singular value. U rotates the result into the output space. A sphere of unit inputs becomes an ellipsoid; the singular values are the ellipsoid’s axis lengths.

Vᵀ — the input rotation. Its rows are the right singular vectors v₁ … vₙ, an orthonormal basis of the input space. They are the directions in which the matrix acts independently: feed in vᵢ and the output is perpendicular to the output of every other vⱼ.

Σ — the stretch. Diagonal entries σ₁ ≥ σ₂ ≥ … ≥ 0. For a non-square matrix there are zeros beyond the min(m, n) positions, which is why the economy version throws them away.

U — the output rotation. Its columns are the left singular vectors u₁ … uₘ, an orthonormal basis of the output space. The pair is tied together by one equation: A vᵢ = σᵢ uᵢ.

The aha: the three factors are not three matrices you have to memorise — they are a coordinate system for the input (V), a list of how much each coordinate matters (Σ), and a coordinate system for the output (U). Change the basis, and every matrix becomes a diagonal scaling. That is the whole trick.

A singular value σᵢ is the length of the i-th semi-axis of the ellipse that the unit sphere becomes. It is always a non-negative real number, and the list is always sorted from largest to smallest. If σᵢ is zero, that direction was crushed completely: the matrix maps it to the zero vector, and it is not part of the matrix’s rank.

Plain English: the singular values tell you how much room the matrix actually uses. Count the non-zero ones and you have the rank. Square them and add, and you have the matrix’s energy — the quantity that low-rank approximation tries to preserve.

Eigendecomposition fails in two situations: the matrix is not square, or it is square but lacks a full set of independent eigenvectors. Multiply A by Aᵀ and both problems vanish. AᵀA is always square, always symmetric, and symmetric matrices always have a full orthonormal set of eigenvectors with real eigenvalues. Even better, AᵀA is positive semi-definite, so those eigenvalues are ≥ 0 and can safely be square-rooted.

Plain English: the eigenvectors of AᵀA are the directions in which AᵀA only stretches, and its eigenvalues are the squares of the stretches A itself performs. Take the square roots and you have the singular values; the eigenvectors are the right singular vectors. Do the same with AAᵀ and you get the left ones.

The aha: SVD works on any shape because it is eigendecomposition applied to two symmetric matrices you can always build — AᵀA and AAᵀ. The reason to compute it another way is numerical: AᵀA is mathematically convenient and numerically dangerous, which is a pattern worth recognising far beyond this lesson.

Column-by-column multiplication turns A = U Σ Vᵀ into a sum of outer products:

Plain English: the first term is the single strongest pattern in the matrix, the second is the next strongest, and so on. If the singular values fall off quickly, a handful of terms reproduces the matrix to within a rounding error — and you never store the rest.

The aha: whether a matrix is compressible is not a property of its size but of its spectrum. Fast decay — a few large singular values followed by a cliff — means a small rank captures almost everything. Slow decay means the matrix genuinely uses all its dimensions, and truncating it destroys real information. The Eckart–Young theorem guarantees the truncated SVD makes the best of whichever situation you are in.

A grayscale image is a matrix of pixel intensities. Store it with a rank-k SVD and the bill is k(m + n + 1) numbers instead of m · n: the k columns of U, the k columns of V and the k singular values.

PCA is SVD. This is not an analogy. Take a data matrix X with n samples as rows and d features as columns, and center each column by subtracting its mean. The covariance matrix is C = XᵀX/(n − 1). Substitute X = U Σ Vᵀ and the whole thing collapses:

Recommendation systems tell the same story with different words. Write the user × movie ratings as a matrix and its SVD as U = user profiles in latent-factor space, Σ = how important each factor is, and Vᵀ = movie profiles in the same space. A predicted rating is the dot product of a user profile with a movie profile, weighted by Σ. Nobody has completely independent taste, so a handful of latent factors — action versus drama, old versus new — explain most of the matrix, and the missing entries get filled in by the rank-k approximation.

The aha: image compression and PCA look like different subjects — one is computer vision, the other statistics — but both are the statement “the useful structure lives in the top singular values.” Learn the truncation once and you have learned both.

A tall matrix A (m > n) turns Ax = b into more equations than unknowns, and usually no x satisfies all of them at once. The standard compromise is to minimize the squared error ‖Ax − b‖², which is exactly what fitting a line through noisy points does. The Moore–Penrose pseudoinverse A⁺ turns that compromise into a one-liner:

The aha: the pseudoinverse completes the promise SVD made at the start. Rotations that cannot fail to exist, stretches that may be zero, and a transpose to walk back across a rectangular shape — that is enough to “invert” any matrix as well as anything can be inverted, and the answer comes with an error guarantee attached. It is also the last piece of linear algebra you need before moving on: every other application in the course is a variation on these three moves.