A matrix is a machine that reshapes space. Do A, then B, and the whole journey is one combined matrix — written B @ A. The same two machines in the other order land somewhere else.
Multiply M by [1, 0] and you get its first column; multiply by [0, 1] and you get the second. Those two landing spots describe the whole transformation — every other point follows because the grid stretches with them.
M e₁ = (a, c) M e₂ = (b, d)Composition is just multiplication of the machines. The matrix closest to the point acts first, exactly like f(g(x)) is read from the inside out. Swap the order and the endpoint usually changes.
B @ A @ p = B @ (A @ p)Out of every arrow in the plane, a few come back along the same line after the matrix hits them. The stretch factor is the eigenvalue λ. Repeated multiplication then grows them like λᵗ — the whole RNN stability story.
A v = λ vA matrix is not a grid of numbers — it is a verb. Its columns say where the basis arrows land, its determinant says what happens to area, and its eigenvalues say which directions survive being multiplied by it again and again.
By the end you will read any 2×2 matrix as a transformation, compose two of them and predict the order effect, find eigenvalues from the characteristic equation, rebuild a matrix as A = V D V⁻¹, and explain in one sentence why a signal in an RNN explodes or vanishes.
A matrix is a machine that reshapes space. To know everything it does, you only need to know where it sends two arrows: e₁ = [1, 0] and e₂ = [0, 1].
Multiply any matrix by [1, 0] and you get its first column. Multiply by [0, 1] and you get its second column. So a matrix is a compact way of saying “the x-arrow goes here, the y-arrow goes there, and the whole grid stretches to match.”
That single fact lets you read a matrix by eye. Take M = [[2, 0], [0, 0.5]]. Its first column is (2, 0), so the x-arrow doubles; its second column is (0, 0.5), so the y-arrow halves. Numeric check: M @ [3, 1] = (6, 0.5), which is exactly 3 · [2, 0] + 1 · [0, 0.5]. Every point rides along as a mix of the two columns — that property is called linearity.
A rotation by angle θ keeps lengths and angles fixed. The definition of sine and cosine is all we need to write its matrix.
Every linear transformation in 2D is some combination of four moves, and each has a two-by-two matrix you can read at a glance.
Rotation slides every point along a circle. Scaling stretches or compresses along each axis independently. Shearing shifts one coordinate in proportion to the other, sliding rows sideways. Reflection flips space across an axis. Each one is a small matrix, and each one tells you its own area story through the determinant.
| Move | Matrix | What happens to space | det |
|---|---|---|---|
| Rotation by θ | [[cos θ, −sin θ], [sin θ, cos θ]] | Every point slides along a circle; lengths and angles preserved. | 1 |
| Scaling | [[sx, 0], [0, sy]] | Stretch or squash along each axis independently. | sx·sy |
| Shear in x | [[1, k], [0, 1]] | x shifts by k·y; rectangles become parallelograms, area unchanged. | 1 |
| Reflection (y-axis) | [[−1, 0], [0, 1]] | Mirror image; area preserved, orientation flipped. | −1 |
Numbers make both claims checkable. Rotate the unit square’s corners by 45°, then shear them with k = 1.
The determinant sees the parallelogram’s area, not its slantedness. That is why a shear and a rotation can share det = 1 while looking nothing alike.
Applying A and then B is the same as applying one single matrix: B @ A. The matrix nearest the vector acts first, and swapping the order usually lands somewhere else.
Rotate 90° then scale by (2, 0.5) and the point (1, 0) lands at (0, 0.5). Scale first and the same point lands at (0, 2). Same two machines, four times the distance — because matrix multiplication is not commutative. The combined matrices are different too: S @ R = [[0, −2], [0.5, 0]] while R @ S = [[0, −0.5], [2, 0]].
Pick two machines and an order. The animation applies the first, then the second; the readout compares the two orders on the same point.
Rotate 90° then scale (2, 0.5) sends [1, 0] to [0, 0.5]. The other order sends it to [0, 2]. Same two machines; four times the distance.
Draw the unit square, apply the matrix, and measure the parallelogram you get. Its signed area is the determinant — the fastest way to tell whether a transformation is invertible.
For M = [[a, b], [c, d]] the determinant is ad − bc. It is the factor by which the transformation scales area in 2D — and volume in 3D. Numeric check: [[2, 1], [0.5, 1.5]] has det = 2·1.5 − 1·0.5 = 2.5, so the unit square lands as a parallelogram with area 2.5.
det = 1 means area is preserved (rotation, shear). det = 0 means the plane was crushed onto a line or point: information is destroyed and no inverse exists. A negative determinant means area is preserved in size but flipped — the same number, opposite orientation. That sign is the only thing separating a rotation (det = 1) from a reflection (det = −1).
Edit the matrix and watch the unit square land as a parallelogram. The bars on the right compare one unit of area with what the image now holds.
Rotation and shear have det = 1: the shape changes, the area does not. Scaling by (2, 0.5) also has det = 1 — a squash here is paid for by a stretch there. In 3D this same number measures volume.
The matrix [[a, b], [c, d]] sends e₁ = [1, 0] to the first column u = (a, c) and e₂ = [0, 1] to the second column v = (b, d). The unit square lands as the parallelogram spanned by u and v. Draw it inside the rectangle of width a + b and height c + d, then cut away everything that is not the parallelogram: two triangles of area ½ac, two of area ½bd, and two small rectangles of area bc each.
That last line matters for deep networks: applying 10 layers multiplies 10 area factors together. Any single det near zero can squash the whole product flat.
Hit a plane with a matrix and almost every arrow changes direction. A few special arrows come back along the same line, scaled by a factor λ. Those are the eigenvectors, and λ is the eigenvalue.
The whole definition is one equation: A v = λ v, with v not the zero vector. It says: apply the matrix, and the result points along the same line as v — only its length changed. Numeric check with A = [[2, 1], [1, 2]]:
Edit the matrix. Purple dashed lines are the directions the matrix only stretches — the labels are their stretch factors. Try Rotate 90° to see the complex case.
Two shortcuts catch slips: the eigenvalues always add up to the trace (3 + 1 = 4) and multiply to the determinant (3 · 1 = 3). A negative discriminant means no real direction survives.
A negative discriminant under the square root gives complex eigenvalues: the algebra telling you the transformation contains a rotation, and no real arrow survives. Try the Rotate 90° preset in the lab — the pair becomes ±i and the plan view shows a pure spin.
If a matrix has two independent eigenvectors, it can be written as three simpler machines: V, D, V⁻¹. D is diagonal — the hard part is only a pair of coordinate changes.
A = V D V⁻¹. Here V holds the eigenvectors as its columns, D is the diagonal matrix of eigenvalues, and V⁻¹ is the inverse that undoes V. Read right to left: V⁻¹ translates the point into eigenvector coordinates, D stretches each coordinate by its eigenvalue, and V translates the result back to ordinary coordinates.
For A = [[2, 1], [1, 2]], the eigenvectors are [1, 1] and [1, −1], so V = [[1, 1], [1, −1]] / √2 and D = [[3, 0], [0, 1]]. Numeric check: V D V⁻¹ = [[2, 1], [1, 2]] exactly, which is the claim of the decomposition.
A symmetric matrix has perpendicular eigenvectors, so its V is a rotation or reflection and V⁻¹ = Vᵀ — the friendliest case, and the one PCA lives in.
def eigenvalues_2x2(m):
(a, b), (c, d) = m
trace, det = a + d, a * d - b * c
disc = trace ** 2 - 4 * det
if disc < 0: # complex pair: this matrix rotates
return (complex(trace / 2, (-disc) ** 0.5 / 2),
complex(trace / 2, -(-disc) ** 0.5 / 2))
r = disc ** 0.5
return ((trace + r) / 2, (trace - r) / 2)
eigenvalues_2x2([[2, 1], [1, 2]]) # (3.0, 1.0)
import numpy as np
vals, vecs = np.linalg.eig(np.array([[2, 1], [1, 2]], dtype=float))
# vals = [3, 1]
# vecs columns ≈ [0.707, 0.707] and [-0.707, 0.707]
V, D = vecs, np.diag(vals)
np.allclose(V @ D @ np.linalg.inv(V), [[2, 1], [1, 2]]) # True: A = V D V⁻¹A recurrent network updates its hidden state as h ← W h. Every step multiplies by W, so each eigen-direction is raised to a power — and that power decides whether the signal explodes or disappears.
Write the update as h_t = W @ h_(t−1). After T steps, h_T = W^T @ h_0. Decompose h₀ into W’s eigenvectors: each component is multiplied by its own eigenvalue once per step, so after T steps it has been multiplied by λᵀ.
That makes the magnitude of λ the whole story. Along λ = 1.05 the signal grows by 1.05¹⁰⁰ ≈ 131× in a hundred steps — explosion. Along λ = 0.95 it shrinks to 0.95¹⁰⁰ ≈ 0.006 of itself — vanishing. The spectral radius, the largest |λ|, is the single number that predicts which regime the network lands in.
An RNN multiplies its state by the same matrix at every step. Press play to walk the trajectory h ← A h and watch each eigen-direction grow or shrink by its own factor.
A diagonal matrix stretches each axis by its eigenvalue: 1.05²⁰ ≈ 2.65 while 0.95²⁰ ≈ 0.36. After 100 steps those become ≈ 131 and ≈ 0.006 — the exploding/vanishing signal problem in one picture.
Take a diagonal weight matrix so the eigen-directions are just the axes: W = [[1.05, 0], [0, 0.95]] and h₀ = [1, 1]. Then h_T = [1.05ᵀ, 0.95ᵀ], entry by entry.
In a real RNN, W is not diagonal, but the same statement holds in its eigenvector coordinates: the h₀-component along each eigenvector is multiplied by λᵀ. Vanishing and exploding gradients are what happens when training pushes through those same powers in reverse.
Two dozen lines of Python reconstruct every transformation in this lesson — and the composition rule is just nested loops around a sum.
Notice that mat_mul never looks at the numbers as geometry. It only obeys the shape rule from Lesson 02: row i of the left matrix meets column j of the right one. The geometry emerges from the multiplication — which is why the same six lines implement rotations, shears, and whole neural network layers.
import math
def rotation_2d(theta):
c, s = math.cos(theta), math.sin(theta)
return [[c, -s], [s, c]]
def scaling_2d(sx, sy):
return [[sx, 0], [0, sy]]
def shearing_2d(kx, ky):
return [[1, kx], [ky, 1]]
def reflection_y():
return [[-1, 0], [0, 1]]
def mat_vec_mul(matrix, vector):
return [sum(row[j] * vector[j] for j in range(len(vector)))
for row in matrix]
def mat_mul(a, b):
inner = len(a[0])
return [[sum(a[i][k] * b[k][j] for k in range(inner))
for j in range(len(b[0]))]
for i in range(len(a))]
R = rotation_2d(math.pi / 2) # rotate 90°
S = scaling_2d(2, 0.5) # scale (2, 0.5)
point = [1.0, 0.0]
print(mat_vec_mul(mat_mul(S, R), point)) # [0.0, 0.5] rotate first
print(mat_vec_mul(mat_mul(R, S), point)) # [0.0, 2.0] scale first
# B @ A applies A first. Swapping the order changes the destination.Now compute a composition by hand. Fill in all four entries of M = B @ A, then reveal the arithmetic if you get stuck.
M is the single matrix that applies A first, then B, so M = B @ A. Fill in the four entries, then check. Rows of B meet columns of A.
Answer before you look. The composition-order and eigenvalue-magnitude questions are exactly the ones that come up when training a recurrent network misbehaves.
0 / 5 answered · 0 correct
01What is an eigenvector of a matrix?
02What does the determinant of a 2D transformation represent geometrically?
03Why is R @ S different from S @ R?
04In an RNN, what happens when the weight matrix has eigenvalues with magnitude greater than 1?
05A = [[2, 1], [1, 2]] has eigenvalues 3 and 1. What does A = V D V⁻¹ reveal?
Click a card to swap the lazy description for what it actually means.
Three short problems. Try first; a worked answer is one click away.
Rotation R = [[0.7071, −0.7071], [0.7071, 0.7071]] sends the corners to (0, 0), (0.7071, 0.7071), (0, 1.4142), (−0.7071, 0.7071). All four distances between neighboring corners are still 1 and both diagonals are still √2, because RᵀR = I. Scaling S = [[2, 0], [0, 3]] sends the corners to (0, 0), (2, 0), (2, 3), (0, 3): it stretches side lengths to 2 and 3 and multiplies area by 6. Shear Sh = [[1, 1], [0, 1]] sends them to (0, 0), (1, 0), (2, 1), (1, 1): only the second coordinate of the top edge moved, and the area is still 1 because det Sh = 1·1 − 1·0 = 1.
Trace = 4 + 3 = 7 and det = 4·3 − 2·1 = 10, so the characteristic equation is λ² − 7λ + 10 = 0, which factors as (λ − 5)(λ − 2) = 0. Eigenvalues 5 and 2. For λ = 5: (A − 5I) = [[−1, 2], [1, −2]] gives v = [2, 1]; check A @ [2, 1] = [4·2 + 2·1, 1·2 + 3·1] = [10, 5] = 5 · [2, 1] ✓. For λ = 2: (A − 2I) = [[2, 2], [1, 1]] gives v = [1, −1]; check A @ [1, −1] = [4 − 2, 1 − 3] = [2, −2] = 2 · [1, −1] ✓. Sanity checks: the eigenvalues sum to the trace (5 + 2 = 7) and multiply to the determinant (5 · 2 = 10).
Apply R first, so M = Sh @ S @ R. With R = [[0.8660, −0.5], [0.5, 0.8660]], S @ R = [[1.2990, −0.75], [0.4, 0.6928]], and M = Sh @ (S @ R) = [[1.4190, −0.5422], [0.4, 0.6928]]. The determinants are det R = 1, det S = 1.5 · 0.8 = 1.2, det Sh = 1, so the product is 1.2. Direct check: det M = 1.4190 · 0.6928 − (−0.5422)(0.4) = 0.9831 + 0.2169 = 1.2 ✓. A circle of 8 points would come back as the same circle stretched by area 1.2, sheared and turned — the determinant only tracks area, not shape.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 01, Lesson 03) and the Math Foundations Notebook reference build. Interactive figures, the transformation and eigen labs, the RNN trajectory, worked exercise answers, and the composition trainer are original to this page. Every lab runs in your browser.