Vectors are points. Matrices are machines that move them. The dot product is a similarity meter. Those three ideas quietly run every neural network — so we built them as pictures first.
A vector is just a list of numbers — but each number is a coordinate. Give it 768 numbers instead of two and it can stand for the meaning of a word. The rules never change; the room gets bigger.
length = √(3² + 2²) ≈ 3.61It can rotate, stretch, shear, squash or reflect. A neural-network layer is exactly one of these machines, and the numbers inside it are what training adjusts. In AI, the matrices are the model.
output = W @ x + bMultiply matching coordinates and add them up. Positive means same direction, zero means perpendicular, negative means opposite. Search, recommendations and attention all rank by this number.
a · b = |a| |b| cos θA vector is a point (or an arrow to it), a matrix is a machine that moves points, and the dot product is a similarity meter. Everything else in this lesson is bookkeeping about those three ideas.
By the end you will be able to explain cosine similarity in one sentence, predict what a matrix does to a shape before doing any arithmetic, and spot when a dataset is secretly carrying fewer dimensions than it claims.
Open any machine-learning paper and you meet vectors in the first paragraph. They are not abstract symbols: a vector is a point in space, and every number is a coordinate.
The vector [3, 2] is the point three steps right and two steps up. Drawing an arrow from the origin to that point is the usual picture, and the arrow’s length follows straight from Pythagoras: √(3² + 2²) = √13 ≈ 3.61. The arrow has a direction, the point has a location, and they are the same object.
A word embedding is the same idea at scale: a word becomes a list of 768 numbers, and similar words land at similar points. You cannot draw 768 axes, but the arithmetic — adding, measuring, comparing — never changes. That is the quiet superpower of this subject.
Drag the point. The arrow is the vector; the dashed legs are its coordinates. Nothing changes when the list gets longer — only the number of legs.
A word embedding does the same thing with 768 coordinates. You cannot draw it, but every rule on this page still holds.
The dot product turns two arrows into one number, and that number answers a single question: how much do these two point the same way?
The recipe could not be simpler: a · b = a₁b₁ + a₂b₂ + … + aₙbₙ. For a = [1, 2, 3] and b = [4, 5, 6] that is 1·4 + 2·5 + 3·6 = 32. The result is a single number, not another vector, and its sign is the headline:
Divide by both lengths and the scale falls away: you get cosine similarity, trapped between −1 and 1. It is the number a search engine ranks documents by. The interactive below lets you feel why.
Drag the tips of a and b. Watch the sign flip as they pass 90°, and notice what stretching b does to the meter.
Stretch b: the dot product grows, but cos θ is untouched. Cosine compares directions; the raw dot product also feels length — which is why retrieval systems usually normalize first.
Start from the definition — multiply matching coordinates and add — and ask what it has to do with the angle between the arrows.
Sign check with no angles at all: the dot product is positive exactly when cos θ is positive, which is exactly when θ < 90°. So “positive means similar” is a theorem, not a slogan.
Read a matrix as a table of numbers and it is forgettable. Read it as a set of instructions for moving every point at once and it becomes the most useful object in AI.
The rotation matrix [[0, −1], [1, 0]] turns [3, 1] into [−1, 3] — the same arrow spun 90° counter-clockwise. Nothing was lost or duplicated; every point in the plane turned together. A neural-network layer is one of these machines too, just with more rows and columns: it turns a 3-number input into a 2-number output, and the numbers inside the matrix are exactly what training adjusts. In AI, the matrices are the model.
Here is the key to reading any matrix before multiplying anything: column 1 is where the vector [1, 0] lands, and column 2 is where [0, 1] lands. The whole grid follows. Drag those two arrows below and watch the entire plane obey.
Edit the four numbers of the matrix, or drag the two transformed basis arrows. The purple dashed lines are eigen-directions: the only directions the matrix leaves unturned.
Column 1 is where e₁ = [1, 0] lands; column 2 is where e₂ = [0, 1] lands. Read the matrix, and you already know how space moves.
The second half is the secret of why matrices feel linear: a matrix moves sums like the sums of moved pieces, so knowing where the two basis vectors land tells you where everything lands.
A 2×3 matrix turns three input numbers into two output numbers. Click an output to see the dot product that produced it. This is every dense layer ever built.
No activation here — just the linear part. Stack two of these without a non-linearity and they collapse into a single matrix, which is why every layer adds ReLU or tanh afterwards.
Not every vector you add gives you somewhere new to go. Knowing how many directions you really have is what rank measures — and it decides whether a model has a unique answer or infinitely many.
Vectors are linearly independent when none of them can be built by scaling and adding the others. Take v₁ = [1, 0, 0], v₂ = [0, 1, 0] and v₃ = [2, 1, 0]. The third is 2·v₁ + v₂, so the set is dependent. All three lie flat in the xy-plane; no combination ever reaches [0, 0, 1]. Three vectors, but only two dimensions of freedom.
A basis is a minimal independent set that reaches every point in the space. The rank of a matrix is how many independent columns (equivalently rows) it has: the number of dimensions that actually carry information. Try to escape the plane yourself:
Add vectors one at a time and try to reach the target. Sliders choose the coefficients; the green arrow is the point you can actually build.
A second, independent direction opens up a whole plane.
Rank is the size of the span: 1 for a line, 2 for a plane, 3 for space. Adding a vector that is already a combination adds nothing.
Row reduction is the mechanical way to count independent vectors. Stack the vectors as rows and use two legal moves: swap rows, or subtract a multiple of one row from another. Neither move changes the space the rows span. Stop when each remaining non-zero row starts further right than the one above it. The number of non-zero rows is the rank.
The zero row is the algebra saying “v₃ was nothing but 2·v₁ + v₂”. It is not a coincidence that this matches the picture: a row that reduces to nothing added no new direction.
| Situation | Rank | What it means for ML |
|---|---|---|
| Full rank | = min(m, n) | Unique least-squares solution. Model is well-conditioned. |
| Rank deficient | < min(m, n) | Redundant features. Infinitely many weight solutions. Regularization needed. |
| Rank 1 | 1 | Every column is a scaled copy of one vector. All data lies on a line. |
| Nearly deficient | tiny singular values | Ill-conditioned: small noise in, big swings out. Use SVD truncation or ridge regression. |
Gram-Schmidt turns any independent set into perpendicular unit vectors — the friendliest coordinate system you can compute with. It is one idea applied repeatedly: remove the shadows.
Projecting a onto b keeps only the part of a that points along b: proj_b(a) = (a·b / b·b) · b. With a = [3, 4] and b = [1, 0] the result is [3, 0]: the y-component is simply thrown away. That is dimensionality reduction in its smallest form, and it is what PCA does with the directions of highest variance.
We want the piece of a that lies along b. Call it k·b for some unknown number k — it must point along b, so it is a multiple of b. The leftover, a − k·b, should be perpendicular to b; that is what “the shadow” means.
Turn any independent pair into perpendicular unit vectors: normalize the first, subtract its shadow from the second, normalize what remains.
This is the engine inside QR decomposition, which is how software solves least-squares problems without squaring the condition number.
Nothing new happens in a transformer. Embeddings are vectors, attention is dot products, and fine-tuning is rank. Here is each one as a picture.
Embeddings turn meaning into geometry. A retrieval system stores a vector for every document chunk. Your question becomes a vector too, and the answer is whichever chunks point most nearly the same way. Click around the map to see the ranking come out of the angles.
Each word is a point. Similar meanings sit in similar directions. Click a word — the lines show its three nearest neighbours by cosine similarity, exactly how a vector database ranks results.
RAG, recommendations and deduplication all do this one move: embed, compare directions, return the highest scores. The lengths drop out because cosine ignores them.
Attention is a grid of dot products. Each token proposes a query and offers a key. Multiply them, run the scores through softmax so each row sums to 100%, and the row tells the token where to read from. The sentence below uses four two-dimensional “tokens” — the real thing uses 64–128 dimensions per head.
Give each token a vector. Every query token scores every key token with a dot product, softmax turns the scores into percentages, and the row becomes “where should I look?” Click a row to inspect it.
| query ＼ key | the | cat | sat | mat |
|---|---|---|---|---|
| 27% | 24% | 26% | 24% | |
| 5% | 59% | 2% | 34% | |
| 31% | 14% | 39% | 17% | |
| 8% | 53% | 4% | 35% |
Real transformers use 64–128 dimensions per head, scale the scores, and run dozens of heads at once. The operation at the center is still this table.
Fine-tuning is rank in disguise. A full update of a 4096×4096 weight matrix costs ~16.8 million numbers per matrix. LoRA bets that the useful change lives in a small subspace and writes it as two skinny matrices whose product is still 4096×4096 — for a fraction of the parameters.
Fine-tuning a 4096×4096 weight matrix touches ~16.8 million numbers. LoRA bets the useful update lives in a small subspace and writes it as two skinny matrices. Move the rank slider.
LoRA trains A and B while W stays frozen, then adds A @ B back at inference time. “r = 16” in a config file is this lesson.
Library calls hide arithmetic. Building the operations from scratch once makes every later error message readable — and takes less than an hour.
The lesson insists you build these before reaching for NumPy, and the reason is diagnostic: once you have written dot as “multiply pairs and sum”, a shape error in PyTorch is never mysterious again. Here is the whole class, small enough to read in one sitting.
class Vector:
def __init__(self, components):
self.components = list(components)
def __add__(self, other):
return Vector([a + b for a, b in zip(self.components, other.components)])
def __sub__(self, other):
return Vector([a - b for a, b in zip(self.components, other.components)])
def dot(self, other):
return sum(a * b for a, b in zip(self.components, other.components))
def magnitude(self):
return sum(x**2 for x in self.components) ** 0.5
def normalize(self):
mag = self.magnitude()
return Vector([x / mag for x in self.components])
def cosine_similarity(self, other):
return self.dot(other) / (self.magnitude() * other.magnitude())
def project_onto(self, other):
scalar = self.dot(other) / other.dot(other)
return Vector([scalar * x for x in other.components])
a = Vector([1, 2, 3])
b = Vector([4, 5, 6])
print(a.dot(b)) # 32
print(f"{a.magnitude():.3f}") # 3.742
print(f"{a.cosine_similarity(b):.3f}") # 0.975
print(a.project_onto(b)) # Vector([1.662, 2.078, 2.494])The same operations in NumPy and PyTorch — what you will actually use in practice. Notice that the from-scratch version does not become obsolete; it becomes the thing you can debug.
import numpy as np
a = np.array([1, 2, 3], dtype=float)
b = np.array([4, 5, 6], dtype=float)
a @ b # 32.0 (dot product)
np.linalg.norm(a) # 3.7417 (magnitude)
(a @ b) / (np.linalg.norm(a) * np.linalg.norm(b)) # 0.9746
np.linalg.matrix_rank(np.array([[1, 2], [2, 4]])) # 1 (rank deficient)
# PyTorch: the same operations, with gradients attached
import torch
x = torch.randn(3, requires_grad=True)
y = torch.tensor([1.0, 0.0, 0.0])
dot = torch.dot(x, y)
dot.backward()
print(x.grad) # exactly y: the gradient of a dot productNow run the arithmetic yourself. Type any numbers, pick an operation, and watch the substitution happen before the result — the trace is literally the code above, unrolled.
Edit any entry. Each operation shows the substitution before the result — the same trace the code runs, line by line.
Answer before you look. A wrong answer with an explanation teaches more than a right answer with none.
0 / 5 answered · 0 correct
01What does the dot product of two vectors measure?
02In AI, what does “embedding” refer to?
03v1 = [1, 0, 0], v2 = [0, 1, 0], v3 = [2, 1, 0]. Are they linearly independent?
04What does the rank of a matrix tell you in machine learning?
05How does LoRA use linear algebra to fine-tune large models efficiently?
Click a card to swap the lazy description for what it actually means.
Six short problems. Try first; a worked answer is one click away.
cos θ = (a·b) / (|a|·|b|), so θ = acos(cos) · 180/π. For [1,0] and [0,1]: the dot is 0, both lengths are 1, acos(0) = 90°.
[[2, 0], [0, 3]] @ [1, 1] = [2·1 + 0·1, 0·1 + 3·1] = [2, 3]. Each axis scales by its own factor.
Normalize each vector, then compare all 10 pairs with a dot product of the normalized vectors. Keep the pair with the highest score. Random 50-d vectors are near-perpendicular on average (cosine ≈ 0), so any clear winner is meaningful.
Loop over the returned basis: every |uᵢ| should print 1.000000, and every uᵢ·uⱼ (i ≠ j) should print 0.000000. The lesson's script prints exactly those checks at the end.
For example [[1, 0, 1], [0, 1, 1], [0, 0, 0]] has rank 2. Its columns all lie in a single plane through the origin (an infinite flat sheet) — one dimension of information is missing.
k = (1 + 2 + 3) / (1 + 1 + 1) = 2, so the projection is [2, 2, 2] — the mean of the entries, repeated. It is the constant vector closest to [1, 2, 3]: the best “flat” approximation along the all-ones direction.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 01, Lesson 01) and the Math Foundations Notebook reference build. Interactive figures, added AI visuals, worked exercise answers, and the math console are original to this page. No account, tracking, or server is involved — every lab runs in your browser.