output = relu(W @ x + b) is a neural network layer. Four symbols: a shape rule, a row-by-column dot product, a broadcast add, and one bend.
Matrix multiplication has one non-negotiable rule: (m × n) @ (n × p) = (m × p). The shared inner dimension is consumed; the outer two survive. Almost every shape error you will ever see is this rule being violated.
(128×784) @ (784×1) = (128×1)Each entry of A @ B is one row of A dotted with one column of B. Shapes govern whether it runs; the dot products are what it computes. The columns view is the same product read backwards.
result[i][j] = row_i · col_jW @ x is a column vector, and b is too — but a bias for a whole batch gets stretched across rows. NumPy aligns shapes from the right and repeats any size-1 or missing dimension. No copy is stored.
relu(W @ x + b)The line output = relu(W @ x + b) is a neural network layer: @ mixes the inputs according to the shape rule, + b shifts the result, and relu bends it so stacked layers cannot collapse into one.
By the end you will be able to read any layer’s shapes on sight, tell element-wise multiplication from matrix multiplication in a code review, predict broadcast results, and know when a matrix has no inverse.
An m × n matrix has m rows and n columns. One rule decides whether two matrices can multiply at all — and the rule is checkable at a glance.
Matrix multiplication follows (m × n) @ (n × p) = (m × p). The inner dimensions must match, and they disappear; the outer two survive and become the result. A layer with 784 inputs and 128 outputs uses a 128 × 784 weight matrix, so (128×784) @ (784×1) = (128×1). Almost every “shape mismatch” error you will ever see in PyTorch is this rule being violated.
The rule is not bookkeeping busywork. It tells you how much work the multiplication costs, whether the output can be fed to the next layer, and — once you can see it — exactly which line to fix when a model refuses to run.
Move the four sliders. The inner dimensions must match; the outer dimensions survive and become the result.
Each of the 4 result entries is a dot product of length 3: 12 multiply-adds total.
A layer with 784 inputs and 128 outputs uses a 128×784 weight matrix, so (128×784) @ (784×1) = (128×1). Same rule, bigger numbers.
Notice the batch dimension rides along on the left. That is why production code writes weights as (out, in), not (in, out): the examples stay stacked in rows and the whole batch is one multiplication.
Before matrices mix coordinates, vectors are moved one coordinate at a time. These are the operations training code runs thousands of times per second.
Addition walks one arrow, then the other — head to tail, either order, same destination. Scalar multiplication stretches or flips an arrow without turning it. Element-wise multiplication (the Hadamard product, written * in NumPy) multiplies matching positions and needs identical shapes. Each has a one-line implementation and a picture you can hold in your head.
Drag the tips of u and v, pick an operation, and watch the result. Every one of these is used on activations and gradients every training step.
Addition moves a point along another vector; scaling changes how far it goes; element-wise multiplication acts on each coordinate independently. Matrix multiplication is the operation that mixes coordinates together — next chapter.
| Operation | What it does | Neural network use |
|---|---|---|
| Addition | Element-wise combine | Adding the bias |
| Scalar multiply | Scale every element | lr × gradient |
| Matrix multiply | Transform vectors / mix coordinates | The layer's forward pass |
| Transpose | Swap rows and columns | Backprop (gradients flow through Wᵀ) |
| Determinant | One number: area / volume scale factor | Checking invertibility, normalizing flows |
| Inverse | Undo a transformation | Solving linear systems |
| Identity | Do nothing | Initialization, residual connections |
The same product can be read as a grid of dot products or as one matrix mixing the columns of another. Both views are used constantly in ML code.
Matrix multiplication takes a dot product of each row of the left matrix with each column of the right matrix. Element-wise multiplication is a different operation that happens to share a symbol in some languages. Compare them on the same two matrices:
Edit A and B, or press scan. In the first view every result cell is a row dotted with a column; in the second, every result column is a combination of A’s columns.
A @ B is not B @ A in general. Multiply a rotation by a scale one way, then the other, and compare.
Matrix multiplication is composition of transformations: applying B then A is the same as applying the single matrix A @ B. Here is the argument, entry by entry.
Adding a bias vector to a matrix of outputs should be a shape error. Instead NumPy and PyTorch stretch the smaller array along missing dimensions — by rule, not by copying memory.
Rows of a 2 × 3 matrix each receive the same 3-element bias. The dashed second row in the classic picture is never stored; broadcasting is a rule for reading indices, not a copy. The alignment direction is the part people miss: shapes line up from the trailing dimension backward, the same way decimal digits line up on the right.
Pick a shape pair. Dimensions are aligned from the right; a missing or 1-sized dimension is stretched. Anything else is an error.
dashed cells are virtually repeated, never stored
The rule is index bookkeeping: a dimension of size 1 has one value, so it can answer for any index. No memory is shared or copied.
Once you can do step 1 in your head, the errors name themselves: “operands could not be broadcast together with shapes (2,3) (2,)” means the 3 and the 2 were compared because both are trailing.
The determinant answers with a single number. Zero means a dimension was crushed flat and no inverse can bring it back.
For a 2 × 2 matrix [[a, b], [c, d]] the determinant is ad − bc. It is the factor by which the transformation scales area. A determinant of zero means the matrix squashes the plane onto a line: information is destroyed and no inverse exists — the matrix is singular. When the determinant is non-zero, the inverse undoes the transformation and A @ A⁻¹ = I, the do-nothing identity.
Edit the matrix, apply it, then undo it with its inverse. Watch the dashed unit circle stretch into an ellipse — the determinant is its area factor.
The recipe for 2×2: swap the diagonal, negate the off-diagonal, divide by the determinant. Try [[1, 2],[2, 4]] and watch the circle collapse onto a line.
Area. The matrix [[a, b], [c, d]] sends the unit square to a parallelogram with sides u = (a, c) (the first column, where [1, 0] lands) and v = (b, d) (the second column, where [0, 1] lands). Draw it inside the rectangle of width a + b and height c + d. Cut away everything that is not the parallelogram: two triangles of area ½ac, two of area ½bd, and two small rectangles of area bc. What remains:
Inverse. We want X with A @ X = I. Try X = (1/det) · [[d, −b], [−c, a]] and multiply it out:
Dividing by zero is impossible — the algebraic face of “singular matrices have no inverse”.
Shapes, matrix multiplication, broadcasting and one non-linearity — put together they are the layer that appears hundreds of times inside a transformer.
The layer is output = relu(W @ x + b). W holds one row per output feature; @ mixes the inputs according to the shape rule; + b shifts each output; relu zeroes negatives. Without the non-linearity, two stacked layers would multiply into a single matrix and depth would buy nothing.
This is output = relu(W @ x + b). Edit any number, select an output row, and watch the shape rule and the non-linearity do their work.
Stack two of these with no activation between them and they collapse into one matrix — the non-linearity is what makes depth worth anything.
A Matrix class small enough to read in one sitting makes the shape rule concrete — and every framework error after that names itself.
Notice the guard at the top of matmul: it checks the inner dimensions and raises a message that includes both shapes. The frameworks do exactly this; writing it yourself once turns their errors from cryptic to obvious.
class Matrix:
def __init__(self, data):
self.data = [list(row) for row in data]
self.rows, self.cols = len(self.data), len(self.data[0])
def matmul(self, other):
if self.cols != other.rows:
raise ValueError(f"cannot multiply {self.rows}x{self.cols} by {other.rows}x{other.cols}")
return Matrix([
[sum(self.data[i][k] * other.data[k][j] for k in range(self.cols))
for j in range(other.cols)]
for i in range(self.rows)
])
def __add__(self, other):
return Matrix([[a + b for a, b in zip(r1, r2)]
for r1, r2 in zip(self.data, other.data)])
def transpose(self):
return Matrix([[self.data[j][i] for j in range(self.rows)]
for i in range(self.cols)])
def __repr__(self):
return "Matrix(" + str(self.data) + ")"
def relu(m):
return Matrix([[max(0, v) for v in row] for row in m.data])
x = Matrix([[0.5], [0.8], [0.2]]) # (3 x 1) input
W = Matrix([[0.2, -0.4, 0.9], [0.7, 0.1, -0.3]]) # (2 x 3) weights
b = Matrix([[0.1], [0.1]]) # (2 x 1) bias
out = relu(W.matmul(x) + b) # (2 x 1)
print(out) # Matrix([[0.06], [0.47]]) <- this IS a dense layerNumPy and PyTorch express the same thing with @, and add broadcasting for free. The batched version at the bottom is the one that runs in production, and it is still a single matrix multiply.
import numpy as np
x = np.array([0.5, 0.8, 0.2]) # (3,)
W = np.array([[0.2, -0.4, 0.9],
[0.7, 0.1, -0.3]]) # (2, 3)
b = np.array([0.1, 0.1]) # (2,)
z = W @ x + b # (2,)
y = np.maximum(0, z) # relu
print(y) # [0.06 0.47]
batch = np.random.randn(32, 3) # (32, 3)
z_batch = batch @ W.T + b # (32, 2) <- W.T: (3, 2)
# a batch is not a loop: it is one matrix multiplyNow run the operations yourself. Edit A and B, pick an operation, and watch the arithmetic happen before the result.
Edit A and B, pick an operation, and read the arithmetic before the result — the same trace the Matrix class runs.
Answer before you look. The shape rule and broadcasting questions are exactly the ones that come up in real debugging sessions.
0 / 6 answered · 0 correct
01For (m × n) @ (n × p), what must be true about the dimensions?
02How is the identity matrix defined?
03What is the key difference between element-wise and matrix multiplication?
04In output = relu(W @ x + b), what role does broadcasting play?
05Which property follows from a zero determinant?
06How does the transpose operation change a matrix?
Click a card to swap the lazy description for what it actually means.
Three short problems. Try first; a worked answer is one click away.
For A = [[1,2],[3,4]]: det = −2 and A⁻¹ = [[−2,1],[1.5,−0.5]], so A @ A⁻¹ = [[1,0],[0,1]]. Try [[2,0],[0,3]] (det 6) and [[0,1],[1,0]] (det −1). When det = 0, as in [[1,2],[2,4]], the inverse formula divides by zero — there is no matrix that can un-crush the plane.
Inverse = (1/det) · adj(A), where adj(A) is the transpose of the cofactor matrix. Each cofactor Cᵢⱼ is the 2×2 determinant left after deleting row i and column j, with alternating signs. Compare entry by entry with np.linalg.inv(A) using np.allclose — expect agreement to ~1e-12.
Shapes: x (3×1), W₁ (4×3), b₁ (4×1), W₂ (2×4), b₂ (2×1). Then h = relu(W₁ @ x + b₁) is (4×1), and y = W₂ @ h + b₂ is (2×1). Check each product: (4×3)@(3×1) → (4×1), (2×4)@(4×1) → (2×1). A batch of 32 examples is x (3×32), h (4×32), y (2×32) — same weights, batched on the right.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 01, Lesson 02) and the Math Foundations Notebook reference build. Interactive figures, added AI visuals, worked exercise answers, and the matrix console are original to this page. Every lab runs in your browser.