A layer is a matrix multiply, a bias, and a squashing curve. Chain layers and raw inputs become features no single line could see. This is the forward pass — the half of training that never learns.
XOR puts [0,1] and [1,0] on one side, [0,0] and [1,1] on the other — and no single line can separate them. Minsky and Papert proved that limit in 1969. The fix is not a smarter neuron; it is a stack of them, where the first layer builds new features and the next layer decides with those features.
stack neurons → new features → curvesA layer holds a weight matrix W with one row per neuron and one column per input — shape (neurons now, neurons before) — plus one bias per neuron. The linear step z = W·x + b moves and stretches the space; the activation a = σ(z) bends it. Both halves appear in every layer of every deep model.
W is (now × before) · a = σ(W·x + b)Data goes in, flows through every layer, comes out. Nothing is learned or changed here: no loss, no gradients, just multiply–add–activate on repeat. The intermediate vectors are the hidden state — for text that can be 768 numbers per token; the last layer reads that state out as a decision.
x → a¹ → a² → … → y, pure computationA network is a composition of functions: x → a¹ → a² → … → y. Each layer rewrites its input into a new set of coordinates, and the whole job of the hidden layers is to build coordinates in which the output layer’s one straight decision is enough.
By the end you will be able to run a forward pass by hand through a 2-2-1 and a 2-3-1 network; check every weight shape before running anything; explain why two linear layers collapse into one matrix and why the activation is the only reason depth pays; count an architecture’s parameters; state the universal approximation theorem without overclaiming; and narrate exactly how a hidden layer makes XOR solvable.
The perceptron from Lesson 03.01 is a line drawer — that is its entire power. In 1969 Minsky and Papert showed XOR is not “hard” for it; it is impossible. The escape route is not a smarter neuron, but a stack of ordinary ones.
A single neuron computes one weighted sum and passes it through a threshold. Geometrically, that is a straight line: everything on one side is class 1, everything on the other side is class 0. For a dataset where a line can do the job — the OR gate, a spam filter with a good keyword — that is enough. XOR is not such a dataset.
Minsky and Papert’s 1969 result was precise: a single-layer network cannot learn XOR, no matter how long you train or how you tune its weights. No line separates the classes. The result landed at the worst possible time, funding drained away, and neural networks went quiet for more than a decade.
The fix, obvious in hindsight, is to stop insisting on one line. Stack neurons into layers: let the first layer carve the input space into new features, and let the second layer combine those features into a decision that no single line could make. That stack is the multi-layer network — the object under every production deep learning model — and the forward pass is the piece you build first.
| x₁ | x₂ | y |
|---|---|---|
| 0 | 0 | 0 |
| 0 | 1 | 1 |
| 1 | 0 | 1 |
| 1 | 1 | 0 |
XOR has exactly four examples and the perceptron from Lesson 03.01 has three parameters. Why can't it solve XOR, even with perfect training?
A multi-layer network arranges neurons in columns: the input holds the raw data, the hidden layers do the work, and the output layer answers the question. Only the hidden and output layers compute — and the middle ones are where the learning lives.
The input layer is not really a layer. It holds your raw data — pixels, sensor readings, words — one node per feature. No weights, no biases, no computation. Two features means two input nodes.
Hidden layers are where work happens. Each neuron takes every output from the previous layer, multiplies each by its own weight, adds a bias, and pushes the sum through an activation function. The results form a vector called the hidden state. “Hidden” because those values never appear in your training data — the network invented them.
The output layer gives the final answer. For binary classification it is one neuron with a sigmoid: a number in (0,1) that reads as a probability. For multi-class problems it is one neuron per class. Everything before it exists to make this last, simple decision possible.
The activation used throughout this lesson is the sigmoid. It squashes any real number into (0,1): large positives approach 1, large negatives approach 0, and zero maps to exactly 0.5. Unlike the perceptron’s hard step, the curve is smooth and differentiable everywhere — its derivative is σ(z)·(1−σ(z)) — which is what makes gradient-based learning possible in Lesson 03.03.
The hidden state is the point of the whole exercise. For text models it expands: a single token enters as a handful of numbers and travels as a 768-number vector (BERT-base) or a 4096-number vector (Llama-2-7B), with each layer rewriting that vector in light of its context. For images it compresses: millions of pixels funnel down to a few hundred numbers that summarize the picture. Either direction, the hidden state is a representation the network built for itself, and the next layer reads it instead of the raw input.
Move the inputs, switch the weights, and watch the numbers climb the network: pre-activations z, activations a, and one output. Nothing here learns — this is pure computation.
With the XOR weights at full scale, σ acts like a step: h₁ ≈ OR, h₂ ≈ NAND, and the output is their AND — which is exactly XOR. Slide the weight scale down and watch the sharp corners melt: the same architecture stops separating the classes because sigmoid is no longer saturating.
The forward pass pushes data through the network, layer by layer, until it reaches the output. Nothing is learned or changed along the way — it is pure computation, and it is the foundation everything else in this phase stands on.
At every layer, three operations happen in sequence. First the linear transformation z = W·x + b: each neuron takes a weighted sum of the previous layer’s outputs and shifts it by its bias. Then the activation a = σ(z) squashes each number through the sigmoid curve. Then the output vector becomes the next layer’s input.
In plain English: W holds one row of weights per neuron, so W·x just says “every neuron listens to every input, with its own volume knob per input.” The bias then sets how easy that neuron is to switch on: a large negative bias means the neuron stays quiet even when its inputs are positive. And σ keeps every value in a bounded range so the next layer receives a consistent, comparable signal.
That is the entire forward pass. No learning happens here — the weights are whatever they are, and the pass just computes. Training (backpropagation, Lesson 03.03) is a separate loop that runs a forward pass, measures the error, and only then nudges the weights.
import math
def sigmoid(z):
z = max(-500.0, min(500.0, z)) # clamp: math.exp(1000) overflows
return 1.0 / (1.0 + math.exp(-z))
class Layer:
def __init__(self, n_inputs, n_neurons, weights, biases):
self.weights = weights # (n_neurons, n_inputs)
self.biases = biases # (n_neurons,)
def forward(self, inputs):
self.last_input = inputs
self.last_output = []
for neuron_idx in range(len(self.weights)):
row = self.weights[neuron_idx]
z = sum(w * x for w, x in zip(row, inputs))
z += self.biases[neuron_idx]
self.last_output.append(sigmoid(z))
return self.last_output
class Network:
def __init__(self, layers):
self.layers = layers
def forward(self, inputs):
current = inputs
for layer in self.layers: # k's output feeds k+1
current = layer.forward(current)
return currentThe XOR network later in this lesson is the famous example; here is a second pass, end to end, so the arithmetic is visible before any of it becomes familiar. Weights and biases are ordinary small numbers — no saturation, no tricks.
Follow one number all the way: h₂ came from z¹₂ = 0.230, which came from x₂ = 0.8 carrying its weight 0.1, plus x₁ = 0.5 carrying 0.7, minus the bias 0.2. Then h₂ went into the output neuron with weight −0.6: it was the only hidden neuron voting against the positive class, and without it z² would have been 0.796 (y = 0.689 instead of 0.613). Every weight matters a little; some matter a lot.
Pick one of the four XOR inputs, then step through the arithmetic of the hand-tuned 2-2-1 network. This is the table a debugger prints.
σ(10) = 0.999955 because e^(−10) is 0.0000454 — the curve is already flat out here. Large weights push every neuron into the flat region, which is what turns a smooth sigmoid into the step function XOR needs.
You run a forward pass on one training example. Which of these does the network change?
Tracking dimensions is the single most important debugging skill in deep learning. The rule fits in one line, and half of all beginner errors — including the silent ones — come from breaking it.
Every matmul has a contract: the number of columns on the left must equal the number of entries on the right. A layer’s weight matrix W is shaped (neurons in this layer, neurons in the previous layer) — rows match the current layer, columns match the previous one — so W·x lines up by construction. The bias has one entry per neuron, so it matches the result’s shape exactly. If the shapes do not line up, you have a bug.
There is no magic beyond that. The activation is element-wise, so it never changes a shape. The output of one layer is a vector whose length equals that layer’s neuron count, and it arrives at the next layer as the “previous layer” whose length the columns must match. Walk the 2-3-1 network from Chapter 02:
| Step | Operation | Dimensions | Result shape |
|---|---|---|---|
| Input | x | — | (2,) |
| Hidden linear | W¹·x + b¹ | W¹: (3, 2), b¹: (3,) | (3,) |
| Hidden activation | σ(z¹) | — | (3,) |
| Output linear | W²·h + b² | W²: (1, 3), b²: (1,) | (1,) |
| Output activation | σ(z²) | — | (1,) |
Two numbers decide whether every layer works: the current layer’s neuron count (rows) and the previous layer’s output length (columns). The bias matches rows. Everything else follows.
Build an architecture, then break it on purpose. Shape errors are the most common bug in deep learning — this board makes the inner dimensions visible before the framework prints a stack trace.
PyTorch writes nn.Linear(in, out) but stores the weight as (out, in) — the same (now × before) rule. The 2-3-1 network from the source has W1 (3×2) and W2 (1×3); swap either and the chain breaks or silently lies.
A hidden layer has 5 neurons and receives the 3-number output of the previous layer. What is the shape of its weight matrix, and what shape comes out?
Two matrix multiplies in a row are just one matrix multiply with extra steps. The activation function between them is the only thing that stops a hundred-layer network from collapsing into a single linear map — and the only reason stacked layers can draw curves.
A dense layer without its activation applies an affine map — a linear transformation plus a shift. Compose two of them and you do not get a richer kind of function — you get one affine map that happens to be computed in two steps. The algebra says it plainly: W²·(W¹·x + b¹) + b² = (W²W¹)·x + (W²b¹ + b²). Any stack of purely linear layers, however deep, is equivalent to one layer with a combined matrix.
The sigmoid between the layers breaks the factorization. σ bends each coordinate independently, so the second layer’s matrix multiplies a warped version of the input rather than the input itself. That is why Chapter 07’s XOR network works: the hidden layer moves the four points into a new coordinate system where they can be separated by a line, and the output layer draws that line.
Take a two-layer stack with W¹ = [[2, 0], [1, −1]] and b¹ = [1, 2], W² = [[1, 1]] and b² = [0.5], and a probe input x = [3, 4].
With no activation the two-layer network and the single matrix agree to the last digit: 8.5 both ways. Add σ and the same weights produce 0.9029 for the same input — different outputs, so the two functions are genuinely different. The difference is also structural: the activated version is bounded inside (0,1) for every input, while the linear version grows without bound. The nonlinearity is not decoration. It is the difference between a deep network and an expensive identity.
The grid is the space of possible inputs, warped by each layer in turn. With no activation between layers, the dashed single matrix W₂W₁ traces the exact same curves as the two-step network. Switch the nonlinearity on and watch the shortcut fail.
The algebra: W₂(W₁x) = (W₂W₁)x. The numeric check above is the whole proof — and it is also the trap: stacking layers is only worth it because the activation happens between them.
You build a 5-layer network but forget to put an activation after each dense layer. What function is your model actually computing?
In 1989 George Cybenko proved that a single hidden layer, given enough neurons, can approximate any continuous function to any accuracy. It is the theoretical license for everything since — and it is easy to read as a promise it never made.
The intuition is the one Chapter 05 set up. Each hidden neuron adds one bend to the function; a neuron with a steep sigmoid is a step, and a step with a negative output weight carves a bump. Place enough bumps and you can trace any smooth curve. More neurons, more bumps, better approximation — up to any accuracy you name, for any continuous function on a bounded region.
What the theorem does not say is that one hidden layer is always best. It is a statement about representation: there exists a set of weights that does the job. Practice cares about two more questions — will training find those weights, and will they generalize to new data? Deeper networks answer both better: a stack of layers reuses features, so it reaches the same accuracy with far fewer total parameters than the shallow-wide network the theorem needs. That is why deep learning works.
Each faint line is one hidden sigmoid neuron with its output weight; the bold line is their sum. Add neurons and watch the staircase melt into the target — the universal approximation theorem, drawn.
The theorem is about representation, not magic: enough neurons can approximate any continuous function, but “enough” can be enormous, and it says nothing about learning those weights from finite data.
The lab above fits a network output to a target curve with fixed neuron positions and least-squares output weights — a teaching sketch, not a training procedure, but a real measurement of approximation error.
The last line is the honest reading of the theorem. A single wide layercan represent the pixel-to-label function; the parameter count to do it grows so fast that a stacked network with many fewer numbers — each layer building on the last — is the only practical way to get there.
The 2-2-1 network from the source solves XOR with 6 weights and 3 biases — chosen by a human, not learned. Walking its forward pass is the clearest proof that a hidden layer can turn an impossible line problem into an easy one.
The architecture: two inputs, two hidden neurons, one output. The hidden weights are deliberately huge — ±20 — because large inputs push sigmoid into its flat regions, where it behaves like a hard step. The first hidden neuron then acts like OR (fires when x₁ + x₂ is past 0.5) and the second like NAND (stays on unless both inputs are 1). The output neuron computes their AND: it fires only when OR and NAND are both on — exactly the two cases where the inputs differ. AND(OR, NAND) is XOR.
These specific numbers were hand-tuned to make each hidden neuron compute a named logic gate. Training will not be told to do that — in Lesson 03.03 backpropagation finds its own weights, and it is free to invent features no human named. What matters here is that the forward pass works, and that it is a complete, checkable computation:
Read the [0,1] row against the [1,1] row. For [0,1] both hidden neurons are saturated on, h = (0.999955, 0.999955), and the output z² = 9.998 lands far above the 0.5 threshold. For [1,1] the second hidden neuron has flipped to 0.000045, so z² = −9.999 and the output collapses to 0. The only thing separating those rows is which hidden neuron carries the signal — a distinction no single line in the original (x₁, x₂) plane can make. Nine numbers, one curved boundary.
Drag the morph slider and watch the four XOR points move from the raw input plane into the hidden layer’s coordinates. Two of them land on top of each other, and a single straight line finishes the job.
This is the whole argument for depth: the hidden layer does not draw the curved boundary itself — it re-coordinates the data so the output layer can draw a straight one.
The same network is also the honest version of “running the model”: change an input and everything downstream recomputes; run it twice on the same input and you get the same output, because nothing is mutated. Frameworks packaged this exact computation. In PyTorch, the whole two-layer network is four lines:
import torch
import torch.nn as nn
model = nn.Sequential(
nn.Linear(2, 8), # weight (8, 2), bias (8,)
nn.Sigmoid(),
nn.Linear(8, 1), # weight (1, 8), bias (1,)
nn.Sigmoid(),
)
xor = torch.tensor([[0.0, 0.0], [0.0, 1.0], [1.0, 0.0], [1.0, 1.0]])
print(model(xor)) # forward pass on all four rows at onceAnswer before you look. The weight-shape question and the forward-pass question are the two that separate a memorized definition from a working instinct.
0 / 5 answered · 0 correct
01What is the purpose of a hidden layer in a multi-layer network?
02What does the forward pass do in a neural network?
03For a layer with 3 neurons receiving input from 2 neurons, what is the shape of the weight matrix?
04What does the Universal Approximation Theorem guarantee?
05Why does the sigmoid activation function make learning possible, unlike the step function used in perceptrons?
Click a card to swap the lazy description for what it actually means.
Four problems with exact numbers — a deeper XOR stack, parameter counting, a width sweep on the circle classifier, and a leaky-step replacement for sigmoid. Try first; a worked answer is one click away. (The source lesson also ships a reusable network-architect prompt at outputs/prompt-network-architect.md — use it when deciding depth, width and activations for a real problem.)
With weights W1 = [[0.5, −0.5], [1, 0], [−1, 1], [0, −1]], b1 = [0, 0, 0, 0], W2 = [[1, 1, −1, −1], [0.5, −0.5, 0.5, −0.5]], b2 = [0, 0], W3 = [[1, −1]], b3 = [0], the pass for input [0, 1] is: z1 = [−0.5, 0, 1, −1] → a1 = [0.378, 0.500, 0.731, 0.269]; z2 = [0.378 + 0.500 − 0.731 − 0.269, 0.189 − 0.250 + 0.366 − 0.134] = [−0.122, 0.170] → a2 = [0.469, 0.542]; z3 = 0.469 − 0.542 = −0.073 → y = 0.482, rounded to class 0 — wrong (expected 1). Input [1, 1]: z1 = [0, 1, 0, −1] → a1 = [0.500, 0.731, 0.500, 0.269]; z2 = [0.462, 0.000] → a2 = [0.614, 0.500]; z3 = 0.114 → y = 0.528, rounded to class 1 — also wrong. That is the point of the exercise: a bigger architecture still needs training; random numbers have no reason to encode XOR. What you should watch is the representation, not the accuracy — by a2 the two inputs are already different points, and after Lesson 03.03 teaches these weights, the same 4-layer chain separates them.
Layer 1: 784 × 256 + 256 = 200,960. Layer 2: 256 × 128 + 128 = 32,896. Layer 3: 128 × 10 + 10 = 1,290. Total: 200,960 + 32,896 + 1,290 = 235,146 trainable numbers. For scale: the lesson's 2-2-1 XOR network has 9, the 2-8-1 circle classifier has 33, and a deeper MNIST variant 784-512-256-128-10 has 401,920 + 131,328 + 32,896 + 1,290 = 567,434. The first layer alone is 85% of the shallow network's parameters — which is why deep networks push the work into later layers instead of widening the first one.
Parameter counts: 2 hidden neurons → layer 1 (2×2 + 2) = 6, layer 2 (2×1 + 1) = 3, total 9; 8 neurons → 24 + 9 = 33; 32 neurons → 96 + 33 = 129 (in general 4N + 1). The output range never changes: sigmoid always lands in (0, 1), whatever N is. What changes is the distribution. The output pre-activation is z₂ = Σ wᵢhᵢ + b over N random terms, so its spread grows roughly with √N while the mean drifts with Σwᵢ; more random features therefore push more outputs toward the saturated ends 0 and 1. That widening does not mean better classification — the features are still random, so accuracy stays near the majority-class baseline. Width changes capacity, not knowledge; knowledge is what training (Lesson 03.03) adds.
It still classifies all four. [0,0]: z1 = [−10, 30] → [−0.1, 1.0]; z2 = 20·(−0.1) + 20·1 − 30 = −12 → −0.12 → class 0. [0,1]: z1 = [10, 10] → [1, 1]; z2 = 20 + 20 − 30 = 10 → class 1. [1,0] is identical by symmetry → class 1. [1,1]: z1 = [30, −10] → [1, −0.1]; z2 = −12 → class 0. With weights of ±20 the nonlinearity only has to be approximately a step, so the exact shape barely matters at this scale. But training requires a slope: the leaky step is flat (derivative 0) for every z ≥ 0, so once a neuron is on, no gradient can tell it how to move, and the kink at 0 is not differentiable. Sigmoid's derivative σ(1−σ) is positive everywhere, so every neuron keeps receiving guidance — that is why smooth wins even when a hard step would work by hand.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 03, Lesson 02) and the Math Foundations Notebook reference build. The six labs (forward-pass playground, hand-forward stepper, shape checker, composition lab, universal-approximation bump lab, XOR hidden-space view), the full 2-2-1 XOR table with activations to six decimals, the second 2-3-1 worked pass, the W₂W₁ collapse check, the parameter counts for the classic architectures, the width-sweep and leaky-step exercise traces, and the factory quality-gate scenario are original to this page. Every number shown is computed live by the labs or verified by hand in the prose.