The perceptron is the atom of neural networks: multiply inputs by weights, add a bias, then step. It learns by fixing its own mistakes — until the data cannot be split by one straight line. That failure is where depth begins.
Multiply each input by its weight, add a bias, and step the total: 1 if it is at or above zero, 0 otherwise. With w = (1, 1) and b = −1.5 the neuron computes AND; keep the weights and raise b to −0.5 and it computes OR. Three numbers, one decision.
z = w·x + b → step(z)wᵢ ← wᵢ + lr · error · xᵢ and b ← b + lr · error. Predicted 0 but should be 1? Add the inputs, scaled. Predicted 1 but should be 0? Subtract. The weights set the line's direction; the bias slides it sideways — and a full pass with zero mistakes ends training.
no error, no changeXOR asks for 1 when exactly one input is 1 — its two class-1 rows sit on one diagonal, its class-0 rows on the other. No straight line splits them; the best any line does is 3 of 4. Stack an OR neuron and a NAND neuron, feed their outputs to an AND neuron, and XOR is solved.
separability is a property of the dataA perceptron is a weighted vote with a threshold: each input votes, its weight says how loudly, the bias says how many votes it takes to win — and learning is the tally being rewritten after every wrong answer.
By the end you will be able to compute a perceptron’s output and boundary by hand; run the learning rule step by step on AND, OR, NOT and NAND; explain why XOR is impossible for a single line at any learning rate; build the two-layer OR × NAND → AND fix; and say exactly what the step function does to gradients — the bridge to backpropagation.
You know vectors and dot products. The perceptron answers the next question — how does a machine learn which transformation to use? — with the smallest possible machine: weights, a bias, and a step.
Take some inputs. Multiply each by a weight. Add a bias. If the total is at least zero, output 1; otherwise output 0. That is the entire perceptron — the source lesson’s words, not a simplification. Every neural network ever built is layers of this idea stacked together.
Each weight does two jobs at once: its sign says which side of the decision that input argues for, and its size says how loudly. The bias is the one number that does not scale any input — it shifts the whole decision up or down, which is why it is described as the threshold. And the step function is the hard part: it makes the output binary, and (as the last chapter shows) it also kills gradients.
Take x = (2, 3), w = (0.5, −0.4). The weights disagree: x₁ argues for firing, x₂ argues against, and w₂ is losing.
In the other direction, w = (0.5, 0.1), b = 0.1 gives z = 1.0 + 0.3 + 0.1 = 1.4 → 1: when both weights agree with the inputs, the same three numbers produce a much more confident 1. Confidence is invisible to the step, though — z = 1.4 and z = 0.01 both print “1”.
Move the five dials and watch the same arithmetic the source code runs: multiply each input by its weight, add the bias, then step the total.
Sign and size both matter: flipping a weight inverts what that input argues for, while |w| sets how loud the argument is. The bias is the one dial that fires with no input at all.
Keeping the weights fixed, you raise the bias b. What happens to the perceptron?
For two inputs, the equation w₁x₁ + w₂x₂ + b = 0 is a straight line, and the perceptron’s answer is simply which side of it you are on. Training does not change the equation’s form — it moves the line until the colors split.
Set z to exactly zero and you get the decision boundary: w₁x₁ + w₂x₂ + b = 0. Every point with z ≥ 0 outputs 1; every point below outputs 0. In two dimensions that is a line; with three inputs a plane; in general a hyperplane — the same object Phase 1 used to define half-spaces.
Solving for x₂ turns the boundary into a familiar slope-intercept line, which makes the roles of the three numbers obvious:
One more piece of geometry worth memorizing: the vector w is always perpendicular to the boundary, pointing toward the class-1 side. And the quantity z measures how far a point is from switching sides, once you divide out the weights’ length: distance = |z| / ‖w‖.
Take the hand-picked AND neuron — w = (1, 1), b = −1.5. Set the boundary to zero and rearrange:
This is why the same weights with b = −0.5 turn AND into OR: only the offset changed, so the line slid down-left and swallowed (0, 1) and (1, 0). Slopes come from weights; thresholds come from bias.
Three weights, one straight line. Drag the sliders and try to separate AND and OR — then switch to XOR and watch the best you can do stall at three of four.
Only the ratio of w and b matters to the decision: multiplying all three numbers by the same positive constant draws the exact same line.
You multiply every weight and the bias by 2 — w = (2, 2), b = −3 for the same AND neuron. What changes?
This is the whole learning algorithm in six lines. Read it once as arithmetic and once as a sentence: whenever the perceptron is wrong, add the inputs to the weights (scaled) if it should have fired, and subtract them if it should not have.
Here is the source lesson’s learning rule, verbatim in spirit:
Four details do most of the work. First, error is one of three numbers — there is no gradient and no calculus here, just a sign. Second, the update adds the input vector itself, scaled: that is why the boundary rotates toward (or away from) the example that was wrong. Third, the bias update has no xᵢ — it fires on every mistake, which is exactly the sliding-the-line motion from Chapter 02. Fourth, a correct prediction contributes nothing: training ends the first time a full pass has zero mistakes.
First, the single update that a beginner most often gets wrong. It is epoch 2 of AND training, with w = (0.1, 0.1), b = −0.1, looking at the row (0, 1) whose target is 0:
Now run the full loop, starting from zeros with lr = 0.1. The tables below are the source algorithm’s actual end-of-epoch states, reproduced from the same update rule:
| epoch | weights at end | bias at end | mistakes in the pass |
|---|---|---|---|
| 1 | (0.1, 0.1) | 0.0 | 2 |
| 2 | (0.2, 0.1) | −0.1 | 3 |
| 3 | (0.2, 0.1) | −0.2 | 3 |
| 4 | (0.2, 0.1) | −0.2 | 0 — done |
| epoch | weights at end | bias at end | mistakes in the pass |
|---|---|---|---|
| 1 | (0.0, 0.1) | 0.0 | 2 |
| 2 | (0.1, 0.1) | 0.0 | 2 |
| 3 | (0.1, 0.1) | −0.1 | 1 |
| 4 | (0.1, 0.1) | −0.1 | 0 — done |
Both runs converge at epoch 4: a full pass with zero mistakes. The two traces fail in different ways, though. The OR run’s weights move only twice (the (0, 1) and (1, 0) mistakes of the first two epochs set w to (0.1, 0.1), which draws the same line as the hand-picked (1, 1)); after that, every correction is pure bias — sliding the line from x₁ + x₂ = 0 down to x₁ + x₂ = 1. The AND run keeps swapping which weight is wrong for three epochs before the direction settles on w = (0.2, 0.1) and the bias slides to −0.2. For linearly separable data this rule is guaranteed to settle in a finite number of updates — Rosenblatt’s convergence theorem. For anything else it never does, and Chapter 05 shows the smallest counterexample.
The source’s training loop, paused between examples. Step through and watch the line move only on mistakes — and stop moving once a full pass is clean.
Weights start at zero, exactly like the source class. Because step(0) = 1, the very first visit — (0, 0) with target 0 — is already a mistake, and its update lands entirely on the bias: the input is (0, 0), so there is nothing for the weights to grab.
Training starts at w = (0, 0), b = 0 and the first example is (0, 0) with target 0. What happens?
AND, OR, NOT and NAND all have something in common: plot their rows in the unit square and a straight line separates the 0s from the 1s. That is precisely the class of problems one perceptron can solve — and the perceptron learning rule finds the line by itself.
The source lesson trains three gates and prints the weights each run converges to. Three facts fall out of that exercise:
The first interesting fact is that the same weights can express different gates — only the bias changes between AND and OR. The second is that the loop discovers these numbers: starting from zeros it walks to a separating line in four epochs for both gates. The third is a warning wrapped in an edge case: the boundary line belongs to class 1. Because the step rule is z ≥ 0 → 1, a row that lands exactly on the line fires 1. That is harmless for OR (its boundary passes through (0, 1) and (1, 0), which are both targets of 1) and dangerous for AND, as the worked check below shows.
| inputs | AND | OR | XOR | NAND |
|---|---|---|---|---|
| 0 0 | 0 | 0 | 0 | 1 |
| 0 1 | 0 | 1 | 1 | 1 |
| 1 0 | 0 | 1 | 1 | 1 |
| 1 1 | 1 | 1 | 0 | 0 |
Pick a gate, press train, and read the weights the perceptron settles on — all from the same fifteen lines of update rule. XOR is not in the list because it never settles; that is Chapter 05.
| x1 | x2 | target | z | pred | verdict |
|---|---|---|---|---|---|
| 0 | 0 | 0 | — | — | — |
| 0 | 1 | 0 | — | — | — |
| 1 | 0 | 0 | — | — | — |
| 1 | 1 | 1 | — | — | — |
1 only when both inputs are 1 — the strict gate. Change the learning rate and retrain: the final weights scale with lr, but the boundary they draw is identical — only the sign of z decides.
The source loop’s AND run ends at w = (0.2, 0.1), b = −0.2, so the boundary is 0.2x₁ + 0.1x₂ = 0.2, i.e. 2x₁ + x₂ = 2. Score every row with it:
The third row is the knife edge. In exact arithmetic (1, 0) has z = 0, and step(0) = 1, so the run would not have converged — it would still be making a mistake on a row whose target is 0. The source code passes only because the accumulated floating-point weights are a hair off: 0.2x₁ + 0.1x₂ − 0.2 evaluates to −2.8 × 10⁻¹⁷ for (1, 0), which is below zero. Nothing is wrong with the algorithm; this is what “the boundary belongs to class 1” means when a training run is allowed to stop the moment the weights happen to land on the line.
Two honest ways to handle it in your own implementation: pick the tie convention deliberately (and know that OR’s clean convergence depends on it), or nudge the converged bias by a small margin so the boundary sits strictly between the classes, as the hand-picked weights do. The lab flags any row whose |z| is below 10⁻⁹ so you can see the seam.
XOR outputs 1 when exactly one input is 1 — the exclusive “either, but not both”. Plot its four rows and they form a checkerboard: 1s on one diagonal, 0s on the other. No straight line splits a checkerboard, and that single sentence is the most famous limitation in the history of neural networks.
Chapter 04’s gates are all linearly separable. XOR is not. Here is the source table:
This is not a hard problem to describe — it is one of the simplest functions on two bits — but a perceptron cannot represent it. In 1969 Marvin Minsky and Seymour Papert proved exactly this, and the critique of single-layer learning that followed is conventionally blamed for helping cool neural-network funding for a decade. The math is uncontroversial: a perceptron can only draw one hyperplane, and XOR needs two decisions. The history is messier — funding politics played a role too — but the theorem stands.
Try the most natural candidate, the line x₁ + x₂ = 0.5 (weights (1, 1), bias −0.5):
So the perceptron’s limit is not about epochs or learning rates. The best a single line can do is 3 of 4 — 75% — and the source’s 1000-epoch run confirms it: every epoch has at least two mistakes at visit time, and from epoch 3 onward all four visits are corrected in turn while the weights orbit the same four values. The loop ends exactly where it started, forever.
Put the two orange truth-table rows on one side and the two blue rows on the other with one straight line. Then switch to “train anyway” and watch the perceptron loop forever without settling.
This is not a bug and not a tuning problem: the update rule is doing its job on data whose two classes cannot be cut by one straight line. No learning rate or epoch count fixes a geometry problem.
Which of these is NOT linearly separable in the (x₁, x₂) plane?
The fix for XOR is not a better perceptron — it is more of them. One layer computes two new numbers, and the output neuron classifies those. The result is a decision boundary no single line can draw.
The trick is a piece of Boolean algebra that looks obvious in hindsight: XOR = (x₁ OR x₂) AND NOT(x₁ AND x₂). “At least one of them, and not both.” Each clause is a single perceptron, and the clauses combine with one more perceptron. The source lesson hand-wires exactly that, with three neurons and these weights:
Notice what the hidden layer did. The input space had four corners and two impossible classes. The hidden space also has four points — except two of them collide: (0, 1) and (1, 0), the rows that made XOR non-separable, both become the hidden pair (1, 1). Once they coincide, a single line separates that dot from the others. A hidden layer is a change of coordinates chosen so the next layer’s problem is linearly separable. That sentence is the whole idea of deep learning in one line.
| input | OR neuron (h₁) | NAND neuron (h₂) | AND neuron (out) | XOR target |
|---|---|---|---|---|
| (0, 0) | z = −0.5 → 0 | z = +1.5 → 1 | z = −0.5 → 0 | 0 |
| (0, 1) | z = +0.5 → 1 | z = +0.5 → 1 | z = +0.5 → 1 | 1 |
| (1, 0) | z = +0.5 → 1 | z = +0.5 → 1 | z = +0.5 → 1 | 1 |
| (1, 1) | z = +1.5 → 1 | z = −0.5 → 0 | z = −0.5 → 0 | 0 |
Each hidden neuron is a line, and the two lines carve the unit square into three regions:
The output boundary in hidden space is the line h₁ + h₂ = 1.5, and it is a straight line — the nonlinearity came from stacking, not from bending any single neuron. The same construction scales: layer k’s outputs are layer k+1’s inputs, and each layer can carve the previous layer’s features into finer pieces.
Flip the two inputs and follow the numbers: an OR neuron and a NAND neuron each draw a line, and a final AND neuron combines their answers. Weights are hand-picked from the source lesson — a teaching construction, not something this network learned.
| x₁ | x₂ | h₁ | h₂ | out | XOR |
|---|---|---|---|---|---|
| 0 | 0 | 0 | 1 | 0 | 0 |
| 0 | 1 | 1 | 1 | 1 | 1 |
| 1 | 0 | 1 | 1 | 1 | 1 |
| 1 | 1 | 1 | 0 | 0 | 0 |
Notice (0,1) and (1,0) both become the hidden pair (1,1): the two points that were on opposite sides in the input space collapse onto the same dot. Chapter 06 draws that collapse.
Same four rows, two views. In input space two lines carve a band; in hidden space those same rows land on three dots — and one line splits them.
A hidden layer is a change of coordinates chosen to make the problem easy for the next layer. Real networks search for those coordinates by gradient descent; here they were chosen by hand so the geometry is visible.
Everything you built so far is the real thing: the source lesson’s thirty-line class is a working learning machine. To scale it into modern networks, exactly one design decision changes — and it changes everything: the step becomes a curve.
The source lesson, in full: a weighted sum, a step, and the update rule — here is the class, unchanged, so you can see there is no magic left out.
class Perceptron:
def __init__(self, n_inputs, learning_rate=0.1):
self.weights = [0.0] * n_inputs
self.bias = 0.0
self.lr = learning_rate
def predict(self, inputs):
total = sum(w * x for w, x in zip(self.weights, inputs))
total += self.bias
return 1 if total >= 0 else 0
def train(self, training_data, epochs=100):
for epoch in range(epochs):
errors = 0
for inputs, target in training_data:
prediction = self.predict(inputs)
error = target - prediction
if error != 0:
errors += 1
for i in range(len(self.weights)):
self.weights[i] += self.lr * error * inputs[i]
self.bias += self.lr * error
if errors == 0:
print(f"Converged at epoch {epoch + 1}")
return
print(f"Did not converge after {epochs} epochs")In production the same core loop returns in one import — and the real differences appear at scale:
The reason the step must go is in its slope. A gradient-following optimizer asks “which way is downhill?” — and the step function’s answer is “flat” everywhere except z = 0, where it is a vertical cliff. The sigmoid answers honestly, with a slope you can multiply through layer after layer. That is the door backpropagation walks through, and it is Lesson 03.
Same weighted sum on the x-axis, two different last-mile functions. Move the marker and watch the slope: the step is flat everywhere (and undefined at zero), the sigmoid always has a little grip.
The sigmoid is exactly the function behind logistic regression, Phase 2’s classifier: same weighted sum plus bias, then a smooth squashing instead of a hard call.
from sklearn.linear_model import Perceptron as SkPerceptron
import numpy as np
X = np.array([[0,0],[0,1],[1,0],[1,1]])
y = np.array([0, 0, 0, 1]) # AND
clf = SkPerceptron(max_iter=100, tol=1e-3)
clf.fit(X, y)
print([clf.predict([x])[0] for x in X]) # [0, 0, 0, 1]Which sentence best captures the difference between the perceptron and logistic regression?
Answer before you look. The XOR question and the tie-convention question are the two that separate a memorized definition from a working instinct — the first is the lesson’s punchline, the second is the edge case that hides inside it.
0 / 6 answered · 0 correct
01What mathematical operation does a perceptron perform on its inputs before applying the activation function?
02What does “linearly separable” mean for a classification problem?
03Why does a single perceptron fail to learn the XOR function?
04In the perceptron learning rule, what happens when the prediction matches the target?
05How is XOR solved using multiple perceptrons?
06The step function outputs 1 when z ≥ 0. What does that mean for a point that lies exactly on the decision boundary?
Click a card to swap the lazy description for what it actually means.
Four problems with exact numbers — train NAND, watch the boundary move through AND training, build a 2-of-3 majority vote, and evaluate the hand-wired XOR network row by row. Try first; a worked answer is one click away.
With lr = 0.1 starting from w = (0, 0), b = 0, the loop converges at epoch 6 with w = (−0.2, −0.1), b = 0.2; the epoch mistake counts are 1, 3, 3, 2, 1, 0. Boundary: −0.2x₁ − 0.1x₂ + 0.2 = 0, which multiplies to 2x₁ + x₂ = 2, i.e. x₂ = 2 − 2x₁. Row check: (0,0) → z = 0.2 → 1 ✓; (0,1) → z = 0.1 → 1 ✓; (1,0) → z = 0.0 → 1 ✓ (the tie lands on the correct side here — target 1); (1,1) → z = −0.1 → 0 ✓. The hand-wired alternative from the source's XOR network is just NOT-AND: w = (−1, −1), b = 1.5, boundary x₁ + x₂ = 1.5. Both lines are valid; the trained one is flatter and lower because the loop walked there from zeros.
From zeros with lr = 0.1 the end-of-epoch states are: epoch 1, w = (0.1, 0.1), b = 0 → x₂ = −x₁ (2 mistakes); epoch 2, w = (0.2, 0.1), b = −0.1 → x₂ = 2 − 2x₁ (3 mistakes); epoch 3, same weights, b = −0.2 → x₂ = 2 − 2x₁ (3 mistakes — only the offset moved); epoch 4, the same line makes 0 mistakes and training stops. The weight change rotates the line (from slope −1 to slope −2); the bias change translates it without changing direction. Early mistakes fix the rotation, later ones slide the line. A footnote you can only see by tracking the boundary: in exact arithmetic the final line passes through (1,0), whose target is 0 — the run is saved by floating-point roundoff (z = −2.8 × 10⁻¹⁷). Tracking the line shows the geometry and the tie convention at once.
Yes, it is separable: w = (1, 1, 1), b = −1.5 gives z = (number of ones) − 1.5, so z ≥ 0 exactly when at least two inputs are 1: z = 0.5 for two ones, z = −0.5 for one, z = 1.5 for three. The boundary is the plane x₁ + x₂ + x₃ = 1.5, slicing the cube between the one-1 corners and the two-1 corners. Trained from zeros at lr = 0.1, the loop converges at epoch 4 with w = (0.1, 0.1, 0.1), b = −0.2 — the same direction scaled by 0.1 (multiply by 10: x₁ + x₂ + x₃ = 2, and the two-one rows give z = 0, counted as 1). Mistake counts read 4, 3, 1, 0. Read it as a vote: each input casts +1, and the bias sets how many votes are needed to win.
(0,0) → OR z = −0.5 → h₁ = 0, NAND z = 1.5 → h₂ = 1; output AND z = 0 + 1 − 1.5 = −0.5 → 0 ✓. (0,1) → OR z = 0.5 → 1, NAND z = 0.5 → 1; AND z = 1 + 1 − 1.5 = 0.5 → 1 ✓. (1,0) → the identical hidden pair (1,1) → 0.5 → 1 ✓. (1,1) → OR z = 1.5 → 1, NAND z = −0.5 → 0; AND z = 1 + 0 − 1.5 = −0.5 → 0 ✓. The hidden layer is a change of coordinates: the two rows that made XOR non-separable, (0,1) and (1,0), land on exactly the same hidden point (1,1), so the output neuron only has to separate (1,1) from (0,1) and (1,0) — and one line, h₁ + h₂ = 1.5, does it. The weights are a teaching construction; Lesson 03 trains them by backpropagation.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 03, Lesson 01) and the Math Foundations Notebook reference build. The eight labs (neuron anatomy, boundary dial, learning-loop stepper, gate trainer, XOR failure search, two-layer network, hidden-space view and step-vs-sigmoid), the hand-worked update arithmetic, the AND / OR / NOT / NAND convergence traces, the z = 0 tie note and its floating-point footnote, and the hidden-space mapping of all four XOR rows are original to this page. Every weight, z value, epoch count and accuracy shown is computed live from the source lesson's algorithm (the same zero initialization, step(z) = 1 if z >= 0, and per-mistake update rule), so the numbers on the page match running the source code.