Every layer ends with one fixed curve and no weights of its own. The curve’s shape decides what the layer can express; its slope decides whether learning survives the trip backward. This is how depth became trainable — and why some units go dark forever.
y = W₂(W₁x + b₁) + b₂ collapses to (W₂W₁)x + (W₂b₁ + b₂) — one matrix, one bias. A hundred linear layers have exactly the expressive power of one. The fixed nonlinear curve between layers is what makes depth mean something.
depth without nonlinearity is a lieSigmoid maps to (0, 1) with derivative peaking at 0.25. Tanh maps to (−1, 1) with derivative peaking at 1. ReLU clips negatives and passes positives, derivative exactly 0 or 1. GELU is smooth, dips to ≈ −0.17, and never reads exactly zero. The shape decides forward expressiveness; the derivative decides backward learning.
max f′ — sigmoid 0.25 · tanh 1 · ReLU 0/1 · GELU ≈ 1.1Output layers report an answer, so the gate must match its shape: one probability (sigmoid), a distribution summing to 1 (softmax), or the unbounded real line (no gate). And the gate's derivative decides failure modes — sigmoid's 0.25 vanishes with depth, ReLU's hard zero kills units, softmax never belongs in a hidden layer.
report probabilities · learn from slopesAn activation is a fixed door between learned rooms: it has no weights of its own, but its shape decides what can pass forward and its slope decides how much learning signal can come back — and the whole modern toolkit of activations is a negotiation between those two jobs.
By the end you will be able to compute every gate’s value and derivative by hand; prove that a stack of linear layers collapses to one matrix; explain saturation, vanishing gradients and dead ReLUs with numbers; implement stable softmax and say why it lives only at the output; and choose a defensible default for any layer in a network — then justify it with a measurement, not a preference.
A linear layer is a matrix multiply plus a shift. Compose two of them and you get one matrix multiply plus one shift. Without a nonlinear gate in between, depth has no representational power to add — it just changes the numbers in the one matrix you already had.
Here is the entire argument in three lines. A layer computes h = W₁x + b₁. A second layer computes y = W₂h + b₂. Substitute the first into the second:
That is one linear layer wearing two layers’ clothing. The same substitution works for five layers, fifty, a hundred — the product of all the matrices is still a matrix, so every deep linear stack is exactly one affine map. The parameters were never what gave depth its power; the missing ingredient is a nonlinearity between the layers.
Insert a gate g after the first layer and the algebra breaks: W₂·g(W₁x + b₁) + b₂ cannot be rewritten as a single matrix, because g bends the space before the second matrix touches it. Each gate adds bends; stacked gates compose those bends into ever-richer piecewise shapes. That is why a network with ReLUs can draw an XOR boundary or wrap itself around a spiral, and a network of pure matrix multiplies cannot — no matter how deep you make it.
Take W₁ = [[1, 2], [0, 1]], b₁ = [0.5, −0.5], W₂ = [[2, 0], [1, 3]], b₂ = [1, 1], and feed the input x = (1, 1).
The collapse lab below runs the same test on a random stack: drag the depth from 2 to 8 and the readout’s difference stays at zero because the layer-by-layer result and A·x + c are the same arithmetic rearranged. Now add a ReLU after every layer and the green curve bends — those bends are the capacity.
Grey is the whole stack with no activation — mathematically identical to a single layer y = A·x + c. Green is the same weights with a ReLU after every layer. Raise the depth and watch the bends multiply; the straight line can never follow.
The check never fails: a product of matrices is a matrix, so the difference is exactly zero in real arithmetic. Depth buys representational power only when a nonlinear gate sits between the layers.
A colleague reports that their 40-layer network with no activations still trains fine. They conclude depth helps. What actually happened?
Every neural network block is h = g(Wx + b), where g is one fixed scalar function applied to every number. Three gates from three eras — and the derivative below each one is what backpropagation actually multiplies.
The gate has no weights of its own — it is a fixed shape, applied elementwise after the layer’s weighted sum z = Wx + b. The network learns where to feed each gate; the shape itself is chosen by you. That shape matters twice: forward, it decides what kinds of functions the layer can represent; backward, its derivative decides how much learning signal survives the trip through it.
Sigmoid squashes any real number into (0, 1), which is why it looks like a probability. Its derivative is beautifully simple, and fatally capped:
No sigmoid output is ever negative, which sounds harmless and is not: when every input to a weight is positive, every weight in that layer receives an update of the same sign, so the parameter vector moves diagonally and corrections zig-zag instead of going straight. Zero-centering fixes that — which is exactly what tanh is for.
Tanh is sigmoid re-centered at the origin: same S-shape, outputs in (−1, 1), inputs of both signs possible. Its derivative is also simple and reaches a full 1.0 at x = 0.
ReLU — the rectified linear unit — does nothing at all for positive inputs and nothing but zero for negative ones. That triviality is the breakthrough: on the positive side the derivative is exactly 1, so the gradient passes through a layer untouched.
ReLU’s hard zero is both its superpower and its flaw: gradient 1.0 on the positive side is why deep networks became trainable after 2010, and gradient exactly 0 on the negative side is why a unit can die permanently — the subject of chapter 5.
| Gate | Forward | Range | Derivative | Zero-centered | Saturates |
|---|---|---|---|---|---|
| Sigmoid | 1 / (1 + e⁻ˣ) | (0, 1) | σ·(1 − σ) — max 0.25 at 0 | no | both tails |
| Tanh | (eˣ − e⁻ˣ) / (eˣ + e⁻ˣ) | (−1, 1) | 1 − t² — max 1.0 at 0 | yes | both tails |
| ReLU | max(0, x) | [0, ∞) | 1 if x > 0 else 0 | no | the whole negative half |
Pick an activation, then drag x. The solid line is the function; the orange dashed line is its derivative — the factor backpropagation multiplies through. The red strip below marks the flat tails where that factor collapses toward zero.
The historical squasher. All-positive outputs and a derivative capped at 0.25 make deep stacks untrainable.
Take one neuron whose pre-activation z — the weighted sum before the gate — is −2, then 0, then 2. Here is the full forward number and the backward factor each gate contributes:
Read the middle column of each block as “what the next layer sees” and the right column as “what backprop gets to multiply.” Sigmoid never exceeds 0.25; tanh can reach 1.0 but pays for it with fast saturation on both sides; ReLU is a perfect 1.0 punctured by a perfect 0.0.
ReLU’s two flaws are its hard zero and its hard corner. Leaky ReLU files the corner, GELU smooths it away, and Swish was found by a search algorithm rather than a human — all three keep a gradient alive where ReLU turns everything off.
Leaky ReLU is the smallest possible repair to the dead-neuron problem: instead of clamping the negative side to zero, give it a whisper of a slope. With α = 0.01, a negative input is scaled to one percent of itself instead of being erased.
A whisper is enough for a neuron to keep moving and eventually climb out, but it is still not centered, still not smooth, and it adds a hyperparameter. Modern practice mostly skips it: GELU solves the same problem with a better gradient and no new knob.
GELU — the Gaussian Error Linear Unit — weights each input by how likely it is to be positive under a standard normal distribution. Φ(x) is that probability (the normal CDF), and the gate is a simple product:
Two numbers carry the whole idea. GELU(−2) is negative — small, −0.045 — so a unit can sit below zero and still matter, unlike ReLU’s hard clip. And GELU′(−2) is −0.085: a small but real gradient pointing back, where ReLU would pass exactly zero. The gate is smooth everywhere, so a stuck unit always has a slope to follow home. That is why BERT, GPT and most transformers use GELU in their feed-forward blocks.
Swish (also called SiLU) is GELU’s near-twin, discovered in 2017 by a search algorithm combing through activation space — a network designing part of a network. It gates x with the sigmoid instead of the Gaussian CDF:
In practice Swish and GELU perform almost identically; Swish shows up in EfficientNet-style vision models, GELU dominates in language models. The lesson is not which one wins — it is that both share the property that matters: smooth, slightly negative, never exactly zero.
Sweep the input range and count how often each derivative drops below the threshold. The curves at the top show why the bars below differ: ReLU zeroes an entire half-line, while GELU and Swish keep a small but nonzero slope almost everywhere.
ReLU’s 50% is the entire negative half of this artificial symmetric range — not a claim that half of your neurons are dead. On real data pre-activations lean positive, which is exactly why the scan is a teaching picture, not a diagnosis.
Scan 241 evenly spaced inputs from −5 to 5 and count how often|f′| drops below 0.01 — the source’s exact experiment, made interactive above:
Two surprises worth keeping. First, tanh has a larger flat zone than sigmoid even though its peak derivative is four times bigger: it races to ±1 in each direction and then sits there. Second, Leaky ReLU’s 0.01 slope lands exactly on the boundary — with a 0.01 threshold it counts as alive, but raise the bar and its whole negative half turns red. A small slope is a rescue rope, not a guarantee.
An activation’s forward shape decides what a layer can express; its derivative decides whether the layer learns at all. Wherever the curve flattens, the local derivative approaches zero — and backprop multiplies by that number at every layer.
Look back at the curves in chapter 02 and read the slopes instead of the outputs. Sigmoid and tanh are flat at both ends: push the input to −8 and the output barely moves, so the derivative is a few ten-thousandths. That region is called saturation, and a unit sitting there is not broken — it still computes a clean forward value. It simply has no slope to learn with. ReLU has the opposite shape: perfect slope 1 on the entire positive side, and a hard zero on the entire negative side. Neither failure is visible in the forward pass; both are written in the derivative.
Depth turns a small per-layer factor into an exponential one. Ten sigmoid layers multiply the backward signal by at most 0.25¹⁰ ≈ 9.5 × 10⁻⁷; the first layer receives roughly a millionth of the output’s gradient while the last layers train normally. The network does not error — it quietly becomes a shallow network with a large attached constant.
One number moves forward through seeded layers, then the gradient walks back multiplying by f′(z) at every stop. Pick a gate and watch what ten — or thirty — multiplications do. Bold is the selected activation; the faint lines are the rest of the family under the same weights.
This is a one-number teaching model, not a full network: it shows the mechanism, not a training run. Re-roll the weights a few times — ReLU’s gradient is either exactly 1 or exactly 0 depending on whether each unit stayed active, and one dead layer zeroes everything behind it.
The lab’s one-number teaching model sends a signal forward through 30 seeded layers and walks the gradient back, multiplying by the local derivative at every stop. Same random weights for every gate; only the activation changes.
Two lessons hide in the table. First, the ordering — sigmoid worst, then GELU, then tanh — follows the typical derivative each gate actually spends time at, not its advertised maximum. Second, ReLU is not sigmoid’s opposite: no exponential shrink, but three dead layers each multiply everything behind them by exactly 0. A decay can be countered by depth, initialization or normalization; a zero cannot. That cliff is the next chapter.
A 30-layer tanh network trains, but the first five layers' weights barely change. The per-layer factors sampled from the data average 0.61. What is the gradient reaching layer 1, roughly?
A unit whose pre-activation is negative on every sample gets output 0, derivative 0, and therefore an update of exactly 0 — on every example, at every step. It is not slow and it is not small. It is switched off, and gradient descent cannot switch it back on.
ReLU’s negative half has derivative exactly 0, not merely small. That makes the failure categorical. A sigmoid unit in its tail still receives a trickle of gradient — 10⁻⁴, 10⁻⁶ — and over enough steps the trickle can move it. A dead ReLU receives zero, and zero times any learning rate is still zero. The update rule runs, the parameter does not move, and the unit stays dead for every remaining epoch.
How does a unit get there? Its weighted sum must be negative on every training sample. That happens when a large negative bias lands on the wrong side of the data — a big learning-rate step, an initialization that starts the unit deep in the negative region, or simply bad luck with a sparse feature. Once there, the unit’s job is taken over by its neighbours, which makes recovery even less likely: nothing in the loss ever asks the dead unit to return.
Twenty neurons read 5 Gaussian inputs, 1000 samples each. Push the biases negative — what a bad initialization step or a too-large learning rate does to some units — and count how many stop learning. The weights never change; only the gate does.
“Active” means the sample gave this neuron a gradient of at least 0.05. A bar at 0% with status DEAD is a unit that will never move again under this gate.
A dead ReLU neuron is not slow or small — it is stuck: output zero, gradient zero, so every update is zero times the learning rate. Leaky ReLU keeps a 0.01 slope on the negative side and GELU keeps a real slope everywhere, so the same unlucky units stay trainable.
One hidden unit sits at z = −2 with upstream gradient ∂L/∂h = 0.05 and learning rate 0.1. Compare the weight update each gate permits at that point.
The Leaky ReLU update looks negligible — 5 × 10⁻⁵ times the input — and for one step it is. But it is nonzero, so z drifts, the unit’s outputs stop being flat zero, its derivative grows, and the next update is larger. The road back is exponential, and it starts with the one thing ReLU cannot give: a gradient that is not exactly zero. GELU gets there faster still, because at z = −2 its slope is already eight times steeper than Leaky ReLU’s.
During training you notice 30% of a ReLU layer's units output 0 for every input in the batch. Which statement is correct?
A classifier’s last layer does not owe you a probability — it owes you a distribution over mutually exclusive classes. Softmax manufactures one: exponentiate every score, then divide by the total. The exponent makes big scores disproportionately big; the division makes the shares sum to 1.
Softmax takes a vector of raw scores — logits, the unbounded outputs of the last linear layer — and turns it into a probability distribution over the classes:
Two design choices carry the meaning. Exponentiation is monotone, so the ranking of the scores survives — the argmax of the probabilities is the argmax of the logits. And because the shares are computed from ratios, only differences between logits matter: shifting every score by +10 produces the same distribution. That last property is not a curiosity; it is the trick that makes softmax numerically stable, and the difference between a working classifier and a row of NaNs.
Four class scores (logits) go in; four probabilities that sum to 1 come out. Temperature divides the logits before exponentiating: low temperature sharpens toward the argmax, high temperature flattens toward uniform.
All four bars share one total. The largest logit does not own the largest bar — it owns the largest share, and the shares always sum to 1.0000.
Softmax is an output gate, not a hidden one: it collapses the vector’s magnitude into a fixed total, which is exactly what a classifier needs to report and exactly what an intermediate representation must not do.
The lab’s default logits are z = [2, 1, 0.1, −1]. Exponentiate, total, divide:
The class with logit 2 was never five times better than the class with logit 0.1 — the logits are not measurements, just scores — and softmax does not pretend otherwise. What it does is commit: at T = 1 the model reports 64% confidence for the winner; turn the dial to T = 0.25 and the same scores become 98% confidence. Temperature trades decisiveness for caution without touching the model’s parameters, which is exactly why language models expose it as a sampling knob.
A model outputs logits [0.1, 0.2] for two mutually exclusive classes. What does softmax report, and what would a sigmoid report for the first class?
Activation choice is a small decision that beginners treat as a large one. The field converged on a short list of defaults for the obvious combinations — start there, and only deviate when a learning curve gives you a reason.
The table below is the whole decision, and none of it is arbitrary. Hidden layers want a gate whose derivative stays usable across the operating range — GELU in transformers, ReLU-family in vision and tabular models, tanh where the state is multiplied repeatedly. Output layers want the gate that matches the shape of the answer: a probability for two classes (sigmoid), a distribution over many (softmax), or no gate at all when the answer is a real number.
| where the gate sits | default | why |
|---|---|---|
| Hidden · transformer / NLP | GELU | Smooth, never exactly zero; the default in BERT, GPT and friends. |
| Hidden · CNN / vision | ReLU (Swish with headroom) | Sparse and fast; decades of tuning from AlexNet onward. |
| Hidden · RNN / LSTM | tanh | Bounds the repeatedly-multiplied hidden state; never explodes. |
| Hidden · simple MLP / tabular | ReLU | The cheapest strong default; escalate only if curves stall. |
| Output · binary classification | Sigmoid | One score → one probability; pair with binary cross-entropy. |
| Output · multi-class classification | Softmax | A distribution over mutually exclusive classes; sums to 1. |
| Output · regression | None (linear) | Predictions must span the reals; any squash would cap them. |
The source lesson’s decision flow, made clickable. Two questions — where the gate sits and what you are building — collapse to a default you can defend. Start with the default; change it only when a measurement asks you to.
QUESTION 1 / WHERE DOES THE GATE SIT?
Hidden layers need a nonlinear gate between every affine map. Output layers need the gate that matches the loss — or none at all.
Activation choice is a default-setting decision, not a research project. The measurement that justifies changing it is a learning curve — loss going flat while units sit dark, or gradients shrinking by orders of magnitude.
You are predicting tomorrow's temperature (a real number, roughly −30 °C to +45 °C). What does the output layer look like, and why?
Answer before you look. The collapse question and the softmax question are the two that separate a memorised activation list from an understanding of what each gate does to a signal — forward and backward.
0 / 6 answered · 0 correct
01You stack five linear layers with no activation functions. What do you actually have?
02What is the output range of the ReLU activation function?
03What is the “dead neuron” problem in a ReLU network?
04Which activation function is the hidden-layer default in modern transformers like BERT and GPT?
05Why does softmax belong in the output layer only, never in a hidden layer?
06Sigmoid's derivative peaks at 0.25. What does a 10-layer sigmoid stack pass back from the output to the input?
Click a card to swap the lazy description for what it actually means.
Four problems with exact numbers — learn PReLU’s slope, walk the vanishing experiment to 50 layers, rescue an all-negative unit with ELU, and compute what a sigmoid output layer is actually doing wrong. Try first; a worked answer is one click away.
Forward: pReLU(z) = z if z > 0 else αᵢ·z, one α per unit. Backward adds one line per unit: ∂L/∂αᵢ = Σ_samples (∂L/∂hᵢ) · zᵢ · 1[zᵢ < 0] — the negative pre-activation, weighted by the upstream gradient. Numeric check: a unit that is dead under ReLU has z = −2 on some sample and upstream gradient ∂L/∂h = 0.1. With learning rate 0.1 the PReLU update is Δα = −lr · (0.1 × (−2)) = +0.02; three such steps lift α from 0.25 to 0.31, the negative-side slope grows, the unit's output stops being flat zero, and it starts contributing to the loss. The ReLU unit at the same z cannot make that move: its own derivative is 0, so every update is 0. Practical cautions: constrain α ≥ 0 (libraries store log α or clamp), and use a separate learning rate if α moves too fast. On the circle dataset the accuracy difference versus fixed Leaky ReLU is small — the point of the exercise is the gradient path and the recovery, not the score.
With a constant factor f per layer the gradient is fᴸ. Sigmoid: 0.25¹⁰ = 9.54 × 10⁻⁷ crosses the bar at layer 10; by layer 30 it is 8.67 × 10⁻¹⁹. Tanh: 0.5¹⁰ = 9.77 × 10⁻⁴, 0.5²⁰ = 9.54 × 10⁻⁷ — crossed near layer 20. ReLU: 1ᴸ = 1 wherever units stay active; the first inactive layer multiplies everything behind it by exactly 0, so the curve is a cliff, not a decay. GELU: 0.8ᴸ falls below 10⁻⁶ only past layer 60 (0.8⁵⁰ = 1.43 × 10⁻⁵). The lesson's sampled chain (seed 7, 30 layers) lands close to that ordering: sigmoid 3.59 × 10⁻²¹, tanh 3.77 × 10⁻⁷, GELU 3.25 × 10⁻¹⁰. Honesty note: real factors are data-dependent and vary layer to layer, so read the plot as an order-of-magnitude story, not a prediction — and remember ReLU's failure is a hard zero, which no amount of depth can soften.
ELU's derivative is 1 for positive inputs and α·eˣ for negative ones — never exactly zero, approaching zero only as x → −∞. Numeric check: a unit stuck at z = −3 has ReLU output 0 and gradient 0; with α = 1 the ELU unit outputs e⁻³ − 1 = −0.9502 and its gradient is e⁻³ = 0.0498. With upstream gradient δ = 0.05 and learning rate 0.1, the ReLU weight update is 0.1 × 0.05 × 0 = 0, while the ELU update is 0.1 × 0.05 × 0.0498 = 2.5 × 10⁻⁴ times the input — small, but nonzero, so z drifts, eᶻ grows, and the unit can climb out of the negative region. On the lesson's 20-neuron ward at bias shift −4.5, ReLU buries 5 units while ELU shares Leaky ReLU's property that the permanently-dead count is 0. The price is an exp() per unit and a derivative that is not monotone, which can make the mean activation harder to interpret.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 03, Lesson 04) and the Math Foundations Notebook reference build. The seven labs (activation plotter, linear-collapse proof, gradient dead-zone scan, 30-layer vanishing chain, dead-neuron ward, softmax with temperature and stability toggle, activation picker), the per-gate value/derivative table at x = −2/0/2, the seeded 30-layer gradient products, the ELU/PReLU recovery arithmetic, the softmax overflow demonstration and the defaults table are original to this page. Every number shown is computed live by the labs or verified by hand in the prose.