Weights are the only thing a network gets to choose before it has learned anything. Set them to zero and 512 neurons behave as one; set their scale wrong and the signal is 10²⁷ by layer ten — or a thousandth of itself. This lesson is the five-minute decision that makes the next 50 layers trainable.
Every neuron does the same job: multiply inputs by weights, add a bias, apply an activation. Start them all at the same value — zero, or any constant — and each one computes the same output, receives the same gradient, and updates by the same amount. A 512-unit layer stays 512 copies of one neuron. Randomness is the brute-force fix; the size of the randomness is the science.
same weights → same gradient → same updateWith fan_in inputs, Var(z) = fan_in · Var(w) · Var(x). Set Var(w) = 1 with fan_in = 512 and the factor is 512 per layer — 512¹⁰ ≈ 1.2×10²⁷ by layer ten. Set Var(w) = 0.001 and the factor is 0.512 — down to 1.2×10⁻³ after ten layers. The target is the product fan_in · Var(w) = 1, adjusted for what the activation does to the signal.
fan_in · Var(w) · (activation) = 1Xavier's Var(w) = 2/(fan_in + fan_out) balances the forward and backward passes when the activation is roughly linear near zero. ReLU zeroes half its outputs, so it needs the extra factor of 2: Var(w) = 2/fan_in. And in transformers, every residual addition pumps variance into the stream — GPT-2 scales branch weights by 1/√(2N) to keep a 126-layer stack bounded.
tanh → Xavier · ReLU → Kaiming · residuals → 1/√(2N)Initialization is choosing the network’s starting point: random enough that no two neurons are copies, scaled so that one layer’s variance multiplier — fan-in × weight variance × the activation’s factor — is as close to 1 as the architecture allows.
By the end you will be able to implement zero, random, Xavier, Kaiming and orthogonal initialization; explain why zero init wastes a layer and why random scale alone is not enough at depth; derive Var(w) = 2/(fan_in + fan_out) and Var(w) = 2/fan_in from the variance-propagation rule; read activation statistics to diagnose vanishing, exploding and dead networks; and place GPT-2’s 1/√(2N) residual scaling in the same framework — all with numbers you can check by hand.
Initialization is the only decision a network makes before it has learned anything. The most tempting starting point — all weights zero — turns out to be the worst one: it makes every neuron in a layer an exact copy of every other, and gradient descent has no way to tell them apart.
Every neuron in a layer does the same arithmetic: multiply its inputs by its weights, add a bias, apply an activation. If every neuron starts with the same weights, they all compute the same output. The backward pass then hands every one of them the same gradient, and the update step moves every one of them by the same amount. The layer never diversifies: after 10,000 epochs it is still 512 copies of a single neuron. You paid for 512 parameters and got one.
This is called symmetry, and random initialization is the brute-force way to break it. Each neuron starts at a different point in weight space, so each one sees a slightly different gradient and learns a slightly different feature. The source’s main.py makes the point with four neurons and two inputs: with zero weights every neuron outputs sigmoid(0) = 0.5, and after ten thousand steps the four outputs are still identical to the last decimal.
The failure is not that the output is zero — it is that the output is the same for every unit. A layer of identical neurons has no more expressive power than a layer of width one, no matter how many units you buy.
The same inputs feed four independent sigmoid neurons, each with its own weights. Give them identical starting weights and press step: they receive identical gradients and stay identical forever. Randomize and watch them separate.
Constant weights fail the same way zero does — it is theshared value, not the value zero, that creates the tie. Random is the brute-force fix; the next chapters decide howlarge that randomness should be.
Take two neurons with the same inputs, one started at zero and one started at random weights. Compare what one gradient step does to thedifference between them.
The numbers are ordinary arithmetic — no luck involved. Randomness gives the units different starting points, and different points see different gradients, which keeps them apart. The rest of this lesson is about choosing the scale of that randomness.
Your colleague initializes a hidden layer with every weight equal to 0.01 and says 'at least it's not zero, so the neurons will differ.' What actually happens?
Random weights break symmetry. Wrong-sized random weights break everything else. One line of probability — the variance of a weighted sum — explains why a scale that works at depth 3 detonates or dies at depth 50, and it produces every formula in the rest of the lesson.
Look at a single neuron before any activation: z = w₁x₁ + w₂x₂ + … + w_n x_n. If the inputs and weights are independent, zero-mean random variables, the variance of the sum is the sum of the variances of the terms, and each term contributes Var(w)·Var(x):
That single line is the whole engine. The layer’s variance multiplier is fan_in · Var(w), and the danger is that it compounds: after L layers the signal is multiplied by (fan_in · Var(w))^L. A factor of 1.05 sounds harmless until you raise it to the 50th power. The design goal is exact: pick Var(w) so the product equals 1 — ideally on both the way forward and the way back.
Take a 512-input layer and input variance 1, exactly the source’s opening scenario. Watch what one layer does, then ten.
Note that the source prints 0.00013 for the ten-layer shrinkage; careful arithmetic gives 1.24×10⁻³ (0.512 multiplied by itself ten times). Neither number changes the story: with variance 0.001 the signal shrinks by roughly an order of magnitude every four layers, and depth finishes the job. The lesson’s lab recomputes the real thing with Monte-Carlo samples.
Push random data through 50 same-width layers, measuring the mean squared activation each layer. Pick an initialization and an activation: the healthy curves stay inside the band for the whole trip; the broken ones leave the chart in either direction.
Simulation, not theorem: one seed, 32 units wide, 96 samples. Xavier and Kaiming preserve the expected variance; any single deep run wanders, which is the honest reason practitioners still measure activation statistics instead of trusting the formula blind.
Two refinements turn fan_in · Var(w) = 1 into the formulas practitioners use. First, the activation is not free: sigmoid and tanh are roughly linear near zero — where well-initialized activations live — so they pass variance through unchanged, but ReLU zeroes every negative value and multiplies the variance by about ½. Second, the backward pass has its own fan-in: a neuron’s gradient depends on its fan_out, so protecting the gradient suggests a different constant than protecting the forward signal. Xavier and Kaiming are the two ways of resolving those refinements.
A 512-input layer is initialized with weights drawn from N(0, 1) — standard normal. What happens to the activation variance after ten such layers?
Protecting the signal on its way in wants Var(w) = 1/fan_in. Protecting the gradient on its way back wants Var(w) = 1/fan_out. Glorot and Bengio’s answer was to take the harmonic mean of the two — the formula that made sigmoid networks trainable at depth.
Chapter 02 asked for fan_in · Var(w) = 1, which reads as Var(w) = 1/fan_in. That is the forward ideal: it keeps the activation variance constant as data flows toward the loss. But backpropagation has its own variance budget: a neuron’s gradient is a weighted sum over its fan_out outgoing connections, so keeping the gradient variance stable asks for Var(w) = 1/fan_out. When a layer is square (fan_in = fan_out) the two ideals agree; when it is rectangular they conflict.
Xavier/Glorot initialization (2010) resolves the conflict by choosing the harmonic mean of the two ideals: 2/(fan_in + fan_out). The harmonic mean leans toward the smaller number, which is the larger of the two fans — exactly the edge of the layer where the signal is most at risk. In practice the weights are drawn either normally with that variance, or uniformly on ±√(6/(fan_in + fan_out)), which is constructed to have the same variance.
Xavier assumes the activation is roughly linear near zero — true for sigmoid and tanh in their responsive range, which is where properly initialized activations live. That assumption is why it is the right answer for sigmoid/tanh networks and one factor of two short for ReLU, the subject of the next chapter.
That split is the honest reading of the formula: 2/(fan_in + fan_out) is not magic, it is a compromise. For square layers — the common case inside a transformer block or an MLP trunk — the compromise is exact and the factor is 1 in both directions.
Give a layer shape and an activation; read the standard deviation each scheme prescribes. The forward and backward columns are the variance factors fan_in × Var(w) and fan_out × Var(w) — a scheme that balances both is the whole point of Xavier’s harmonic mean.
| scheme | std | value | Var(w) | fwd × | bwd × | 10 layers |
|---|---|---|---|---|---|---|
| N(0, 1) — the naive default | 1 | 1.0000 | 1.000 | 400.0 | 400.0 | 1.05×10^26 |
| N(0, 0.01) | 0.01 | 0.0100 | 1.00×10^-4 | 0.04000 | 0.04000 | 1.05×10^-14 |
| Xavier normal | √(2/(fan_in + fan_out)) | 0.0500 | 0.002500 | 1.000 | 1.000 | 1.000 |
| Kaiming normal ✓ | √(2/fan_in) | 0.0707 | 0.005000 | 2.000 | 2.000 | 1024 |
| LeCun normal | √(1/fan_in) | 0.0500 | 0.002500 | 1.000 | 1.000 | 1.000 |
| Xavier uniform L = √(6/(fan_in+fan_out)) | ±L | 0.0866 | 0.002500 | 1.000 | 1.000 | 1.000 |
Simplified teaching model: it tracks variance through the weighted sum and post-activation scaling, ignoring correlations and finite-width effects. The empirical check on the right measures the real thing on 400 inputs × 20,000 samples.
Try the 400 → 100 preset with each family. Xavier keeps the arithmetic balance between 1.6 forward and 0.4 backward; Kaiming and LeCun each bet on one direction. There is no universally right answer — only the right pairing with your activation.
import math
import random
def zero_init(fan_in, fan_out):
return [[0.0 for _ in range(fan_in)] for _ in range(fan_out)]
def random_init(fan_in, fan_out, scale=1.0):
return [[random.gauss(0, scale) for _ in range(fan_in)] for _ in range(fan_out)]
def xavier_init(fan_in, fan_out):
std = math.sqrt(2.0 / (fan_in + fan_out))
return [[random.gauss(0, std) for _ in range(fan_in)] for _ in range(fan_out)]
def kaiming_init(fan_in, fan_out):
std = math.sqrt(2.0 / fan_in)
return [[random.gauss(0, std) for _ in range(fan_in)] for _ in range(fan_out)]Notice what did not change: every function returns a list of lists with the same shape. Initialization is not a new architecture or a new optimizer — it is three characters in a formula. xavier_init(400, 100) draws from N(0, 0.06325²), and those weights already know their layer’s fan-in and fan-out. That is the entire trick: let the starting scale depend on the layer’s shape instead of hoping one constant works everywhere.
Xavier assumes the activation passes variance through. ReLU does not: it turns every negative value into zero, which cuts the layer’s signal in half. He et al. (2015) added the factor that compensates — and with it, the initialization that made deep ReLU networks finally trainable.
Here is the ReLU problem in one line: ReLU(z) = max(0, z) deletes every negative value. With symmetric, zero-mean pre-activations, that deletes roughly half of them — and variance is an average of squares, so deleting half the values halves the variance. The layer’s multiplier is no longer fan_in · Var(w) but fan_in · Var(w) / 2. Set that equal to 1 and solve:
The factor of 2 is not a fudge — it is the measured halving, undone. The same logic covers the whole ReLU family: Leaky ReLU with negative slope a keeps a fraction a of the negative side, so the correction becomes Var(w) = 2 / ((1 + a²) · fan_in); GELU and Swish are smooth enough that the same 2/fan_in works in practice, which is why PyTorch and HuggingFace models reach for Kaiming for every modern transformer-style block.
Take fan_in = 400 and compare Xavier and Kaiming before and after the ReLU, using the single-layer variance experiment (the lab’s numbers, 20,000 Monte-Carlo samples):
This is a rare case in deep learning where theory and measurement agree to within a couple of percent, because the two approximations — symmetric pre-activations and independence across units — are essentially exact at initialization.
One column per layer, one row per magnitude decade: color shows where the activations live. A healthy init keeps the mass near 10⁰ in every column; a bad one drifts upward, collapses to the floor, or freezes at a single value.
This is the diagnostic a practitioner runs first: forward a batch of real data and look at each layer’s activation statistics. The source’s rule of thumb — mean magnitude between 0.5 and 2.0 through all layers — is exactly this heat-map staying on the middle rows.
A different idea: orthogonal initialization. Everything so far has tuned the variance of an iid Gaussian matrix so that its average row length is 1. An orthogonal matrix does that exactly, by construction: its columns are perpendicular unit vectors, so QᵀQ = I, every singular value is 1, and multiplying by Q rotates the input without stretching it in any direction. The check is short enough to do by hand:
Orthogonal init keeps the signal norm exactly intact at any depth — it was standard in recurrent networks and very deep non-residual nets — but it costs an SVD or QR decomposition per layer and constrains the whole spectrum uniformly, which is why the big models in practice mostly reach for Kaiming and normalization layers. It is a good reminder that the goal is norm preservation; variance tuning is just the cheap way to approximate it.
A residual connection is a gradient highway: it lets the signal skip a block unchanged. But it also lets variance pile up, block after block, and 126 of them will bury an otherwise perfect initialization. GPT-2’s 1/√(2N) is the scaling term that keeps the pile bounded.
A transformer block is built around a residual stream: the input x flows through, and each sub-layer adds its output back to it. x = x + sublayer(x). For the gradient, this is wonderful — the + passes it through unchanged, and Chapter 03’s vanishing-gradient problem disappears.
For the variance, it is a leak. Each addition injects the sub-layer’s output variance into the stream, and the stream never gets a chance to shrink it back. If each branch contributes one unit of variance, the stream after N transformer blocks — two additions each, attention and MLP — carries 1 + 2N units. At GPT-2-small’s 12 blocks that is already 25×; at Llama 3’s 126 blocks and 405 billion parameters it is 253×, and the activations would be far from the range the rest of the initialization was designed for.
GPT-2’s fix is one constant: multiply the output weights of each residual branch by 1/√(2N). Squaring the scale makes each branch contribute 1/(2N) units instead of one, so the worst-case growth over the whole stack is 2N × 1/(2N) = 1 extra unit — a stream that doubles at most, no matter how deep the model. (The original paper states 1/√N per sub-block; with two sub-blocks per layer the two bookkeepings carry the same constant.)
The simplified teaching model from the lab: the stream starts at variance 1, every residual addition contributes a fixed branch variance of 1 unit, and a transformer block contains two additions.
Simplified teaching model, stated plainly: real transformers have LayerNorm inside each branch, branch gains that vary with attention entropy, and depth-dependent scaling schemes beyond GPT-2’s (LayerScale, DeepNorm, μP). What survives every version is the accounting: residual additions accumulate variance, so something must scale with depth to cancel it.
Every residual addition pumps a little variance into the stream. Left alone it accumulates linearly with depth; scaled by 1/√(2N) it stops mattering how deep you go. Slide the depth from GPT-2-small to Llama-3-large.
The model is a deliberately simple bookkeeping device: every branch contributes a fixed unit of variance and each block has two branches. Real transformers mix in LayerNorm, attention entropy and branch gains — but the qualitative fact survives: unscaled residual variance grows with depth, and 1/√(2N) is the term that cancels it.
Initialization bugs do not throw exceptions. They leave statistics behind: activation magnitudes that climb, collapse, freeze, or look perfectly healthy while the gradients quietly die. Learning to read those numbers is cheaper than learning to guess.
The source’s health check is one sentence: verify that activation magnitudes stay between 0.5 and 2.0 through all layers. That band is not a law of nature — it is the range where most activations still behave usefully. One forward pass over a batch, recording the mean absolute activation per layer, tells you almost everything:
The last row is the one that fools people. A saturating sigmoid outputs values near 0 or 1: not tiny, not huge — and its derivative a(1−a) collapses toward zero anyway. If you only watch magnitudes you will see a healthy network; if you watch the gradient at layer one you will see 10⁻⁹. That difference — magnitude versus derivative — is the diagnostic skill this chapter is after. It is also why the same diagnostic run on a trained network can show a bimodal histogram (many units at 0, many at 1) — the failure mode is units parked on the flat tails.
Six forwards of a 32-wide stack, six fingerprints. Pick an observed symptom and read the diagnosis — the same chart a practitioner squints at when a network refuses to learn.
A 512-wide sigmoid layer: every unit outputs exactly 0.5000 on the first batch, and every unit's gradient is identical.
Diagnosis. Zero (or constant) weight initialization. The units are the same function, so backprop hands them the same gradient and the symmetry never breaks — the layer is effectively 1 neuron wide with 512 copies.
Prescription. Initialize weights randomly. Biases can stay at zero; it is the weight matrix that must differ across units. Confirm by checking that outputs differ across units on the first batch.
The zero-init, small-init, too-large-init and Xavier-plus-ReLU sparklines are computed live from the same simulation as the other labs. The dead-ReLU curve is illustrative: a permanently dark unit is a per-unit statistic, not a layer average.
Order the diagnoses by cost of being wrong: identical units waste the entire layer; vanishing units waste the depth; exploding units waste the run.
Biases deserve their own footnote, because they are initialized separately and follow different conventions. The default is zero everywhere; biases are not where the capacity lives, and a zero bias keeps a ReLU unit’s fate decided by its inputs rather than a constant offset. Two well-known exceptions: the classic LSTM forget-gate bias is often set to 1 so the memory starts wide open (a constant bias cannot break symmetry between units, but it can set a useful operating point), and transformer normalization layers initialize their scale at 1 and shift at 0. The general rule stands: randomness in the weights, constants in the biases.
Finally, note what initialization cannot fix. If a network is broken because of a bug, bad data, or a learning rate that is off by 100×, better starting weights will not rescue it. But if training diesimmediately — flat loss, NaNs in the first epochs, gradients that are zero or enormous before any learning happens — the starting scale is the first suspect, and it is the cheapest one to test.
A 30-layer sigmoid network is stuck: mean |a| ≈ 1.0 at every layer — perfectly healthy-looking — but the first layer's gradients are ~10⁻⁹ and the loss will not move. What is the most likely cause?
The source closes with a real training run: 200 points, a 2-8-1 network, six initialization/activation pairs, 300 epochs each. Small networks forgive more than deep ones — but the loss curves still tell you which starting points let the signal do its job.
The experiment is deliberately modest: a circle dataset (label 1 inside a radius, 0 outside), a network with 8 hidden units, one output sigmoid, squared error, per-sample gradient descent at learning rate 0.1, 300 epochs. Nothing about the data, the architecture or the optimizer changes between rows — only the initial weights and the activation. The port in this lesson follows the source’s training_comparison() line for line, with a deterministic JavaScript RNG, so the numbers in the lab are the numbers below.
Read the first row carefully, because it is the chapter’s whole point in miniature: the tiny-init sigmoid run improves 5.5% and stops at 0.2108 — barely under the 0.2158 of always predicting the class prior. It is not fitting; it is outputting a constant. The same tiny weights paired with ReLU recover (94.6%), because ReLU’s derivative is 1 where it is active, so the signal can regrow — exactly the opposite of the sigmoid row, whose derivative is at most 0.25. And notice that the two large-init runs also learn here: this network is 2 layers deep and 8 units wide, where a bad scale bends the curve rather than breaking it. Fifty layers, the setting of chapter 02’s experiment, is where they break.
This gap between “shallow networks forgive” and “deep networks do not” is why initialization gets a reputation for being folklore. People try three layers, see everything work, and conclude the choice does not matter. Depth is the variable that turns a 5% annoyance into a dead run — and modern models are nothing but depth.
The source’s training comparison, computed live: a 2-8-1 network on 200 circle-shaped points, six initialization/activation pairs, 300 epochs each. Same data, same learning rate — only the starting weights change.
Honest caveats: this network is tiny and the problem is easy, so several bad initializations still find their way — small-init ReLU recovers because ReLU’s derivative is 1, not because tiny weights are safe. The 50-layer experiment is where they stop recovering. Re-roll the seeds to see which conclusions are robust.
In production you rarely write these formulas by hand. Every framework ships them — and, importantly, already applies a fan-in-aware default when you use a standard layer. PyTorch’s nn.Linear(512, 256) initializes its weight uniformly on ±1/√fan_in, a Kaiming-uniform relative whose bound is tied to the layer’s shape. That default is why most simple networks “just work.” You override it when you build custom architectures, when you go deeper than roughly 20 layers, or when you use an activation the default was not designed around — and you always own the depth terms, like residual scaling, that the framework cannot know about.
import torch
import torch.nn as nn
layer = nn.Linear(512, 256)
nn.init.xavier_uniform_(layer.weight) # sigmoid / tanh
nn.init.xavier_normal_(layer.weight)
nn.init.kaiming_uniform_(layer.weight, nonlinearity='relu') # ReLU / GELU
nn.init.kaiming_normal_(layer.weight, nonlinearity='relu')
nn.init.zeros_(layer.bias) # biases stay at zeroThe practical recipe. For each layer: (1) draw weights randomly so no two units are copies — never zeros or constants; (2) choose the scale by activation — Xavier if the gate is roughly linear near zero (sigmoid, tanh), Kaiming if it zeroes half its range (ReLU, GELU, Swish), LeCun if the design assumes unit variance (SELU); (3) set biases to zero unless a specific component (an LSTM forget gate, a normalization scale) needs otherwise; (4) if the architecture has residual connections, add the depth term 1/√(2N); and (5) verify with one forward pass that the mean activation magnitude stays between 0.5 and 2.0 across all layers. Five rules, five minutes, and the next 50 layers have a chance.
Answer before you look. The Kaiming formula and the Xavier-pairing questions are the two that separate memorized acronyms from a working understanding of why the factors are there.
0 / 5 answered · 0 correct
01What happens if you initialize every weight in a neural network to zero?
02Why does the scale of random weight initialization matter?
03What is the formula for Kaiming/He initialization variance?
04When should you use Xavier/Glorot initialization instead of Kaiming/He?
05Why does GPT-2 scale residual layer weights by 1/√(2N)?
Click a card to swap the lazy description for what it actually means.
Four problems with exact numbers — add LeCun init and compare it to Xavier, implement GPT-2’s residual scaling, watch the safe window narrow as fan-in grows, and verify orthogonal initialization by hand. Try first; a worked answer is one click away.
LeCun's standard deviation is 1/√fan_in. For the source's square 64→64 layers, Xavier gives √(2/(64+64)) = √(2/128) = 0.125 and LeCun gives 1/√64 = 0.125 — identical, because for fan_in = fan_out the harmonic mean 2/(n+n) equals 1/n. So the 50-layer run at width 64 shows no difference in the linear regime (both hold E[a²] near 1; measurement noise and tanh saturation account for the drift). The schemes separate only when the layer is rectangular: for 64→256, Xavier std = √(2/320) = 0.0791 while LeCun std = 0.125. The forward factors diverge accordingly: LeCun keeps 64 × 0.015625 = 1, Xavier gives 64 × 0.00625 = 0.4 per layer — 0.4¹⁰ ≈ 1.0×10⁻⁴ after ten layers. LeCun keeps the forward signal exactly; Xavier deliberately gives some of that up to protect the backward pass, which is the right trade when the gradient is the scarce resource.
Model each transformer block as two residual additions (attention and MLP), each adding one unit of branch variance to a stream that starts at 1. Unscaled, 2N additions give Var = 1 + 2N: ×25 for N = 12, ×253 for N = 126 (Llama 3's depth). Scaling the branch weights by 1/√(2N) multiplies each branch's variance by 1/(2N), so the stream grows to 1 + 2N/(2N) = 2 — one extra unit, no matter how deep. Concretely: scale = 1/√24 = 0.2041 for GPT-2-small and 1/√252 = 0.0630 for the 126-layer version. The check is that the scaled final value is independent of N while the unscaled one grows linearly with N.
Take a fixed weight standard deviation of 0.1, so Var(w) = 0.01. The per-layer factor is fan_in × Var(w): 16 × 0.01 = 0.16 for the small layer (0.16¹⁰ ≈ 1.1×10⁻⁸ — vanished by layer ten) and 1024 × 0.01 = 10.24 for the large one (10.24¹⁰ ≈ 1.3×10¹⁰ — exploded). Same weights, opposite failures, purely because fan-in changed by 64×. The scheme that adapts, std ≈ 1/√fan_in, gives 0.25 for fan_in = 16 and 0.03125 for fan_in = 1024. In practice you do not need the exact point: a factor between about 0.63 and 1.58 keeps a 10-layer stack within a 100× band, which is a std window of [0.198, 0.315] at fan_in 16 but only [0.025, 0.039] at fan_in 1024 — an 8× narrower target. That is why large layers need the formula, not a guess.
An orthogonal matrix has orthonormal columns, so QᵀQ = I and every singular value is 1 — the matrix rotates and reflects but never stretches. Hand check with θ = 45°: Q = [[cos θ, −sin θ, 0], [sin θ, cos θ, 0], [0, 0, 1]]. Columns have norm 1 (cos²+sin² = 1) and are mutually perpendicular. Applying Q to x = [1, 1, 0] gives z = [cos θ − sin θ, sin θ + cos θ, 0] = [0, 1.4142, 0], and |z|² = 2 = |x|². For variance: Var(z) = Var(x) exactly, with no 1/fan_in factor — the iid Gaussian matrix needed Var(w) = 1/fan_in only to make its row norm √(fan_in × Var(w)) = 1. For ReLU, divide the orthogonal weights by √2 to pay for the halving, just as Kaiming does. At 50 layers orthogonal init keeps the signal at unit norm by construction; the catch is cost (an SVD or QR decomposition per layer) and that it constrains all singular values equally rather than tuning the spectrum, which is why practice usually reaches for Kaiming.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 03, Lesson 08) and the Math Foundations Notebook reference build. The seven labs (four-neuron symmetry stepper, 50-layer variance experiment, fan-in/fan-out calculator with a Monte-Carlo variance check, activation-magnitude heat map, GPT-2 residual-scaling curve, init clinic, and the six-configuration training sweep) are original to this page, as are the worked one-step symmetry ledger, the corrected 0.512¹⁰ arithmetic, the two Xavier worked examples, the uniform-variance check L²/3, the single-layer Kaiming measurements, the 12/126-block residual table, the LSTM forget-gate note, and the orthogonal-init hand check. Every number shown is computed live by the labs or verified by hand in the prose.