A classifier’s raw scores become a distribution through softmax(zᵢ) = e^zᵢ / Σ e^zⱼ, cross-entropy charges the model for low confidence in the right answer, and sample averages fall into a bell curve.
A classifier's output is a whole distribution over classes: every outcome gets a probability, and the values add to 1. A continuous variable gets a density instead, where probability is the area under the curve over an interval — not the height at a point.
Σ P(X = k) = 1 · ∫ f(x) dx = 102 / CENTRE AND SPREAD
E[X] is the balance point; Var is the wobble.
The expected value is the probability-weighted average — where the distribution would balance on a fulcrum. The variance is the expected squared distance from it. A fair die balances at 3.5 (a value it cannot roll) with variance 35/12 ≈ 2.917.
Var(X) = E[X²] − (E[X])²03 / SOFTMAX TO DISTRIBUTION, CROSS-ENTROPY TO GAP
Scores in, probabilities out, one loss.
Softmax exponentiates raw logits and divides by their sum: positive numbers that add to 1 and keep the ordering. Cross-entropy then charges the model by how little probability it gave the correct class, −log q(true).
loss = −log q(true class)
MENTAL MODEL IN ONE SENTENCE
Every prediction a model makes is a probability distribution; every loss measures the distance between the predicted distribution and the true one; every training step nudges one toward the other. Softmax turns scores into a distribution, and cross-entropy measures the gap.
By the end you will be able to read a PMF and a PDF without confusing them, compute an expectation and a variance by hand, explain why averages become normal, implement a numerically stable softmax, and turn logits into a cross-entropy loss without a library.
01
SAMPLE SPACES AND EVENTS
Everything chance can do, listed in one set.
The sample space is the set of every outcome an experiment can produce. An event is a subset of it, and probability is a number between 0 and 1 attached to that subset.
A coin flip has sample space S = {H, T}; a die roll has S = {1, 2, 3, 4, 5, 6}. The event “the die shows an even number” is the subset {2, 4, 6}, and its probability is 3/6 = 0.5. Once you can name the outcomes and the event, most probability questions become counting questions.
Three axioms carry the whole subject: probabilities are never negative, the entire sample space has probability 1, and the probabilities of mutually exclusive events add. Bayes’ theorem, expectations, and every distribution in this lesson follow from those three rules.
A probability is a long-run frequency. The individual outcome is still uncertain — that is exactly the uncertainty a model expresses with its distribution.
Derivation: why a probability is a long-run frequency
Roll a fair die n times and count how often face 1 appears. Each roll contributes 1 with probability p = 1/6 and 0 otherwise, so the count is a sum of n independent Bernoulli(p) variables — and the empirical fraction = count/n has mean p (it is unbiased) and variance p(1 − p)/n. As n grows, the spread shrinks and the fraction concentrates on p.
p = 1/6, 1 − p = 5/6
sd of the empirical fraction = √( p(1−p)/n ) = √( (5/36)/n )
n = 100 → sd = √0.001389 = 0.0373 (±2 sd ≈ ±7.5%)
n = 1,000 → sd = √0.000139 = 0.0118 (±2 sd ≈ ±2.4%)
n = 1,000,000 → sd = 0.000373 (±2 sd ≈ ±0.07%)
The bell around the target is itself the Central Limit Theorem in disguise.
This is why simulated frequencies in the lab above creep toward the theoretical values. “Probability 1/6” does not promise anything about the next roll; it describes the long-run share.
Quick check
Two fair dice are rolled. Which is more likely: the sum is 2, or the sum is 7?
02
RANDOM VARIABLES, PMF vs PDF
Read the height or measure the area.
A random variable turns outcomes into numbers. Discrete numbers get exact probabilities from a PMF; continuous numbers get a density, and only areas under it are probabilities.
A random variable is a function from the sample space to the real numbers: a coin becomes 0 or 1, a die becomes 1 through 6. When the possible values are countable, the distribution is a probability mass function (PMF): each value k has a probability P(X = k), every probability is at least 0, and they sum to 1. When the variable can take any value in an interval, the distribution is a probability density function (PDF): the curve’s height at a point is a density, and probability is the area under the curve between two values.
The distinction is not academic. A classifier’s output over classes is a PMF: 0.7 for cat is a probability you can read off. A VAE’s latent space is described by a PDF: a single point has probability zero, and only a region has nonzero probability.
Discrete — read it off:
Fair die: P(X = k) = 1/6 ≈ 0.1667 for k = 1..6
P(X = 2) + P(X = 4) + P(X = 6) = 3/6 = 0.5
Σ P(X = k) = 1
Continuous — integrate it:
Uniform(0, ½): f(x) = 1/(0.5 − 0) = 2 for 0 ≤ x ≤ 0.5
f(0.2) = 2 ← a density (and 2 > 1 is fine)
P(0.1 ≤ X ≤ 0.3) = 2 × 0.2 = 0.4 ← an area
∫₀^0.5 f(x) dx = 2 × 0.5 = 1 ← the whole area is 1
Normal(0,1): peak f(0) = 1/√(2π) ≈ 0.3989
P(−1 ≤ X ≤ 1) ≈ 0.6827 = the shaded area
A height on the left, an area on the right
Both panels describe one fair die and one uniform variable. On the left, read probability straight off a bar. On the right, move the interval — only the shaded area is a probability.
PMF P(X = 3) = 1/6 ≈ 0.1667 ← the height IS the probability
P(X = 3) + P(X = 4) = 0.3333
Σ over all six faces = 6 × 1/6 = 1.0000
PDF f(x) = 1 / (0.5 − 0) = 2 for 0 ≤ x ≤ 0.5
density at a point: f(0.2) = 2 ← a density, not a probability
P(0.10 ≤ X ≤ 0.30) = 2 × 0.20 = 0.4000
that number is the shaded AREA, and areas can be at most 1
The density can be 2, 10, or 100 — only its integral must be 1. This is the single most common PMF/PDF mix-up.
Derivation: why a single point has probability zero (and the density can be 3)
For a continuous variable the probability of landing in an interval is the area under the density. Shrink the interval to a single point x: its width goes to zero, so the area goes to zero. Formally, P(X = x) = ∫xx f(t) dt = 0. That is why “the probability the bus arrives at exactly 8:00:00” is zero, and why continuous models always talk about ranges.
Uniform(0, ½), f(x) = 2:
P(0.19 ≤ X ≤ 0.21) = 2 × 0.02 = 0.04
P(0.199 ≤ X ≤ 0.201) = 2 × 0.002 = 0.004
P(X = 0.2) = 2 × 0 = 0
Yet f(0.2) = 2. The height is a density — it tells you how
quickly probability accumulates, not how much sits at the point.
Only ∫ f(x) dx must be 1; f itself can be any non-negative number.
For a discrete variable the story is opposite: each value carries a real chunk of probability, so P(X = 2) = 1/6 for a fair die is perfectly meaningful.
Quick check
A Normal(0, σ = 0.1) density at x = 0 is ≈ 3.99. What does that tell you?
03
A FIELD GUIDE TO DISTRIBUTIONS
Five shapes cover most of ML.
Bernoulli for yes/no, binomial for counts, uniform for ignorance, normal for noise, exponential for waiting. Each one is a formula for a shape, a mean, and a variance.
Every distribution answers the same two questions: what does the probability of each outcome look like, and where is the mass centred with how much spread? The Bernoulli is the coin inside every binary classifier. The categorical is the same idea for many classes — it is exactly the output shape of softmax. The uniform says “all values equally likely”, which is the right starting point when you genuinely know nothing. The normal is the bell that shows up whenever many small independent effects add. The exponential is the memoryless waiting time: the chance of the next event in the next second does not depend on how long you have already waited.
Distribution
Formula
Mean / Variance
Models
Bernoulli(p)
P(1) = p, P(0) = 1 − p
p / p(1 − p)
Binary classification: one yes/no
Binomial(n, p)
P(k) = C(n,k)·pᵏ·(1−p)ⁿ⁻ᵏ
np / np(1 − p)
Successes in n independent trials
Categorical(p₁…pₖ)
P(i) = pᵢ, Σ pᵢ = 1
—
Multi-class output, next-token choice
Uniform(a, b)
f(x) = 1/(b − a)
(a+b)/2 / (b−a)²/12
Random initialization, an honest “no idea”
Normal(μ, σ²)
f(x) = e^(−(x−μ)²/2σ²) / √(2πσ²)
μ / σ²
Noise, weight init, everything via the CLT
Exponential(λ)
f(x) = λe^(−λx), x ≥ 0
1/λ / 1/λ²
Waiting time between rare events
Poisson(λ)
P(k) = λᵏe^(−λ)/k!
λ / λ
Counts of rare events per interval
Simplified teaching table: real systems compose these primitives — a Gaussian mixture for pixels, a categorical over 50,000 tokens — but the same shapes, means, and variances are underneath.
One explorer, five distributions
Pick a distribution, move its parameters, and shade a region. The readout shows the exact probability and the mean and variance formulas this chapter derives.
Bernoulli(p = 0.30)
P(X ≤ 1) = 1.0000
P(X = 1) = 0.3000 · E = p = 0.3000 · Var = p(1−p) = 0.2100
Σ P(X = k) over all 2 outcomes = 1.0000
Discrete outcomes get exact probabilities; continuous variables get densities, and only their areas are probabilities.
Reading the normal formula piece by piece, with numbers
f(x) = 1/√(2πσ²) · exp( −(x − μ)² / (2σ²) )
(x − μ) is the distance from the centre. Squaring it makes left and right symmetric and punishes distance.
Dividing by 2σ² measures that distance in units of σ. A wide σ makes a wide bump; a small σ makes a spike. For σ = 1 and x = 2, the exponent is −4/2 = −2.
exp(−…) turns distance into small density, smoothly: at the centre exp(0) = 1, at 1σ exp(−0.5) ≈ 0.6065, at 2σ exp(−2) ≈ 0.1353, at 3σ exp(−4.5) ≈ 0.0111.
1/√(2πσ²) is the number that makes the total area exactly 1.
Standard normal (μ = 0, σ = 1), peak density = 1/√(2π) ≈ 0.3989:
f(0) = 0.3989
f(1) = 0.3989 × 0.6065 = 0.2420
f(2) = 0.3989 × 0.1353 = 0.0540
f(3) = 0.3989 × 0.0111 = 0.0044
Areas (from the standard normal table):
P(−1 ≤ X ≤ 1) ≈ 0.6827 68% inside one σ
P(−2 ≤ X ≤ 2) ≈ 0.9545 95% inside two σ
P(−3 ≤ X ≤ 3) ≈ 0.9973 99.7% inside three σ
Second worked example: Binomial(10, 0.3), exactly 3 successes
The binomial counts successes in n independent trials, each with success probability p. The n choose k factor counts which trials succeed, and pᵏ(1−p)ⁿ⁻ᵏ gives the probability of any one such pattern.
The check that the probabilities add up: summing P(X = k) over k = 0 to 10 gives exactly 1. The explorer above shows the same curve for any n and p.
04
EXPECTATION AND VARIANCE
Find the balance point, then measure the wobble.
The expected value is the probability-weighted average — where the distribution balances. The variance is the expected squared distance from it: how wide the wobble is.
Expected value weights every outcome by its probability: E[X] = Σ x·P(X = x) for a discrete variable, or ∫ x·f(x) dx for a continuous one. It is the long-run average of repeated samples, and it need not be an outcome the variable can actually take — a fair die balances at 3.5, which no face shows.
Variance measures spread around that centre: Var(X) = E[(X − μ)²], the average squared distance. Its square root, the standard deviation σ, is in the same units as X, which is why people quote σ rather than σ². In ML, the loss you minimize is an expected value over the data distribution, estimated on a mini-batch; high variance in those estimates is what “noisy training” means.
Each bar is a weight of size P(X = x). The fulcrum sits where the distribution balances — the expected value — and the dashed band shows one standard deviation each way.
Fair die
E[X] = Σ x·P(X = x) = 1·0.17 + 2·0.17 + 3·0.17 + 4·0.17 + 5·0.17 + 6·0.17
= 3.5000
E[X²] = Σ x²·P(X = x) = 15.1667
Var = E[X²] − μ² = 15.1667 − 12.2500 = 2.9167
SD = √2.9167 = 1.7078
The fulcrum sits at 3.5 — an outcome the die cannot roll. The expected value is a long-run average, not a prediction.
Averaging n independent copies divides the variance by n: Var(mean) = σ²/n. That fact is the engine of both mini-batch training and the Central Limit Theorem.
Let μ = E[X]. Expand the square: (X − μ)² = X² − 2μX + μ². Expectation is linear — the average of a sum is the sum of the averages, and constants pull out — so
The shortcut matters because E[X²] is usually easier to compute than a sum of squared deviations. For a coin it also shows the maximum: p(1−p) is largest at p = 0.5, so a fair coin is the most unpredictable Bernoulli.
Why averaging n samples divides the variance by n (the engine of the CLT)
Variances of independent quantities add, and scaling a variable by c multiplies its variance by c². The average of n independent copies is (X₁ + … + Xₙ)/n, so:
Var(mean) = (1/n²)·(σ² + σ² + … + σ²) = (1/n²)·nσ² = σ²/n
Fair die, σ² = 35/12 ≈ 2.9167:
n = 1 → Var = 2.9167 SD = 1.7078
n = 2 → Var = 1.4583 SD = 1.2076
n = 30 → Var = 0.0972 SD = 0.3118
Averaging 100 samples cuts the standard deviation by √100 = 10.
This is exactly why mini-batches work: the gradient on a batch of 32 is noisy, but its variance is 1/32 of the single-example variance, so the step direction is a much steadier estimate of the true gradient. It is also half of the Central Limit Theorem: the centre stays at μ while the spread shrinks like 1/√n.
Quick check
You roll a fair die twice and average the two results. What are the mean and variance of that average?
05
INDEPENDENCE AND CONDITIONAL PROBABILITY
Knowing something changes the odds.
P(A | B) is the probability of A once B is known to have happened. It shrinks the world to the outcomes in B and asks what fraction of them are also in A.
Conditional probability is defined by P(A | B) = P(A and B) / P(B). From a deck of cards, P(King | face card) = P(King and face) / P(face) = (4/52)/(12/52) = 4/12 = 1/3. Two events are independent when knowing one tells you nothing about the other: P(A | B) = P(A), which is equivalent to the product rule P(A and B) = P(A)·P(B). Coin flips are independent; cards drawn without replacement are not, because the deck remembers what left.
Two variables together live in a joint distribution P(X, Y). Summing one variable out gives its marginal: P(X = x) = Σ_y P(X = x, Y = y). The courier team’s table below records rain against late deliveries. The marginals are the row and column totals; the conditionals are ratios inside a row or column.
Y = on time
Y = late
Marginal P(X)
X = sun
0.40
0.10
0.50
X = rain
0.05
0.45
0.50
Marginal P(Y)
0.45
0.55
1.00
The same 1.00 of probability split two ways. Reading down a column gives a conditional distribution; reading across a row or down a margin gives a marginal.
Derivation: the conditional formula comes from counting, and the independence test from it
Suppose the sample space S is a set of equally likely outcomes, so P(A) = |A|/|S|. Being told “B happened” throws away every outcome outside B — the world shrinks to B. Inside that smaller world, A happens exactly on the outcomes in both A and B:
P(A | B) = |A ∩ B| / |B| (count inside B)
Divide top and bottom by |S|:
P(A | B) = (|A ∩ B|/|S|) / (|B|/|S|)
= P(A and B) / P(B) ✓
Deck of cards:
|King ∩ face| = 4, |face| = 12, |S| = 52
P(King | face) = 4/12 = 1/3 ≈ 0.3333
check via the formula: (4/52) / (12/52) = 0.0769 / 0.2308 = 0.3333 ✓
Courier table:
P(late | rain) = P(rain and late) / P(rain) = 0.45 / 0.50 = 0.90
independence would need P(rain and late) = 0.50 × 0.55 = 0.275
but it is 0.45 → rain and lateness are NOT independent.
06
WHY THE BELL CURVE IS EVERYWHERE
Average enough noise and a bell appears.
The Central Limit Theorem: the mean of many independent random variables is approximately normal, no matter what distribution they came from. The centre is μ and the spread is σ/√n.
One fair die is perfectly flat. Average two dice and the histogram becomes a triangle — sums near 7 have more ways to happen than sums near 2 or 12. Average thirty dice and the shape is a nearly perfect bell. The remarkable part is that the source distribution does not matter: uniform, lopsided, or long-tailed, the averages converge to the same family of curves.
This is why measurement errors are approximately normal (many small independent causes add up), why weight initializers draw from normal or uniform distributions, and why the noise in stochastic gradient descent looks Gaussian: a mini-batch gradient is a sum of per-example gradients, and sums of many independent things become bells.
Roll 1 die: flat μ = 3.5, σ = 1.7078
Average of 2 dice: triangular μ = 3.5, σ = 1.2076
Average of 30 dice: a bell μ = 3.5, σ = 0.3118
σ² of the average = σ²/n = 2.9167/30 = 0.0972
σ of the average = 1.7078/√30 = 0.3118
Same centre; the spread shrinks like 1/√n.
Averages become a bell, whatever the source
Start with n = 1 to see the raw source shape. Raise the number of samples per average and watch the histogram of averages turn into the dashed normal curve.
source: Fair die (flat)
true μ = 3.5000 true σ² = 2.9167
true variance of the average σ²/n = 2.9167/1 = 2.9167
true standard deviation σ/√n = 1.7078
empirical mean = 3.4794
empirical variance = 2.8828
n = 1: the histogram is the source distribution itself — flat, lumpy, or skewed.
Averaging is how noise is tamed: ten times as many samples cuts the standard deviation of the average by √10 ≈ 3.16.
Derivation: the normal that the averages approach
Each sample Xᵢ has mean μ and variance σ². The average has the same mean, and its variance is σ²/n (derived in the previous chapter). The Central Limit Theorem adds the shape: as n grows, the distribution of the average approaches
mean of the average: μ
variance of the average: σ²/n
standard deviation: σ/√n
so the dashed curve in the lab above is Normal(μ, σ²/n)
Numeric check with the fair die (σ² = 35/12 ≈ 2.9167): for n = 30 the predicted variance is 0.0972 and the predicted standard deviation is 0.3118. The histogram of 5,000 simulated averages lands on those numbers and the dashed curve traces its outline.
Two honest caveats. First, “n ≥ 30” is a rule of thumb, not a law: very skewed or heavy-tailed sources need more samples before the bell looks convincing. Second, the theorem is about the shape of the average, not about the source data itself — individual samples keep their original distribution.
07
SOFTMAX AND CROSS-ENTROPY
Raw scores become a distribution, then a loss.
A network’s last layer emits unnormalized logits. Softmax turns them into a probability distribution; cross-entropy measures how little probability the model gave the correct answer.
Logits are raw real-valued scores, one per class. Softmax exponentiates each and divides by the sum: every output lands in (0, 1), all of them sum to 1, and the ordering is preserved. The exponential amplifies gaps: a logit lead of 2 becomes a probability ratio of e² ≈ 7.39.
Cross-entropy between a true distribution p and a predicted distribution q is −Σ pₖ·log qₖ. For a labelled example the true distribution is one-hot — a 1 on the correct class and 0 elsewhere — so every term except the true class vanishes and the loss collapses to −log q(true). Predict 0.9 for the right class and the loss is 0.105; predict 0.01 and it is 4.605. The loss is unbounded as confidence in the right answer goes to zero, which is exactly the pressure you want on a wrong, confident model.
softmax(zᵢ) = exp(zᵢ) / Σⱼ exp(zⱼ)
Properties:
all outputs in (0, 1) Σ outputs = 1
order is preserved equal logits → uniform probabilities
exp amplifies differences logit gap 2 → probability ratio e² ≈ 7.39
Log probabilities: products become sums
P(50-word sentence) = 0.01⁵⁰ = 10⁻¹⁰⁰ (still representable in float64)
log P = 50 · ln 0.01 = −230.2585 (a finite number)
At 500 words the raw product 10⁻¹⁰⁰⁰ underflows to exactly 0.0,
while the log-probability −2302.585 stays perfectly finite.
Every language model therefore scores sentences in log space.
Logits → probabilities → loss
Move the raw scores, pick the correct class, and change the temperature. The model always uses the stable max-shifted softmax; the loss punishes low confidence in the true class.
stable softmax, step by step:
scaled z/T = [2.00, 1.00, 0.10, -1.00]
max m = 2.00
shifted = [0.00, -1.00, -1.90, -3.00] ← largest is 0, exp cannot overflow
Σ exp = 1.5672
softmax q = [0.6381, 0.2347, 0.0954, 0.0318]
sum(q) = 1.0000 ✓
log-softmax[fish] = -1.00/T − m − ln(Σ) = -3.4493
q(fish) = 0.0318
cross-entropy = −log q(fish) = 3.4493
for comparison:
q(true) = 0.90 → loss 0.1054
q(true) = 0.01 → loss 4.6052
uniform q = 0.25 → loss ln 4 = 1.3863 (no information)
worse than guessing uniformly: the gradient is large and the weights will move a lot.
Temperature divides the logits before softmax. Below 1 sharpens the top choice; above 1 flattens toward uniform — the same knob language models expose when sampling.
Derivation: softmax is unchanged by subtracting the max, and log-sum-exp keeps it finite
Subtract any constant c from every logit: e^(zᵢ − c) / Σⱼ e^(zⱼ − c).
Use e^(a−b) = e^a·e^(−b). The numerator is e^zᵢ·e^(−c) and the denominator is e^(−c)·Σⱼe^zⱼ.
The e^(−c) factors cancel, leaving the original expression. Same probabilities — but choosing c = max(z) makes the largest exponent 0 and all others negative, so nothing can overflow.
z = [1000, 1001]: the naive exp(1001) overflows to infinity
m = 1001: shifted = [−1, 0] Σ exp = 1.367879
log Σ e^z = 1001 + ln(1.367879) = 1001 + 0.313262 = 1001.313262
log-softmax = [1000 − 1001.313, 1001 − 1001.313] = [−1.313, −0.313]
probabilities = [e^(−1.313), e^(−0.313)] = [0.2689, 0.7311] ✓ finite
Log-softmax from the same shift, for any z:
log qᵢ = zᵢ − m − ln( Σⱼ e^(zⱼ − m) )
Derivation: cross-entropy collapses to −log q(true), worked twice
General cross-entropy between a true distribution p and a predicted distribution q over K classes is H(p, q) = −Σₖ pₖ log qₖ. With a one-hot label, pₖ = 1 for the true class t and 0 elsewhere, so every other term multiplies by zero:
H(p, q) = −1·log qₜ − 0 − 0 − … = −log q(true class) ✓
Since 0 < qₜ ≤ 1, log qₜ ≤ 0 and the loss is ≥ 0, hitting 0 only
when the model puts probability 1 on the right answer.
Example A — logits [2.0, 0.5, −1.0, 3.0, 0.1], true class 3:
m = 3.0; shifted = [−1.0, −2.5, −4.0, 0.0, −2.9]
Σ exp(shifted) = 0.367879 + 0.082085 + 0.018316 + 1 + 0.055023
= 1.523303
log-softmax[3] = 0 − ln(1.523303) = −0.420881
loss = 0.42088 q(class 3) = 1/1.523303 ≈ 0.65647
check: −ln(0.65647) = 0.42088 ✓ (PyTorch agrees to 4 decimals)
Example B — logits [1, 2, 3], true class 0:
m = 3; shifted = [−2, −1, 0]
Σ exp = 0.135335 + 0.367879 + 1 = 1.503214
log-softmax[0] = −2 − ln(1.503214) = −2 − 0.407606 = −2.407606
loss = 2.4076 q(class 0) = 0.0900
check: −ln(0.0900) = 2.4076 ✓
In code, use log-softmax directly rather than log(softmax(x)): the second form exponentiates and then takes a log, losing precision and risking a log of zero. PyTorch’s nn.CrossEntropyLoss fuses the two for exactly this reason.
Quick check
Why does the stable softmax subtract the maximum logit before exponentiating?
08
CHECK YOURSELF
Five questions. Then the terms worth keeping.
Answer before you look. The PMF/PDF and cross-entropy questions are the ones that show up in real debugging sessions and code reviews.
0 / 5 answered · 0 correct
01What is the difference between a PMF and a PDF?
02What does the Central Limit Theorem state?
03Why does softmax subtract the maximum logit before exponentiating?
04Cross-entropy simplifies to −log(q(true class)). What does this mean intuitively?
05Why do language models work with log probabilities instead of raw probabilities?
Key terms, demystified
Click a card to swap the lazy description for what it actually means.
Exercises from the lesson
Four problems. Try first; a worked answer is one click away.
Implement inverse transform sampling for the exponential distribution: x = −ln(1 − u)/λ. Sample 10,000 values with λ = 2 and compare the histogram to the true density f(x) = 2e^(−2x).Show one worked answer
The exponential CDF is F(x) = 1 − e^(−λx). Set u = F(x) and solve: 1 − u = e^(−λx), so x = −ln(1 − u)/λ. (Using u instead of 1 − u works too — both are uniform.) With λ = 2 the true mean is 1/λ = 0.5 and the variance is 1/λ² = 0.25, so the standard deviation is 0.5. A numeric check at u = 0.5 gives x = ln 2 / 2 ≈ 0.3466, exactly the theoretical median. Across 10,000 draws the empirical mean should land within about 2·0.5/√10000 = 0.01 of 0.5; the histogram should fall off like 2e^(−2x).
Build a joint table for two loaded dice. Compute the marginal distributions and test independence.Show one worked answer
Let die A be loaded with P(A = 6) = 1/3 and P(A = 1…5) = 2/15 each; die B is fair. Collapse each die to the event 'shows 6' and suppose the observed joint is P(A=6, B=6) = 0.10. The marginals are P(A = 6) = 0.10 + 0.2333 = 1/3 and P(B = 6) = 0.10 + 0.0667 = 1/6. Independence would force P(A=6, B=6) = (1/3)(1/6) = 1/18 ≈ 0.0556, but the table says 0.10 — so the dice are not independent, even though both marginals are single-die probabilities. The full 2×2 table is [0.10, 7/30; 1/15, 3/5], and every row and column sums to its marginal: 0.10 + 0.2333 = 0.3333, 0.10 + 0.0667 = 0.1667, total 1.
Compute the cross-entropy loss for a 5-class classifier with logits [2.0, 0.5, −1.0, 3.0, 0.1] when the correct class is index 3. Then verify with PyTorch's nn.CrossEntropyLoss.Show one worked answer
Shift by the max logit 3.0: [−1.0, −2.5, −4.0, 0.0, −2.9]. The exponentials sum to 0.367879 + 0.082085 + 0.018316 + 1 + 0.055023 = 1.523303. log-softmax of class 3 is 0 − ln(1.523303) = −0.420881, so the loss is 0.42088. The class-3 probability is 1/1.523303 ≈ 0.65647, and −ln(0.65647) = 0.42088 ✓. nn.CrossEntropyLoss(logits_tensor, torch.tensor(3)) returns tensor(0.4209): PyTorch applies log-softmax internally and never forms log(softmax(x)).
A 50-word sentence has each word at probability 0.01. Compute the total log-probability and the raw probability. Then extend to 500 words and explain what changes.Show one worked answer
log P = 50 · ln(0.01) = 50 · (−4.60517) = −230.2585, and the raw probability is 0.01⁵⁰ = 10⁻¹⁰⁰. That value is still representable in float64 (the smallest normal is ≈ 2.2 × 10⁻³⁰⁸), so the product survives at 50 words. At 500 words the raw probability is 10⁻¹⁰⁰⁰, which underflows to exactly 0.0, while the log-probability is 500 · (−4.60517) = −2302.585: still a perfectly finite number. That is why every language model works in log space — products of small probabilities become sums of logarithms, and sums do not underflow.
Terms this lesson borrows from later lessons (or outside)
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
token — A word or word-piece that a language model reads or writes one at a time. The softmax distribution is over the next token.
language model — A model trained to predict the next token given the previous ones. Sampling from its softmax is how it writes.
mini-batch — A small random subset of the training data used for one gradient step. The loss on it estimates the expected loss over the whole dataset.
prior / posterior — Bayesian inference: start with a belief (the prior), see evidence, and compute an updated belief (the posterior). (Lesson 7)
dropout — Randomly zeroing some neurons during training so the network cannot rely on any single one. It is a Bernoulli sample per neuron per step.
diffusion model — A generative model that turns pure random noise into an image one small denoising step at a time; each step samples from a learned distribution.
latent space — The compressed internal vector representation a model learns. In a VAE it is described by a probability distribution rather than a single point.
one-hot — A vector of zeros with a single 1 marking the correct class, e.g. [0, 0, 1, 0]. It is the true distribution cross-entropy compares against.
calibrated — A model is calibrated when its stated probabilities match reality: events it calls 70% likely happen about 70% of the time.
noise — Random fluctuation in the gradient because each mini-batch is a different random sample of the data. High-variance gradients are what make training unstable.
KEEP GOING
A picture is a start. Practice is the rest.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 01, Lesson 06) and the Math Foundations Notebook reference build. Interactive figures, the falling-samples hero, worked exercise answers, and the distribution, expectation, PMF/PDF, and sampling labs are original to this page. Every lab runs in your browser.