I(p) = −log₂ p turns probability into surprise: a fair coin is 1 bit, a 1-in-1000 event is 9.97 bits. Average that surprise over a distribution and you have entropy — the quantity every loss function is built on.
An event with probability p carries I(p) = −log₂p bits. A fair coin is 1 bit, a six on a die is 2.58 bits, a 1-in-1000 event is 9.97 bits, and a certainty is 0 bits. Use the natural log instead and the same quantity is measured in nats (1 nat = 1.4427 bits).
I(0.5) = 1 bit · I(0.001) = 9.97 bitsH(P) = −Σ p log p is the expected surprise of a distribution. A fair coin carries exactly 1 bit; a coin that lands heads 99% of the time carries only 0.081 bits, because you already know the answer. A uniform distribution over K outcomes maximizes entropy at log₂K bits.
H(coin) = 1 bit · H(0.99 coin) = 0.081 bitsCross-entropy H(P,Q) = −Σ p log q is the average surprise when reality is P but the model believes Q. It splits cleanly: H(P,Q) = H(P) + D_KL(P‖Q). The true distribution is fixed during training, so minimizing the loss is exactly minimizing the extra bits wasted by the wrong model.
H(P,Q) = H(P) + D_KL(P‖Q)An unlikely event is surprising, and surprise is measured by −log p. Average surprise is entropy. The extra surprise you suffer by believing the wrong distribution is KL divergence, and cross-entropy — the loss you call every day — is entropy plus that extra surprise.
By the end you will be able to compute surprise, entropy, cross-entropy, KL divergence and perplexity from a handful of numbers, explain why minimizing cross-entropy is the same as minimizing KL, read a perplexity score like a practitioner, and see information theory behind classification losses, decision trees and language models.
When something unlikely happens, it tells you more than something you expected. Information theory makes that instinct exact with one formula: I(p) = −log p.
A coin landing heads is barely news. A lottery win is a headline. The difference is probability, and the information content of an event with probability p is I(p) = −log p. Log base 2 measures the answer in bits — the number of yes/no questions it would take to pin the event down. Log base e measures the same quantity in nats, and 1 nat = 1.4427 bits.
Plain English: take the probability, ask how many times you would have to halve it to reach 1, and that count is the surprise. A certainty never needs a question, so it carries 0 bits. A 1-in-1000 event needs almost ten.
Drag the probability and watch the curve point and the bar. Halving a probability adds exactly one bit of surprise.
Base 2 gives bits; base e gives nats. The two answers are the same surprise in different units: 1 nat = 1.4427 bits.
Ask what any reasonable measure of surprise should do, then find the only function that does it.
I(1) = 0.I decreases as p grows.I(p·q) = I(p) + I(q). Learning both pieces of news tells you the two things, no more.I(p) = c·log p. Positive and decreasing forces c < 0; choosing base 2 and c = −1 makes the unit the bit. So I(p) = −log p, forced by the three requirements.The logarithm is not decoration. It is exactly what turns “two independent things happened” into “add the surprises”.
The forecast said 90% rain, and it rained. Your friend's app said 10% rain, and it rained. Whose correct prediction carried more information?
A distribution is a whole family of possible events. Entropy is the expected surprise of one draw from it — the uncertainty you cannot remove, no matter how clever your encoder.
Entropy takes the surprise of every outcome and weights it by how often that outcome happens:
Plain English: entropy is the average number of yes/no questions a draw from the distribution costs. A fair coin is exactly 1 bit per flip. A coin that lands heads 99% of the time is only 0.081 bits, because you already know the answer. A fair six-sided die is log₂6 = 2.585 bits.
Slide the weights of the four outcomes. Only their shape matters: uniform is the most uncertain distribution, and a certain outcome has zero entropy.
Zero-weight outcomes contribute nothing: 0·log 0 is defined as 0. Entropy counts the surprise of what can actually happen.
The two terms are the two outcomes’ contributions. The rare outcome is individually surprising (6.64 bits), but it almost never happens, so its contribution is multiplied by 0.01 and the average collapses. This is why entropy is sometimes read as the amount of randomness: it is the uncertainty of the source, not of one dramatic outcome.
Two conventions worth fixing now. A term with probability zero contributes nothing — 0·log 0 is defined as 0, because an impossible outcome can never surprise you. And entropy is a property of the distribution’s shape: relabeling the outcomes does not change it.
A coin is so biased that it lands heads 999 times out of 1000. Roughly how much entropy does each flip carry?
Entropy depends on how probability is spread, not on how many labels there are. Among all distributions over K outcomes, the uniform one always wins — and that ceiling is worth memorizing.
The uniform distribution over K outcomes has entropy log₂K: every draw is maximally hard to predict. At the other extreme, a distribution that puts all its mass on one outcome has entropy 0 — that draw is free. Real distributions live in between, and the interesting ones are usually far closer to certainty than to uniform.
Plain English: entropy is the fair price of uncertainty. A model that knows the next word pays almost nothing to encode it; a model choosing among a uniform vocabulary of 50,000 tokens pays log₂50000 = 15.61 bits per token.
Write entropy as H(P) = Σ p·log(1/p) and apply Jensen’s inequality. Because log curves downward (it is concave), E[log X] ≤ log E[X]. Use the probabilities themselves as the weights and let X = 1/p:
Numeric checks with K = 4:
The dyadic example makes the rule visible: halving a probability adds exactly one bit of surprise, so probabilities of 1/2, 1/4, 1/8, 1/8 contribute 1, 2, 3 and 3 bits weighted by their own mass.
Reality follows P; your model believes Q. The average surprise of reality under the model’s beliefs is cross-entropy — and for a one-hot label it collapses to one clean number.
Cross-entropy uses the true probabilities to weight the model’s surprise:
Plain English: events really come from P, but you encode them with Q. You pay −log q(x) whenever x happens, and it happens with probability p(x). If Q = P you pay exactly the entropy. Any mismatch makes the average bill larger. For classification, P is a one-hot vector, so every term with p = 0 vanishes and only the true class survives:
The model’s raw scores are logits z; softmax turns them into probabilities, q = softmax(z). Then the loss is just the surprise of the correct answer, in nats. Three viewpoints agree that this is the right objective: it counts wasted bits (information), it is the negative log-likelihood of the data (statistics), and its gradient with respect to the logits is the beautifully simple p − y (optimization).
Move the logits, pick the true class, and change the temperature. The loss depends only on the probability the model gives the correct class: −ln p(true).
Temperature only rescales the logits: T > 1 flattens the distribution and raises its entropy; T < 1 sharpens it. With a one-hot label, the loss is exactly the surprise of the correct answer.
Softmax subtracts the largest logit before exponentiating. The shift cancels in the normalization, so it changes nothing mathematically and prevents overflow.
Notice how fast the bill grows when the true class is unlikely. Predicting 0.01 for the right answer costs 4.605 nats, more than ten times the 0.417-nat loss of a model that is on the right track.
Write the loss for true class t using the log-sum-exp form of softmax:
The gradient is bounded, smooth, and needs no exponentials beyond the forward pass. That is why softmax and cross-entropy are always paired, and why frameworks fuse them into a single numerical function.
A classifier gives the correct class probability 0.01. What is the cross-entropy loss in nats, and in bits?
If reality is P and your model believes Q, the extra surprise over perfect knowledge is the KL divergence. It is the piece of the loss training actually pushes on.
Cross-entropy measures the total bill. Entropy is the part you would pay even with perfect beliefs. The difference is the price of the wrong beliefs:
Plain English: for each outcome, compare what you paid (−log q) with the best possible price (−log p). The gap is extra bits wasted by using Q. Weight the gaps by how often each outcome happens, and you have KL divergence. It is zero exactly when Q matches P, and positive otherwise — but it is not symmetric: D(P‖Q) ≠ D(Q‖P) in general, so it is a divergence, not a distance.
Shape the truth P and the model Q. The cross-entropy bar splits into the entropy of the truth plus the extra bits KL charges for the wrong beliefs — and swapping P and Q shows the price is not symmetric.
truth P weights
model Q weights
KL is zero only when Q matches P, and it is infinite when Q gives zero probability to an outcome P allows. Swapping shows it is not a distance.
Take truth P = [0.7, 0.2, 0.1] and two models: a good Q = [0.6, 0.25, 0.15] and a bad Q = [0.1, 0.1, 0.8]. Everything in bits, base 2.
Why KL can never be negative. Use ln x ≤ x − 1 for every positive x, with equality only at x = 1 (the line is the tangent to the concave log curve at 1, so log lies below it). Put x = q/p:
Equality needs q/p = 1 everywhere, i.e. Q = P. So cross-entropy is always at least the entropy, and minimizing cross-entropy is exactly minimizing KL — perfect knowledge is the floor the loss is trying to reach.
Which statement about D_KL(P‖Q) is true?
Joint entropy, conditional entropy and mutual information measure dependence itself — including the curved, non-monotonic relationships correlation misses. Decision trees run on this idea.
Joint entropy H(X,Y) is the entropy of the pair. Conditional entropy H(Y|X) = H(X,Y) − H(X) is how much uncertainty is left about Y after you observe X. Mutual information is the reduction:
Plain English: how many bits of uncertainty about one variable disappear when someone tells you the other. If X and Y are independent, knowing one tells you nothing: I = 0. If X completely determines Y (Celsius and Fahrenheit), H(Y|X) = 0 and mutual information equals H(Y). It is symmetric, never negative, and I(X;X) = H(X) — a variable shares everything with itself.
The fourth formula above compares the true joint distribution p(x,y) with the “pretend they are independent” product p(x)·p(y). That is exactly a KL divergence — and since KL is never negative, I ≥ 0, with equality precisely when p(x,y) = p(x)p(y), the definition of independence.
Mutual information detects any statistical dependence, linear or not. Pearson correlation only sees straight lines; a U-shaped relationship can have correlation near zero while mutual information is high. That difference is why MI is used to rank features and why decision trees ask which feature removes the most uncertainty.
Eight examples, four labels “yes” and four “no”, so the label entropy starts at exactly 1 bit. Pick a split and watch how much uncertainty each branch leaves behind.
Information gain is the drop in label entropy: IG = H(Y) − H(Y|X). A good split leaves branches that are nearly pure.
| # | label | sky | humidity | wind |
|---|---|---|---|---|
| 1 | yes | sunny | high | none |
| 2 | yes | sunny | high | none |
| 3 | no | sunny | low | light |
| 4 | no | sunny | low | light |
| 5 | yes | cloudy | high | strong |
| 6 | no | cloudy | low | strong |
| 7 | yes | cloudy | low | gale |
| 8 | no | cloudy | low | gale |
Cross-entropy is measured in nats; perplexity converts it back into a head-count of equally likely choices. Label smoothing and temperature are the same information story told from the opposite side.
Perplexity is the exponential of cross-entropy. It answers one question: if the model’s uncertainty were spread evenly, over how many choices would it be spread?
Plain English: a model with perplexity 50 is, on average, as uncertain as a fair 50-sided die — at every token. Lower is better: perplexity 1 would mean it always knew the answer. GPT-2 reached roughly 30 on common benchmarks, and modern models sit in the single digits on well-represented text. A uniform model over a 50,000-token vocabulary has perplexity 50,000; that is the ceiling to compare against.
Set the probability the model gave each true token. Perplexity is the exponential of the average surprise — the effective number of equally likely tokens it is choosing between.
Lower is better. Perplexity 1 means the model was certain and right; perplexity equal to the vocabulary size means it was guessing uniformly.
Temperature divides the logits before softmax: q = softmax(z / T). It does not change the ranking, only how sharply the probability mass concentrates. Watch the entropy of the prediction move, for z = [2, 1, 0.1]:
Label smoothing works on the target instead of the prediction. With ε = 0.1 and four classes, the hard one-hot [0, 0, 1, 0] becomes:
The smoothed target has positive entropy, so the model is never asked to be 100% certain — a finite, reachable goal. The loss becomes L = (1−ε)·CE(hard target, q) + ε·H(uniform, q), where the second term is a direct penalty on confident predictions. That is regularization and calibration with an information-theoretic name.
Answer before you look. The KL and perplexity questions are the ones that separate having read about the loss from understanding what it charges your model for.
0 / 5 answered · 0 correct
01What does “entropy” measure in information theory?
02What is the cross-entropy loss commonly used for in neural networks?
03Why is minimizing cross-entropy equivalent to minimizing KL divergence during training?
04A language model has perplexity 50 on a test set. What does this mean?
05How does mutual information differ from Pearson correlation for feature selection?
Click a card to swap the lazy description for what it actually means.
Four short problems. Try first; a worked answer is one click away.
Uniform: H = log₂ 26 = 4.700 bits — the most uncertainty a 26-letter alphabet can carry. Real English is uneven (E ≈ 12.7%, T ≈ 9.1%, but Z and Q ≈ 0.1%), and published frequencies put the entropy near 4.1 bits. The uniform distribution is higher: uneven probabilities make common letters unsurprising, so the average surprise falls. Small dyadic check: uniform over 4 symbols is 2 bits; moving to [0.5, 0.25, 0.125, 0.125] gives 0.5·1 + 0.25·2 + 0.125·3 + 0.125·3 = 1.75 bits.
Shift by the max (5) so exponentials stay safe: [0, −3, −4.5]. Σeᶻ = 1 + 0.049787 + 0.011109 = 1.060896, so p₁ = e⁻³/1.060896 = 0.046929. Loss = −ln 0.046929 = 3.059 nats, which is 3.059 × 1.4427 = 4.413 bits. Zero loss would need p₁ = 1, i.e. the true logit infinitely above the other two — a finite gap always leaves some probability mass elsewhere, so training approaches zero but never reaches it.
D_KL(P‖Q) = 0.9·ln(0.9/0.5) + 0.1·ln(0.1/0.5) = 0.9·ln 1.8 + 0.1·ln 0.2 = 0.5290 − 0.1609 = 0.368 nats (0.531 bits). D_KL(Q‖P) = 0.5·ln(0.5/0.9) + 0.5·ln(0.5/0.1) = −0.2939 + 0.8047 = 0.511 nats (0.737 bits). They differ because the same two log-ratios (ln 1.8 and ln 0.2) are weighted by different probabilities: P weights the terms by [0.9, 0.1], Q by [0.5, 0.5]. Direction matters — which distribution supplies the weights is exactly what “P‖Q” names.
Per-token NLL: 0.6931 + 1.3863 + 1.3863 = 3.4657 nats; average 1.1552 nats/token; perplexity = e^1.1552 = 3.175. The shortcut is PPL = 1 / (p₁p₂p₃)^(1/3) = 1 / (0.5·0.25·0.25)^(1/3) = 1 / 0.3150 = 3.175. For p = 1/50 everywhere, average NLL = ln 50 = 3.912 nats and PPL = e^3.912 = 50 exactly: the model's uncertainty is the same as choosing uniformly among 50 tokens. In code: ppl = exp(mean(−log(p) for p in probs)).
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 01, Lesson 09) and the Math Foundations Notebook reference build. Interactive figures, the animated surprise hero, exact numeric checks, worked exercise answers and the entropy, KL, softmax, information-gain and perplexity labs are original to this page. Every lab runs in your browser.