posterior ∝ likelihood × prior Start with what you believe, see what happened, ask how well each hypothesis would have predicted it, and renormalize. That is the whole recipe — and it runs the medical test, the spam filter and the A/B test.
The prior says what you believed before. The likelihood says how well each hypothesis predicted the evidence you actually saw. Multiply them, divide by the total probability of the evidence, and you have the posterior — your updated belief.
posterior = likelihood · prior / evidenceFor a disease at 1 in 10,000, the test catches the one sick person but false-alarms on about 100 healthy ones. Among roughly 101 positives, only about 1 is real. The prior is tiny, and it dominates.
1 true positive vs ~100 false alarmsA Beta(a, b) belief about a coin's bias, updated with s heads and f tails, becomes Beta(a+s, b+f). No integrals, no retraining — the past posterior becomes the next prior, which is how online learning works.
Beta(1,1) + 7H 3T + 5H 5T = Beta(13,9)Bayes’ theorem is the arithmetic of changing your mind: posterior = likelihood × prior / evidence, where the evidence is nothing more than the total probability of what you just saw — and the number people forget to compute.
By the end you will be able to tell P(A|B) from P(B|A) on sight, compute a posterior by hand (and explain why a 99% test can leave you 1% sure), smooth a Naive Bayes classifier, update a Beta belief flip by flip, and explain why every weight-decay term is secretly a prior.
P(A|B) restricts attention to the worlds where B happened and asks what fraction also have A. Swap the two letters and you have a different question with a different answer.
Conditional probability is the arithmetic of narrowing the world. P(A|B) is read “the probability of A given B”: take every case where B happened, and ask in what fraction of them A also happened.
Roll two dice. Let A = “the sum is 8” and B = “the first die shows 3”. There are 36 equally likely outcomes.
| Expression | Plain English | Name in ML |
|---|---|---|
P(positive | sick) | among sick people, how many test positive? | sensitivity · 99% |
P(sick | positive) | among positive tests, how many are sick? | the posterior · 1/102 ≈ 0.98% |
P(flag | fraud) | of all fraud, how much does the model catch? | recall · 95% |
P(fraud | flag) | of everything flagged, how much is fraud? | precision · depends on the base rate |
Every statement conditions on something. Decide which probability it reports — then check the reveal.
1 / 5 · score 0
A test's brochure says: “detects 99% of infections”.
Which probability is the brochure quoting?
P(A|B) restricts the world to the cases where B happened. P(B|A) restricts it the other way. Same numerator, different denominator — and a completely different number.
Conditional probability defines both directions. Set the two expressions for the joint probability equal and Bayes’ theorem falls out in one rearrangement.
Start from Lesson 06. The probability of A given B is the joint probability divided by P(B); the probability of B given A is the same joint probability divided by P(A). Both statements describe the same overlap, so the two products are equal.
| Term | Name | Plain meaning |
|---|---|---|
P(A|B) | Posterior | What you believe about A after seeing B |
P(B|A) | Likelihood | How probable the evidence is if A is true |
P(A) | Prior | What you believed about A before any evidence |
P(B) | Evidence | How probable the evidence is overall; makes the posterior sum to 1 |
Move the three sliders. The bars share one 0-to-1 axis, so the shift is the whole story.
Set the false alarm rate above the likelihood and the posterior falls below the prior: evidence only moves belief in the direction it discriminates. That ratio is all of Bayes in one line.
Either A happens or it does not — there is no third option, and the two cannot both happen. So the event B splits cleanly into two non-overlapping pieces: “B and A” and “B and not A”. Probabilities of non-overlapping events add, and each piece expands with the product rule:
Second worked example — two urns. One urn is chosen at random: urn 1 holds 3 red and 7 blue balls, urn 2 holds 8 red and 2 blue. First compute the evidence, then flip the condition with Bayes.
Two hypotheses, one observation, exact fractions — and the posterior for each hypothesis still sums to 1. That is the evidence term doing its only job.
In the medical example, P(+) = 0.99·0.0001 + 0.01·0.9999. Which rule is that?
A test is 99% accurate. You test positive. Before you panic, ask how rare the disease is — because the healthy majority is where most positive results come from.
A disease affects 1 in 10,000 people. The test catches 99% of sick people (sensitivity) and false-alarms on 1% of healthy people. You test positive. Bayes says:
Every square is one person out of 10,000. Move prevalence, sensitivity and the false positive rate, then read who makes up the positives.
The exact posterior uses the unrounded prior; the grid rounds to whole people. Watch a rare disease make most positives false alarms — and a second test restore the balance.
Prevalence, sensitivity, specificity — the three numbers a real test is described by. Everything else is Bayes.
composition of all positive tests: 0.98% true · 99.02% false alarm
A specificity of 0.99 means 1 in 100 healthy people test positive. With 9,999 healthy people, that is about 100 false alarms — which is why the posterior, not the sensitivity, is the number a patient wants.
Both the numerator and denominator of the posterior are decimals with six digits after the point, so multiply top and bottom by one million:
A second, independent positive test. Today’s posterior becomes tomorrow’s prior. Substitute 1/102 for 0.0001 and run Bayes again:
Two tests did not add 99% + 99%. They moved the belief from 1 in 10,000 to about 1 in 2 — because the second test starts from a world where you are already a hundred times more likely to be sick.
| Evidence so far | P(sick) | As a percentage |
|---|---|---|
| before any test | 1 / 10,000 | 0.01% |
| after one positive | 1 / 102 | ≈ 0.98% |
| after two independent positives | 99 / 200 | 49.5% |
A 99%-sensitive test screens for a disease that affects 1 in 10,000 people. Your test is positive. The chance you actually have the disease is closest to:
When the prior and the posterior come from the same family, updating is arithmetic instead of integration. For a coin’s bias, two numbers are the whole state of your belief.
Bayesian inference is a loop: write down a prior, observe data, compute the posterior, and let that posterior become the next prior. Nothing needs to be retrained or replayed — the belief is the summary. When prior and posterior share a family, the update skips the integral entirely. That is what conjugate means.
The Beta distribution is the conjugate prior for a coin: Beta(a, b) represents your belief about a probability parameter p, where p is the coin’s bias. a counts successes plus one, b counts failures plus one, and the update rule is dead simple.
Each head or tail updates Beta(a, b) → Beta(a+1, b) or Beta(a, b+1). The curve sharpens; the mean slides.
The MLE is the peak of the likelihood; the MAP is the peak of the posterior; the mean is the center of mass. They agree only when the prior is flat — press “Simulate 100” with a flat prior and a strong one to feel the difference.
Notice the day-3 update: five heads and five tails is perfectly balanced, yet it did not leave the belief where it was — it diluted the earlier evidence. The mean moved from 0.667 toward 0.5 because the posterior is an average of everything seen so far, weighted by counts.
| Likelihood | Conjugate prior | Posterior | Example |
|---|---|---|---|
| Bernoulli | Beta(a, b) | Beta(a + successes, b + failures) | coin bias, click-through rate |
| Normal (known variance) | Normal(μ₀, σ₀) | Normal(weighted mean, smaller variance) | sensor calibration |
| Poisson | Gamma(a, b) | Gamma(a + Σ counts, b + n) | arrival rates |
| Multinomial | Dirichlet(α) | Dirichlet(α + counts) | topic models, language models |
Starting from Beta(3,2) you observe 5 heads and 5 tails. What is the posterior, and its mean?
One word can move a spam score from 30% to 95%. A whole message multiplies many such clues — and a naive independence assumption makes that multiplication legal.
Start with one word. Thirty percent of your mail is spam; “lottery” appears in 5% of spam and 0.1% of legitimate mail. What is P(spam | “lottery”)?
A real message has many words. Bayes wants P(w₁, w₂, …, wₙ | class) — the joint probability of every word together — which is hopeless to estimate because there are too many word combinations. The naive assumption says each word depends only on the class, not on the other words. Every factor collapses to a single-word likelihood, and the joint becomes a product:
The independence assumption is obviously false — “New” and “York” travel together — but the classifier only needs to rank classes, not produce perfectly calibrated probabilities, and it works far better than the assumption deserves. Two engineering details make it usable:
Log space. Multiply 200 probabilities and the product underflows to zero; add 200 logs and it stays finite. Laplace smoothing. P(w|c) comes from counts — the maximum likelihood estimate — so a word never seen in spam training data gets probability zero, and one zero annihilates the entire product. Add a small count to every word instead:
Smoothing is still a probability distribution. Sum the smoothed probabilities over every word in the vocabulary V:
Worked message on the lab’s 12-email corpus: 5 spam with 21 tokens, 7 ham with 29 tokens, vocabulary 44. With α = 1, P(w|spam) = (count+1)/65 and P(w|ham) = (count+1)/73. Score “free money waiting for you”:
“waiting” and “for” were never seen in training. Without smoothing they would contribute log 0 = −∞ to both classes, making the comparison meaningless. With α = 1 they contribute a small penalty to each side.
Type a message. Every word contributes one log-probability to each class; the bigger total wins.
| word | spam count | log P(w | spam) | ham count | log P(w | ham) |
|---|---|---|---|---|
| free | 5 | -2.383 | 0 | -4.290 |
| money | 1 | -3.481 | 0 | -4.290 |
| waiting ⚠ | 0 | -4.174 | 0 | -4.290 |
| for ⚠ | 0 | -4.174 | 0 | -4.290 |
| you | 1 | -3.481 | 0 | -4.290 |
⚠ = never seen in training; smoothing keeps it finite and nearly uninformative. Vocabulary 44 words · spam tokens 21 · ham tokens 29.
Words seen in both classes push the two scores apart only a little; a word that is common in spam and absent from ham pushes hard. Set α near 0.01 and type a word the model has never seen to watch the veto disappear — smoothing is what keeps it finite.
A word appears only in the ham training data. With no smoothing (α = 0), what happens when Naive Bayes sees it in a test message?
One extra factor in the maximization — the prior over parameters — turns maximum likelihood into MAP, and turns weight decay into a Bayesian statement.
Maximum likelihood (MLE) picks the parameters that make the observed data most probable: argmax P(data | θ). For counts, that is just the plain relative frequency. Maximum a posteriori (MAP) multiplies in a prior over the parameters themselves: argmax P(data | θ)·P(θ). If your prior says “weights should be small”, the extra term penalizes large weights — which is precisely what a regularization term does.
| Estimation | Optimizes | ML equivalent |
|---|---|---|
| MLE | P(data | θ) | Unregularized training |
| MAP | P(data | θ) · P(θ) | L2 (Gaussian prior) or L1 (Laplace prior) |
| Aspect | Frequentist | Bayesian |
|---|---|---|
| Output | A point estimate (one number) | A distribution over values (uncertainty included) |
| Uncertainty | Confidence interval: about the procedure | Credible interval: about the parameter |
| Small data | Can overfit the sample | The prior acts as regularization |
| Computation | Usually faster (SGD, closed forms) | Often needs sampling (MCMC) or conjugate shortcuts |
When is Bayes the right frame? When uncertainty is the product you are selling: a medical or safety decision where “0.9 probability” needs a calibrated range around it; a cold-start recommender with almost no data, where a sensible prior is better than a noisy frequency; a system that must update as data streams in; or a model comparison where you want to weigh evidence for a hypothesis, not just reject a null. Most production ML is frequentist — stochastic gradient descent to a point estimate — and that is fine. Bayesian methods earn their extra cost exactly where a distribution is more useful than a number.
Answer before you look. The base-rate and MAP questions are the two that show up most often in real model reviews.
0 / 5 answered · 0 correct
01In Bayes' theorem, what is the prior?
02A disease affects 1 in 10,000 people. A 99% accurate test returns positive. Roughly, what is P(disease | positive)?
03What is Laplace smoothing in Naive Bayes, and why is it necessary?
04How does MAP differ from MLE?
05Starting from Beta(1,1) you observe 7 heads and 3 tails. What is the posterior?
Click a card to swap the lazy description for what it actually means.
Four short problems. Try first; a worked answer is one click away.
First test: P(+) = 0.99·0.0001 + 0.01·0.9999 = 0.000099 + 0.009999 = 0.010098, and P(sick|+) = 0.000099/0.010098 = 1/102 ≈ 0.0098 (0.98%). Second test, with 1/102 as the new prior: P(+) = 0.99·(1/102) + 0.01·(101/102) = (0.99 + 1.01)/102 = 2/102 = 1/51. Posterior = (0.99/102) / (1/51) = 0.495 — exactly 49.5%. Two independent positives move the belief from 1 in 10,000 to about 1 in 2.
In the 12-email corpus: 5 spam mails with 21 word tokens, 7 ham mails with 29 tokens, vocabulary 44. P(“free”|spam) = (5 + α)/(21 + 44α). α = 0.01 → 5.01/21.44 ≈ 0.2337. α = 0.1 → 5.1/25.4 ≈ 0.2008. α = 1 → 6/65 ≈ 0.0923. α = 10 → 15/461 ≈ 0.0325. Bigger α pulls every estimate toward the uniform value 1/44 ≈ 0.0227. At α = 0, any word never seen in spam gets probability 0, and log 0 = −∞ vetoes the entire message regardless of the other words — one unseen word becomes infinitely strong evidence against spam.
Message length is one more Bernoulli feature, so it contributes one more factor. In the corpus 4 of 5 spam mails are short (only “claim your prize today free” has 5 words) and 4 of 7 ham mails are short. With add-one smoothing: P(short|spam) = 5/7 ≈ 0.7143, P(short|ham) = 5/9 ≈ 0.5556, so P(long|spam) = 2/7 ≈ 0.2857 and P(long|ham) = 4/9 ≈ 0.4444. For “free money waiting for you” (5 words → long): spam score −18.5694 + log(2/7) = −19.8222, ham score −21.9913 + log(4/9) = −22.8022. The verdict is still spam and the gap narrows from 3.42 to 2.98 — a weak feature adds a small nudge, it does not overrule the words.
Posterior ∝ p^(2−1)(1−p)^(2−1) · p⁷(1−p)³ = p⁸(1−p)⁴ = Beta(9,5). MAP = posterior mode = (9−1)/(9+5−2) = 8/12 ≈ 0.6667, below the MLE 7/10 = 0.7; the prior pulled the estimate toward 0.5. The posterior mean is 9/14 ≈ 0.6429. The pull shrinks as data accumulates: with 70 heads in 100 flips the same prior gives (70+1)/(100+2) = 71/102 ≈ 0.6961, barely below the MLE 0.7.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 01, Lesson 07) and the Math Foundations Notebook reference build. Interactive figures, the animated Beta-updating hero, the fallacy trainer, the medical calculator, the worked exercise answers and the population-grid canvas are original to this page. Every lab runs in your browser.