P(spam | words) ∝ P(spam) · Π P(word | spam) is the whole classifier: one honestly wrong assumption (words arrive independently), a table you fill by counting, and a verdict read off the bigger log score.
The corpus says 40% of emails are spam, so that is the prior. Seeing “free” is 81/153 likely under spam but only 6/118 likely under not-spam, and the posterior jumps to 0.874; two more words take it to 0.997. Bayes' rule is the engine — Naive Bayes only decides how to compute the likelihood.
0.40 → 0.874 → 0.997A 10,000-word vocabulary has 2^10,000 possible word bundles — no dataset could estimate that joint. Assuming words are independent given the class collapses it to one count per word. Correlated words then get counted twice, which inflates confidence but usually keeps the correct winner.
naive ratio 15 · true ratio 5Word frequencies go to Multinomial NB. Short texts where only presence/absence is reliable go to Bernoulli NB — it also penalizes missing words. Continuous measurements go to Gaussian NB, one mean and variance per feature per class. In every variant, training is counting and prediction is a dot product.
counts · bits · measurementsNaive Bayes is Bayes’ rule with a countable likelihood: count how often each word appears per class, add a little smoothing so nothing is impossible, add up each word’s log-likelihood, and pick the class with the larger score — a wrong assumption used to produce a right ranking.
By the end you will be able to derive the smoothed probability formula from the counts, trace a message through priors, likelihoods and logs to a posterior by hand, explain why the independence assumption still ranks well, tell the three variants apart on sight, guard against the unseen-word zero, and know when to reach for logistic regression instead.
Bayes’ rule turns “how likely are these words in spam?” into “how likely is spam given these words?” Four pieces, one multiplication — and Naive Bayes only has to decide how to count one of them.
We want P(class | features) — the probability that a message belongs to a class once we have read its words. That is hard to measure directly, but the reverse is easy to count: take all the spam in the training set and ask how often each word appears. Bayes’ rule connects the two directions:
Plain English: start from what you believed, multiply by how well the evidence fits each story, then compare. Because the evidence term is identical for spam and not-spam, only prior × likelihood matters for picking the winner. The posterior we report at the end is just that product rescaled so the two classes sum to 1.
The corpus for the whole lesson: spam messages contain free 80 times, money 60, meeting 10 — 150 words total. Not-spam messages contain free 5, money 10, meeting 100 — 115 words total. And 40% of emails are spam. One email arrives containing free once.
One word moved the belief from 0.40 to 0.89, because “free” is about twelve times more common in spam (0.5333 / 0.0435 = 12.27). Two more words and the posterior reaches 0.997 — the arithmetic the lab below performs live with Laplace smoothing switched on.
Edit the message. Every known word adds its own log-likelihood to each class score; the bigger score wins. Nothing here is trained — the two tables below are the whole model.
bar = P(ham) blue · P(spam) orange. Priors are 0.4 / 0.6; the smoothing strength is α = 1.
| word | count · spam | count · ham | P(w | spam) = (c+1)/(150+3) | P(w | ham) = (c+1)/(115+3) |
|---|---|---|---|---|
| free | 80 | 5 | 0.5294 | 0.0508 |
| money | 60 | 10 | 0.3987 | 0.0932 |
| meeting | 10 | 100 | 0.0719 | 0.8559 |
| term in message | n | n · log P(spam) | n · log P(ham) | log-likelihood ratio per occurrence |
|---|---|---|---|---|
| free | 2 | -1.272 | -5.958 | +2.343 |
| money | 1 | -0.920 | -2.373 | +1.453 |
| log priors | -0.916 | -0.511 | -0.405 |
Comparing two classes, the shared denominator P(features) appears in both and can be removed. What survives is a score per class. To turn scores back into probabilities that sum to 1, divide each score by the sum of scores — for two classes that is the familiar logistic shape:
The difference of logs is the log-odds: e5.7336 = 309.1, so the odds are about 309 to 1 in favor of spam. Logs turn multiplication into addition, and classification into comparing two sums.
A word is 10× more likely in spam than in ham, and the priors are equal. A message contains it once. What is P(spam | message)?
Estimating P(all words together | class) needs more data than exists. Assuming the words do not interact makes the likelihood countable — and, surprisingly, keeps the ranking.
The likelihood in Bayes’ rule is a joint event: these exact words appearing together. With a 10,000-word vocabulary there are 210,000 possible word bundles; even the web could not fill that table with reliable counts. The naive assumption fixes this:
The assumption is obviously false — “machine” and “learning” travel together in every real document. Three things explain why it still classifies well. First, ranking beats calibration: we need the right top class, not a right-to-the-last decimal posterior. Second, high bias is low variance: a strongly constrained model may be wrong about the world but is stable when data is scarce. Third, redundant evidence cancels: correlated words are double-counted — but double-counted for the class they support, so the winner often survives. A fourth, practical reason: training is one pass of counting, and prediction is a single matrix multiply.
Two words in a 100-document TECH corpus and a 100-document SPORTS corpus. Slide the coupling ρ: at 0 the words are independent, at 1 they always appear together. The product the naive model multiplies never changes.
The assumption inflates confidence by counting correlated words twice; the ranking survives while the posterior gap between classes is wide. Set P(TECH) = 0.10 with ρ = 1 and the double-counted evidence is enough to flip the verdict — confidence became the decision.
Take the lab’s corpora. In TECH, machine appears in 60 of 100 documents and learning in 50. In SPORTS, machine is in 20 and learning in 10. Independence says the joint probability is the product:
So the honest statement is not “the assumption does not matter”. It is: with many individually weak and roughly class-aligned features, the systematic error inflates confidence without usually changing the winner; when a few strongly correlated features carry the decision, it can flip. The lab lets you find both regimes.
Naive Bayes reports P(spam) = 0.99999 where the true posterior is about 0.70. Why is the classifier still useful?
Naive Bayes is a family, not a model. The variant only changes how P(feature | class) is estimated — Multinomial counts, Bernoulli checks presence, Gaussian measures a bell curve.
Pick the variant by asking what a feature is. A word that can appear several times is a count. A short message where one mention is all you get is a bit. A sensor reading or measurement is a real number. Each choice gives a different likelihood formula, but the Bayes’ rule skeleton and the log-space sum stay identical.
| Variant | P(feature | class) | Feature type | Best for | Example |
|---|---|---|---|---|
| Multinomial | (count + α) / (total + α·|V|) | non-negative counts | bag-of-words, TF-IDF | email spam, topic labels |
| Bernoulli | (docs with word + α) / (docs + 2α) | 0 / 1 presence | short texts, binary vectors | SMS spam, symptoms |
| Gaussian | 𝒩(x; μ, σ²) per feature | continuous values | tabular, sensor features | iris flowers, measurements |
Two continuous features, two classes. Every class learns one mean and one variance per feature; the decision line is where the two log likelihoods tie. Move the probe and watch the posterior.
This is why sklearn’s GaussianNB adds var_smoothing: it keeps every variance positive so the log never becomes −∞. Continuous features do not need scaling — the per-class statistics already absorb it.
For a continuous feature, each class stores a mean μ and a variance σ² per feature. The likelihood of a value x is the height of the normal bell at x:
With equal variances the boundary is the midpoint of the means. When the variances differ, the quadratic term tilts the boundary, which is why Gaussian NB draws curved (quadratic) boundaries — visible in the lab.
Treat the lesson corpus as 40 spam documents and 60 ham documents with presence counts: in spam, free appears in 32 docs, money in 28, meeting in 6; in ham, free in 4, money in 7, meeting in 45. With α = 1, Bernoulli uses (present docs + 1) / (docs + 2), and it must also score every absent word.
Bernoulli and Multinomial disagree in detail but not in verdict here. Bernoulli loses the repetition information (“free free free” looks like “free”) and gains explicit absence evidence — which is why it wins on very short texts, where counts are noisy but presence is reliable.
Your features are TF-IDF weights: non-negative but real-valued, with most entries 0. Which variant fits, and why?
There is no gradient descent hiding in this lesson. Fitting a Multinomial Naive Bayes model visits every token once, writes counts into a table, and turns the table into log probabilities.
fit(X, y) does four things. For each class: count how often every word appears in its documents; add α to each count (next chapter); divide by the class’s total (plus α·|V|) to get probabilities; store the log of each. It also stores the prior — the log of the class’s share of the training examples. The result is a table with one row per class and one column per word, plus a bias per class.
The vocabulary itself is built from the training split only. Words that never appear in training have no column; they are simply ignored at prediction time. Everything the model knows is visible on this page — no hidden state, no epochs, no learning rate.
| word | spam count | ham count | P(w | spam), α = 1 | P(w | ham), α = 1 |
|---|---|---|---|---|
| free | 80 | 5 | 81/153 = 0.529412 | 6/118 = 0.050847 |
| money | 60 | 10 | 61/153 = 0.398693 | 11/118 = 0.093220 |
| meeting | 10 | 100 | 11/153 = 0.071895 | 101/118 = 0.855932 |
| total | 150 | 115 | priors: 40 spam / 100 emails → 0.4 · 60 ham → 0.6 | |
class MultinomialNB:
def __init__(self, alpha=1.0):
self.alpha = alpha
def fit(self, X, y):
self.classes_ = np.unique(y)
for i, c in enumerate(self.classes_):
X_c = X[y == c]
self.class_log_prior_[i] = np.log(X_c.shape[0] / X.shape[0])
counts = X_c.sum(axis=0) + self.alpha
self.feature_log_prob_[i] = np.log(counts / counts.sum())
return self
def predict_log_proba(self, X):
# counts @ log-prob table + log priors
return X @ self.feature_log_prob_.T + self.class_log_prior_Stack the log probabilities as rows = words, columns = classes, and the message “free free money” as the count vector [2, 1, 0]. Prediction is literally the product below — the same operation as a single linear layer.
Training cost: one pass over the documents. Prediction cost: for n messages, d words and k classes, one (n × d) @ (d × k) multiply. That is linear in every dimension — the reason this model runs on millions of documents.
One word with probability zero multiplies an entire class score down to zero. Laplace smoothing gives every word a small positive count so no single gap can veto the evidence.
Suppose “discombobulate” never appears in a spam training message: its count for spam is 0, so P(discombobulate | spam) = 0/150 = 0 and the whole spam score becomes 0 — even if the message also contains “free” ten times. A probability is a product, and one zero destroys a product. The fix is to add a small α to every count before dividing:
The corpus plus one rare word, “discombobulate”. Slide α from 0 to 1 and watch P(word | class) = (count + α) / (total + α·|V|) move every word away from zero.
With α = 1 the standard Laplace rule adds one phantom occurrence of every vocabulary word; with α = 0.1 it adds a tenth. Larger α lifts rare words more, which helps with small corpora and unseen words — but it also drags common words toward uniform and blurs the signal.
Why total + α·|V| and not just total? Because the smoothed values must still be a probability distribution that sums to 1. Work it through with the lesson’s spam counts (free 80, money 60, meeting 10, N = 150, |V| = 3), α = 1:
A probability interpretation: smoothing is what Bayes would do. Place a uniform Dirichlet prior with α pseudo-counts on every word, observe counts, and the posterior mean is exactly (count + α) / (total + α|V|). α is how many phantom documents you are willing to add; it is a knob, not a fudge.
Laplace smoothing is not the only option — Lidstone smoothing uses the same formula with α < 1, and interpolation smoothing blends the class-specific estimate with a global one — but they all share the goal: never let a zero vote.
Extend the corpus with discombobulate: it appeared 3 times in the ham training words and 0 times in spam, so the totals become 150 (spam) and 118 (ham), |V| = 4. Message: “free discombobulate”.
The same message gives three different verdicts depending on α, because the rare word is the only thing separating them. That is why α is chosen on validation data, and why α = 1 is the safe starting point rather than a law.
| α | Effect | When to use it |
|---|---|---|
| 0.001 | almost no smoothing — trust the counts | very large corpus, no unseen words expected |
| 0.1 | light smoothing | large corpus |
| 1.0 | standard Laplace smoothing | the default starting point |
| 10.0 | heavy smoothing — distributions flatten | tiny corpus, many unseen words expected |
As α → ∞, what happens to P(spam | “free free money”)?
A message is a product of hundreds of probabilities. Done directly it underflows to zero; done in logs it is a sum that stays finite — and the prediction becomes the dot product from the counting chapter.
Multiply enough numbers smaller than 1 and the result leaves the range of floating point. A double can represent values down to about 1e−308, so a product of a few hundred small probabilities rounds to exactly 0.0 — and once it is 0, every class that shares it is tied at 0 and the comparison is meaningless. The logarithm fixes this without changing the ranking, because log is increasing and turns products into sums:
A message of n identical words that each have probability p under a class. The raw product is what a naive implementation multiplies; the log score is what the classifier should add.
log(a·b) = log(a) + log(b), so adding logs is the same arithmetic without the tiny intermediate numbers. A single repeated word is the worst case; real messages mix probabilities and die far sooner.
The full model at α = 1: each word’s smoothed probability, its log, then the weighted sum. Message A is “free free money”; message B is “free meeting”.
Both traces used only additions. The exponentials are needed once, at the end, only if you want probabilities rather than a ranking — and a numerically careful implementation subtracts the maximum log score before exponentiating (the log-sum-exp trick) so even the final step cannot overflow.
The log score is the log of the unnormalized product, so exponentiating it must return the product. Numeric check with message A:
The numbers agree exactly because log and exp are inverses. The difference is only in what the machine can represent along the way: the raw product of 300 small probabilities is not representable, while the sum of their logs is an ordinary negative number.
The independence assumption is a bias knob. With little data it is a gift; with lots of data it is a ceiling. Know which regime you are in before choosing the classifier.
Naive Bayes fails when the independence assumption changes the ranking, not merely the probabilities. Three failure modes are worth memorizing. Strong interactions: if the class depends on a combination of features (an XOR-like pattern) and neither feature alone carries signal, the product of marginals sees nothing. Correlated features with opposing evidence: if A and B always agree in reality but the table says A favors spam and B favors ham, the double count fabricates a conflict. Plenty of data: with enough examples, logistic regression learns the true boundary and overtakes the constrained model.
It wins on the opposite regime: many features, each individually weak, few labeled examples, and a text-shaped problem where errors wash out. That is exactly why it is the baseline everyone keeps.
| Aspect | Naive Bayes | Logistic regression |
|---|---|---|
| Type | generative: models P(X | Y) and P(Y) | discriminative: models P(Y | X) directly |
| Training | count frequencies, one pass | optimize a loss, many passes |
| Small data | better — the strong prior keeps it stable | worse — not enough data to fit the weights |
| Large data | worse — the wrong assumption starts to bind | better — the flexible boundary wins |
| Features | assumes conditional independence | handles correlations |
| Speed | single pass; prediction is one matmul | iterative optimization |
| Calibration | poor — probabilities are overconfident | better probabilities |
Forty held-out messages with their model scores. Slide the decision threshold and watch precision, recall and F1 move — accuracy barely notices.
positive class = spam · green = correct · red = error. The threshold line is drawn from the slider, and every number on the right is recounted from these forty rows.
| n = 40 | predicted spam | predicted ham | row total |
|---|---|---|---|
| actual spam | TP 17 | FN 3 | 20 |
| actual ham | FP 5 | TN 15 | 20 |
| column total | 22 | 18 | 40 |
The console’s forty held-out messages with their model scores. At the default threshold 0.5 on P(spam):
Why accuracy alone misleads. Suppose 95% of a corpus is ham. The constant classifier “always ham” scores 95% accuracy and catches 0% of spam. Accuracy answers “how often are we right?” when the question that matters is “of the spam, how much did we catch, and how much good mail did we burn?” — precision and recall, read together, answer that.
Which threshold ships? It depends on the cost ratio: a false positive hides a real email, a false negative lets spam through. If the company can tolerate a spam or two but not a lost invoice, 0.75 is right; if a spam message is expensive (phishing), move the threshold left and accept more false alarms.
Answer before you look. The independence question and the Multinomial versus Gaussian question are the two that separate a memorized definition from a working instinct.
0 / 6 answered · 0 correct
01What is the “naive” assumption in Naive Bayes?
02What does Laplace smoothing prevent in Naive Bayes?
03The naive independence assumption is clearly wrong for text. Why does Naive Bayes still classify well?
04When should you use Multinomial NB versus Gaussian NB?
05An email contains “free” twice and “money” once. In Multinomial NB with log probabilities, how is the spam score computed?
06Why does Naive Bayes predict in log space instead of multiplying probabilities directly?
Click a card to swap the lazy description for what it actually means.
Four problems with exact numbers — an α sweep, an independence test, a Bernoulli implementation and a confusion matrix. Try first; a worked answer is one click away.
Using P(word | class) = (count + α) / (total + 3α), the posterior for “free free money” is 0.997828 at α = 0.01, 0.997741 at α = 0.1, 0.996775 at α = 1, 0.977733 at α = 10 and 0.690897 at α = 100. Small α keeps the data's counts and the evidence stays sharp; at α = 100 every word probability drifts toward 1/3 (P(free | spam) = 180/450 = 0.400, P(money | spam) = 160/450 = 0.356, P(meeting | spam) = 110/450 = 0.244), so the likelihood ratio between spam and ham compresses and the posterior slides back toward the prior 0.4. The unseen-word test is the other extreme: at α = 0 an unseen word makes both class products 0, so the verdict is 0/0 — undefined. α is a hyperparameter: start at 1.0 and tune it on validation data, never on the test set.
Naive estimate: P(machine)P(learning) is (60/100)(50/100) = 0.30 for TECH and (20/100)(10/100) = 0.02 for SPORTS, a likelihood ratio of 15. True joint with full correlation: 50/100 = 0.50 for TECH and 10/100 = 0.10 for SPORTS, a ratio of 5 — the naive model overstates the evidence threefold. Posterior check with prior P(TECH) = 0.10: naive sees 0.10 × 0.30 = 0.030 for TECH against 0.90 × 0.02 = 0.018 for SPORTS, so it still picks TECH (0.625). The true joints give 0.10 × 0.50 = 0.050 against 0.90 × 0.10 = 0.090, so the Bayes-optimal answer is SPORTS (0.643). The independence assumption did not just muddy the probability — it flipped the winner, because the correlated words double-count evidence that a strong prior was already arguing against.
Bernoulli probabilities with α = 1: P(free | spam) = 33/42 = 0.785714, P(money | spam) = 29/42 = 0.690476, P(meeting | spam) = 7/42 = 0.166667; P(free | ham) = 5/62 = 0.080645, P(money | ham) = 8/62 = 0.129032, P(meeting | ham) = 46/62 = 0.741935. For the message, “money” is absent, so use 1 − p: log-spam = log 0.4 + log 0.785714 + log 0.833333 + log 0.309524 = −4.121933; log-ham = log 0.6 + log 0.080645 + log 0.258065 + log 0.870968 = −3.465165. Posterior P(spam) = 0.341466, slightly lower (less confident for ham) than Multinomial's 0.368302. Bernoulli uses presence/absence only — it treats “free free meeting” like “free meeting”, losing the repetition that Multinomial counts — but it gains the explicit absence test for “money”, which is rare in spam and common in ham here. For very short texts where counts are noisy, that presence/absence signal is often the more reliable one.
At 0.5: precision = 17/(17+5) = 0.7727, recall = 17/(17+3) = 0.85, F1 = 0.8095, accuracy = (17+15)/40 = 0.80. At 0.75: precision = 13/13 = 1.0, recall = 13/20 = 0.65, F1 = 0.7879, accuracy = 33/40 = 0.825. The higher threshold ships zero false alarms but lets 7 spam through; whether that is better depends on the cost of a false positive (a real email lost in the spam folder) versus a false negative (one spam in the inbox). Accuracy hides all of it: a corpus with 95% ham can be “95% accurate” by predicting ham for everything, with recall 0 on spam — which is why precision, recall and F1 come out together.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 02, Lesson 14) and the Math Foundations Notebook reference build. Interactive figures, the six labs, the worked Bayes' rule and smoothing traces, the Bernoulli-versus-Multinomial table, the Gaussian numeric check, the underflow thresholds and the worked exercise answers are original to this page. Every probability a lab displays is computed from the counts printed next to it.