p = σ(w·x + b) turns any number into a probability. A threshold turns that probability into a decision, and a loss turns every mistake into a direction to move.
Fit a line to 0/1 labels and you get outputs like −0.05 or 1.25. Neither is a probability, a single far-away point drags the whole line, and a threshold placed on the line moves whenever the slope changes. Classification needs an output bounded to [0, 1].
y = 0.1455·hours − 0.2 → 1.2545 at 10 hCompute the same linear score z = w·x + b, then pass it through σ(z) = 1/(1+e^(−z)). Large positive z approaches 1, large negative z approaches 0, and z = 0 lands exactly at 0.5 — the decision boundary.
σ(0) = 0.5 · σ(2) = 0.8808 · σ(−2) = 0.1192Binary cross-entropy charges −ln(p) for a true 1 and −ln(1−p) for a true 0. A confidently wrong p = 0.1 costs 2.3026 — over three times the 0.6931 of a coin flip, and twenty-two times the 0.1054 of being sure and right. Its gradient simplifies to p − y, so learning never stalls when the model is confidently wrong.
L = −[y·ln p + (1−y)·ln(1−p)]Logistic regression is a linear score plus a sigmoid plus a threshold: z = w·x + b says how strongly the evidence points to class 1, p = σ(z) converts that score into a probability, and training nudges w and b until the labels you observed become likely.
By the end you will be able to compute a probability and its log-odds by hand, explain why MSE stalls and log loss does not, write the gradient update from memory, and choose a threshold instead of accepting 0.5 by default.
Labels are 0 or 1 — a step, not a ramp. Linear regression is built for ramps: it minimizes squared error and can output any number at all.
Take ten students and record hours studied against pass (1) or fail (0). The labels jump from 0 to 1 between hour 4 and hour 5. Least squares fits the best straight line through those points: y = 0.1455·hours − 0.2. At hour 1 it predicts −0.0545; at hour 10 it predicts 1.2545. Negative probability and more-than-certain probability are both nonsense, and thresholding the line at 0.5 does not fix the deeper problem: the 0.5 crossing sits at hour 4.81 only because of the slope, so any change in the data slides the boundary.
Classification needs three things linear regression does not provide: bounded output in (0, 1), a sharp transition from one class to the other, and insensitivity to far-away points. The next two chapters build exactly that, starting from a quantity you already use without naming: odds.
Least squares picks the slope and intercept that minimize the sum of squared vertical errors. With n = 10 students:
Numeric check: at 5 hours the model says 0.14545·5 − 0.2 = 0.5273, and it crosses 0.5 at x = (0.5 + 0.2)/0.14545 = 4.8125 hours. Nothing in the fit prevents the line from leaving [0, 1].
Add a single student who studied 50 hours and passed. The means move — and so does every prediction, even for students nowhere near hour 50.
One observation changed the meaning of the model for every other student. Logistic regression will not be immune to outliers in the features, but a sigmoid keeps the output bounded, so a far point cannot make the predicted probability 1.25 or −0.05.
A linear model trained on pass/fail labels predicts 1.7 for a new student. What is the right interpretation?
Before the S-curve, there is a smaller idea: the same belief can be written as a probability, as odds, or as log-odds — and the model works in the last one.
If a probability is p = 0.8, the odds are p/(1−p) = 0.8/0.2 = 4, which bookmakers write as 4:1. Flip it around: odds of 4:1 mean 4 parts success to 1 part failure, so p = 4/5 = 0.8. The log-odds (also called the logit) take the natural log of the odds: ln(4) = 1.3863.
Log-odds look like a strange choice until you see their shape: a probability squeezes a lot of meaning into the last hundredth near 0 and near 1, while log-odds stretch that whole interval across every real number. Odds of 1 (p = 0.5) give log-odds 0; confidence for class 1 grows positive, confidence for class 0 grows negative, and the two directions are mirror images.
| Probability p | Odds p/(1−p) | Log-odds ln(p/(1−p)) |
|---|---|---|
| 0.10 | 0.1111 | −2.1972 |
| 0.20 | 0.2500 | −1.3863 |
| 0.50 | 1.0000 | 0.0000 |
| 0.80 | 4.0000 | 1.3863 |
| 0.90 | 9.0000 | 2.1972 |
| 0.95 | 19.0000 | 2.9444 |
The sigmoid was not pulled from thin air. It is exactly the inverse of the log-odds function, which is why the model’s raw output z can be read as a log-odds.
Numeric check, exactly. If p = 0.8 then odds = 4 and z = ln 4 = 1.3863. Feed that back: e^(−1.3863) = 1/4, so σ(1.3863) = 1/(1 + 1/4) = 4/5 = 0.8 ✓. Try p = 0.95: odds = 19, z = ln 19 = 2.9444, and 1/(1 + 1/19) = 19/20 = 0.95 ✓.
A model outputs a log-odds of 2.1972 for an email. What probability is that?
The sigmoid takes the linear score z = w·x + b — any number on the logit dial — and bends it into a probability between 0 and 1.
The definition is one line: σ(z) = 1/(1 + e^(−z)). Three values anchor the whole curve: σ(0) = 0.5 (the boundary), σ(2) = 0.8808 (fairly confident class 1), and σ(−2) = 0.1192 (fairly confident class 0). As z grows the output approaches 1 but never reaches it; as z shrinks it approaches 0. The curve is smooth everywhere, which is what makes it trainable.
The sigmoid is also the missing piece from the previous chapter: its output is bounded, so no outlier can produce a probability of 1.25. What an outlier can still do is push z far into the flat tail, where the curve barely responds — a saturation problem we will meet again when gradients vanish.
Weight w sets the steepness, bias b slides the curve left or right, and x₀ is the input you are scoring. The dashed vertical line is the decision boundary, where z = 0 and p = 0.5.
Push w negative and the curve flips: the probability now falls as x grows. Bias only moves the curve — it never changes its shape.
Training needs to know how fast the probability changes when the score moves. Differentiate the quotient, then simplify.
Numeric checks. At z = 0: p = 0.5, so σ′(0) = 0.5·0.5 = 0.25 — the steepest the curve ever gets. At z = 1: p = 0.7311, so σ′(1) = 0.7311·0.2689 = 0.1966. At z = 2: 0.8808·0.1192 = 0.1050. A numerical derivative at z = 1, (σ(1.000001) − σ(0.999999))/0.000002, gives 0.19661 — matching p(1−p) to four decimals.
The shortcut to remember: the slope of the sigmoid at a point is p(1 − p), where p is the height at that point. It peaks at the decision boundary and fades toward zero in both tails.
A probability needs a grade. Binary cross-entropy scores the probability the model gave to the truth — and it grows without bound as that probability shrinks.
The loss for one example is −(y·ln p + (1−y)·ln(1−p)). If the true label is 1 the second term vanishes, leaving −ln p: sure and right costs 0.1054 at p = 0.9, a coin flip costs 0.6931 at p = 0.5, and confidently wrong costs 2.3026 at p = 0.1. If the true label is 0 the formula mirrors: −ln(1−p). The average over every example is the training loss, usually just called log loss or binary cross-entropy.
Drag the confidence slider. The true label is fixed; watch what each loss charges for a wrong answer at probability p.
Near p = 0.5 all three losses agree in spirit. Push p away from the true label and cross-entropy grows without bound, while MSE levels off — and its gradient shrinks exactly when the model is most wrong.
There is a reason this formula and not some other. It is the loss that comes from asking the model to make the observed labels as likely as possible.
Numeric check. For a two-example batch, (y=1, p=0.9) and (y=0, p=0.8): the per-example costs are −ln(0.9) = 0.1054 and −ln(0.8) = 0.2231, so the average is (0.1054 + 0.2231)/2 = 0.1643. Notice the loss never uses ln(0.9) for the second example — it uses the probability the model assigned to the label that actually happened, 0.8.
The instinct is to reuse MSE = (p − y)². Put a sigmoid in the middle and the derivative picks up an extra factor of σ′(z) = p(1−p):
Two failures in one: MSE’s gradient vanishes when the model is confidently wrong (the tail flattens), and its surface is not convex, so there can be local minima and plateaus. BCE is convex in w and b and keeps a large honest gradient where it matters. Teaching simplification: this is the single-example picture; for a full model the same argument applies term by term, and convexity holds because z is linear in the weights.
The true label is 0 and the model predicts p = 0.9. What is the log loss for this example?
Push the log-loss through the sigmoid with the chain rule and the p(1−p) factors cancel. What is left is (p − y), the prediction minus the label.
Training repeats one loop: compute p for every example, average the errors, and step every weight against its gradient. The surprise is how clean the gradients are — dw = (1/n)·Σ(p−y)·x and db = (1/n)·Σ(p−y). These are the same formulas linear regression uses. The only change is that p now comes out of a sigmoid instead of a straight line.
Write the per-example loss as a function of the score z, and differentiate in two steps.
Numeric checks. If p = 0.7311 and y = 1, the push on z is 0.7311 − 1 = −0.2689: lower z a little. If p = 0.1192 and y = 0, the push is 0.1192 − 0 = +0.1192: raise z. Sign and size both make sense — the farther the probability is from the label, the bigger the correction.
Four examples: (x=1, y=0), (x=2, y=0), (x=3, y=1), (x=4, y=1). Step the gradient descent loop and watch every number move. Defaults match the worked table in the chapter.
| # | x | y | z = w·x + b | p = σ(z) | loss | p − y |
|---|---|---|---|---|---|---|
| 1 | 1 | 0 | 0.0000 | 0.5000 | 0.6931 | 0.5000 ✗ |
| 2 | 2 | 0 | 0.0000 | 0.5000 | 0.6931 | 0.5000 ✗ |
| 3 | 3 | 1 | 0.0000 | 0.5000 | 0.6931 | -0.5000 ✓ |
| 4 | 4 | 1 | 0.0000 | 0.5000 | 0.6931 | -0.5000 ✓ |
Loss history, left to right. Each bar is the average binary cross-entropy after one full-batch update.
At step 0 every p = 0.5, so the errors are large and the first step is the biggest. As p approaches the labels, errors shrink and updates get smaller — the same “error × input” rule linear regression uses, with p coming from the sigmoid.
Data: (x, y) = (1, 0), (2, 0), (3, 1), (4, 1). Start at w = 0, b = 0 with learning rate 0.5. Everything below is exact arithmetic — the same numbers the console shows.
| Step | w | b | p for x = 1, 2, 3, 4 | Loss | dw | db |
|---|---|---|---|---|---|---|
| 0 | 0.0000 | 0.0000 | 0.5, 0.5, 0.5, 0.5 | 0.6931 | −0.5000 | 0.0000 |
| 1 | 0.2500 | 0.0000 | 0.5622, 0.6225, 0.6792, 0.7311 | 0.6250 | −0.0578 | 0.1487 |
| 2 | 0.2789 | −0.0744 | 0.5510, 0.6186, 0.6819, 0.7391 | 0.6124 | −0.0525 | 0.1476 |
Read step 0 carefully. With w = b = 0 every prediction is 0.5, so the errors are (0.5, 0.5, −0.5, −0.5). The biases cancel: db = 0. The weights do not: the first two examples push w down, the last two push it up, but the label-1 examples sit farther out, so dw = −0.5 and w becomes 0.25. Loss falls from 0.6931 to 0.6250; by step 2 it is 0.6124, and the errors keep shrinking. That is the whole algorithm.
import math
def sigmoid(z):
z = max(-500, min(500, z))
return 1.0 / (1.0 + math.exp(-z))
class LogisticRegression:
def __init__(self, n_features, learning_rate=0.1):
self.weights = [0.0] * n_features
self.bias = 0.0
self.lr = learning_rate
def predict_proba(self, x):
z = sum(w * xi for w, xi in zip(self.weights, x)) + self.bias
return sigmoid(z)
def compute_loss(self, X, y):
total = 0.0
for xi, yi in zip(X, y):
p = max(1e-15, min(1 - 1e-15, self.predict_proba(xi)))
total += yi * math.log(p) + (1 - yi) * math.log(1 - p)
return -total / len(y)
def fit(self, X, y, epochs=1000):
n = len(y)
for _ in range(epochs):
dw = [0.0] * len(self.weights)
db = 0.0
for xi, yi in zip(X, y):
error = self.predict_proba(xi) - yi
for j in range(len(dw)):
dw[j] += error * xi[j]
db += error
for j in range(len(dw)):
self.weights[j] -= self.lr * (dw[j] / n)
self.bias -= self.lr * (db / n)
return selfThe model always flips class where z = 0 and p = 0.5. Moving the flip point to 0.8 does not retrain the model — it changes which mistake you would rather make.
With two features the decision boundary is the set of points where w₁x₁ + w₂x₂ + b = 0 — for logistic regression, always a straight line. One side has positive z (predict 1), the other negative z (predict 0), and the probability is exactly 0.5 on the line itself. The weight vector (w₁, w₂) is perpendicular to the line and points toward class 1; its length sets how fast confidence grows as you move away.
Drag any point (or use the controls) to move it. Press train and watch full-batch gradient descent rotate the line until the red error rings disappear.
Start from a bad line: the model is right about class 1 and wrong about class 0. Each step nudges w toward the points it misses, and the boundary swings until both clusters sit on the right side.
Because z is linear, the probability depends only on the signed distance from the line, scaled by ‖w‖. A bigger ‖w‖ makes the same geometry more confident — which is exactly what regularization will push back on.
The 0.5 flip point is convention, not law. For a spam filter you might accept spam only above 0.8 to keep false alarms low; for cancer screening you might flag anything above 0.2 so nothing is missed. The four outcomes of every binary decision are the confusion matrix — TP (flagged and true), FP (flagged but false), FN (missed), TN (correctly ignored) — and they give three numbers worth knowing:
Twelve test emails with their spam probabilities. Lower the threshold to catch more spam (recall up, precision down); raise it to flag only the sure things (precision up, recall down).
Sorted left to right by probability. The accent line is the threshold: everything to its left is predicted spam (1), everything to the right is not (0). Correct calls are tinted; mistakes are red.
Watch accuracy stay flat while precision and recall trade places. That is why one number is never enough for an imbalanced problem: the threshold decides which mistake you would rather make.
Probabilities sorted high to low: 0.95, 0.90, 0.85, 0.80, 0.70, 0.65, 0.55, 0.45, 0.40, 0.30, 0.20, 0.10; the true spam labels are 1, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0 — five spam, seven not. Predict 1 when p ≥ t:
| t | TP | FP | FN | TN | Precision | Recall | F1 | Accuracy |
|---|---|---|---|---|---|---|---|---|
| 0.30 | 5 | 5 | 0 | 2 | 0.5000 | 1.0000 | 0.6667 | 0.5833 |
| 0.50 | 5 | 2 | 0 | 5 | 0.7143 | 1.0000 | 0.8333 | 0.8333 |
| 0.80 | 3 | 1 | 2 | 6 | 0.7500 | 0.6000 | 0.6667 | 0.7500 |
| 0.90 | 2 | 0 | 3 | 7 | 1.0000 | 0.4000 | 0.5714 | 0.7500 |
Watch what each move buys. Raising t from 0.5 to 0.8 removes the false alarm at 0.85·(y=0) and the one at 0.70·(y=0), improving precision from 0.7143 to 0.75, but it also drops the true spam at 0.65 and 0.55, cutting recall from 1.0 to 0.6. Accuracy alone would have called t = 0.9 (0.75) worse than t = 0.5 (0.8333) while its precision is perfect — accuracy hides which errors you are making.
You raise the threshold from 0.5 to 0.8. What usually happens?
A perfect split has no best answer: every weight increase makes the probabilities more extreme and the loss a little smaller. A penalty on weight size gives the optimizer somewhere to stop.
Logistic regression never says “confident enough”. If the classes can be separated exactly, doubling every weight pushes every p closer to 0 or 1 and lowers the log loss — the minimum is at infinity. Real data is rarely perfectly separable, but near-separable data still produces huge weights and wildly overconfident probabilities. The cure is to add a penalty to the loss:
Both models train on the same six points with gradient descent. The dashed boundary is unregularized; the solid one adds an L2 penalty and gives up a little training confidence to keep the weights small.
λ = 0 lets the weights grow as long as they keep lowering the loss. Turn λ up and the boundary pivots back toward the middle: training accuracy may dip, but the model stops chasing the last hard point.
L1 behaves differently because the absolute value has a constant slope: at w = 3 the L1 gradient is 0.1 while the L2 gradient is 0.6; at w = 0.1 the L1 gradient is still 0.1 while L2’s has faded to 0.02. L1 keeps pushing small weights all the way to exactly zero(sparse models), while L2 shrinks everything smoothly and never eliminates a feature on its own.
| Penalty | Adds to the loss | Gradient | What you get |
|---|---|---|---|
| L2 (ridge) | λ·Σ wⱼ² | 2λ·wⱼ, proportional to size | Small, smooth weights; every feature kept; the default |
| L1 (lasso) | λ·Σ |wⱼ| | λ·sign(wⱼ), constant size | Many weights become exactly 0; automatic feature selection |
There is a second way a classifier goes quietly wrong. Take 100 examples with 95 negatives and 5 positives. A model that always predicts “negative” scores 95% accuracy while catching zero positives — recall 0, precision undefined, F1 0. Accuracy alone is blind to imbalance. Two standard responses: change the threshold (predict positive more eagerly), or change the cost by weighting the rare class. Giving each positive 19× the weight of a negative is like duplicating the 5 positives until the classes balance, and the optimizer stops ignoring them.
Answer before you look. The loss question and the threshold question are the two that show up in real model reviews.
0 / 5 answered · 0 correct
01What is the range of the sigmoid function's output?
02Why is logistic regression called “regression” even though it classifies?
03Why is binary cross-entropy used instead of MSE for logistic regression?
04A spam filter has precision = 0.95 and recall = 0.60. What does that mean?
05In softmax regression for 4 classes, what is true about the output probabilities?
Click a card to swap the lazy description for what it actually means.
Four short problems, from feature engineering to a hand-computed gradient step. Try first; a worked answer is one click away.
With class 1 inside radius 1.5 and class 0 between radii 3 and 4.5, every line through the plane must cut both rings, so the best raw-feature accuracy is around 50% plus noise. Adding x1², x2², x1·x2 gives the model z = w1x1² + w2x2² + w3x1x2 + w4x1 + w5x2 + b; the boundary is still a line, but in the 5-dimensional feature space it can be the circle x1² + x2² = c for any c in the empty gap between the rings, say c = 6. One weight on (x1² + x2²) plus a bias is enough, so accuracy jumps toward 100%. Moral: logistic regression is linear in the features you hand it — feature engineering changes what is linearly separable.
Treat each class k as one-vs-rest: TPk is the diagonal cell, FNk is the row sum minus the diagonal, FPk is the column sum minus the diagonal. Suppose a richer test set has 20 examples per class and counts [[16, 4, 0], [3, 15, 2], [0, 1, 19]] (rows = actual, columns = predicted). Class 0: precision 16/(16+3) = 0.8421, recall 16/20 = 0.80. Class 1: precision 15/(15+4+1) = 0.75, recall 15/20 = 0.75. Class 2: precision 19/(19+2) = 0.9048, recall 19/20 = 0.95. Class 1 is hardest: its cluster at (5, 1) sits closest to class 0 at (1, 1) along the x-axis, and class 2 at (3, 5) pulls a few borderline points up. Per-class numbers reveal this; overall accuracy (50/60 = 0.8333) hides it.
Sort by score descending: 0.9(+), 0.7(−), 0.6(+), 0.3(−). Sweep thresholds from 1.0 down to 0.0. At ≥1.0: TP=0, FP=0 → (0, 0). At ≥0.9: TP=1, FP=0, and there are 2 positives and 2 negatives → TPR = 1/2 = 0.5, FPR = 0 → (0, 0.5). At ≥0.7: TP=1, FP=1 → (0.5, 0.5). At ≥0.6: TP=2, FP=1 → (0.5, 1.0). At ≥0.3: TP=2, FP=2 → (1, 1). Trapezoids: (0→0.5) width 0.5 × average height 0.5 = 0.25; (0.5→1) width 0.5 × average height 1.0 = 0.5; AUC = 0.75. A perfect ranking (0.95+, 0.85+, 0.2−, 0.1−) gives AUC = 1.0; a coin flip gives 0.5.
Step 0: z = 0 everywhere, so p = 0.5 for all three points and L = −ln(0.5) = 0.6931. Errors p − y are 0.5, −0.5, −0.5. dw = (0.5·0 + (−0.5)·1 + (−0.5)·2)/3 = −1.5/3 = −0.5000; db = (0.5 − 0.5 − 0.5)/3 = −0.1667. Update: w = 0 − 1·(−0.5) = 0.5, b = 0 − 1·(−0.1667) = 0.1667. Step 1: z values are 0.1667, 0.6667, 1.1667, so p = 0.5416, 0.6608, 0.7625 and L = −[ln(0.4584) + ln(0.6608) + ln(0.7625)]/3 = (0.7801 + 0.4143 + 0.2712)/3 = 0.4885. One step cut the loss from 0.6931 to 0.4885.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 02, Lesson 03) and the Math Foundations Notebook reference build. The sigmoid explorer, decision-boundary trainer, loss comparison, threshold sweep, training console and regularization lab, plus the numeric checks and worked exercise answers, are original to this page. Every lab runs in your browser.