ŷ = wx + b is the simplest model that learns from data — and its loss, MSE, is the template every training loop follows.
w is the slope — how much y changes when x grows by 1. b is the intercept — the prediction when x = 0. The model is not magic; it is a straight line with two numbers you are allowed to turn.
w = 2.5, b = 1 → x = 4 gives ŷ = 11For each point, take the residual y − ŷ, square it, then average. Large misses dominate because 10² = 100, and the square is smooth, so the gradient exists everywhere. Training means finding the bottom of this bowl.
MSE = (1/n) Σ (ŷ − y)² → 0.48The normal equation θ = (XᵀX)⁻¹Xᵀy jumps to the optimum in one exact step. Gradient descent takes many cheap steps against the gradient. Same bottom of the same bowl — different costs.
one exact leap ↔ ~500 small stepsA model is a guess with knobs, a loss is the score of the guess, and training is turning the knobs to lower the score — linear regression is that whole loop with only two knobs.
By the end you will fit a line from data, compute MSE, RMSE, MAE and R² yourself, derive the closed-form solution, run gradient descent by hand, read a residual plot for the assumptions it breaks, and know when a straight line is the wrong tool.
Linear regression assumes the relationship between input and output is a straight line — and asks you to choose the two numbers that make it miss the data as little as possible.
The model is ŷ = wx + b. Read it out loud: the prediction is the input, stretched by the weight w, then shifted by the bias b. The true value is y; the little hat on ŷ means “this is the model’s guess.” Learning linear regression is finding the w and b that make the guesses least wrong across every training example.
w is the slope: increase x by 1 and the prediction changes by w. If x is hours studied and y is a test score, w = 2.5 means each extra hour predicts a 2.5-point rise. b is the intercept: the prediction when x = 0. In context it can be nonsense (the price of a zero-square-foot house), but it is essential — it lets the line sit at the right height. You never drop b because it is awkward.
Grab the line and pivot it around the data’s centre — or use the sliders. Every squared miss is drawn as a square, and the readout recomputes instantly.
The squares are why MSE hates outliers: doubling a miss quadruples its square. Notice the best fit is the one where the misses stop pulling the line either way.
| Symbol | What it is | In the real world |
|---|---|---|
x | the input (feature) | hours studied, house size, dose… |
y | the real, observed output | the number you would like to predict |
ŷ | the model's prediction | ŷ = wx + b, read “y-hat” |
w | weight / slope | how much y moves per +1 of x |
b | bias / intercept | the prediction when x = 0 |
With several inputs, the same idea becomes ŷ = w₁x₁ + w₂x₂ + … + w_dx_d + b, written compactly as ŷ = wᵀx + b. Each weight says how much its own feature moves the prediction, holding the others fixed; the geometry is no longer a line but a hyperplane. Everything else in this lesson — the loss, the closed form, descent — works unchanged.
A model learns ŷ = 3x + 7. What does the number 7 say?
A model is only as good as its score. Mean squared error takes the vertical miss of every point, squares it, and averages — turning a whole scatter plot into one number you can minimize.
For one point, the residual is y − ŷ: how far the actual value sits above or below the line. Some residuals are positive, some negative, so we square them before averaging — otherwise they would cancel and a terrible line could score zero. The cost is MSE = (1/n) · Σ(ŷ − y)². Notice the order inside the square: (ŷ − y)² equals (y − ŷ)², so either convention gives the same number; only the sign of the unsquared residual depends on the order.
Why square instead of taking absolute values? Two reasons from the source. First, big misses hurt disproportionately: an error of 10 contributes 100 to the sum, not 10 — one bad prediction can dominate the score. Second, the square is smooth and differentiable everywhere, so its derivative exists at every point and optimization has a clear downhill direction. There is a price: MSE is measured in y², and it is very sensitive to outliers. That trade-off is a decision, not an accident — RMSE and MAE, later in this lesson, are the other sides of it.
MSE as a function of w and b is a quadratic: expand the square and every term is either constant or a square of a linear expression. A sum of squares is never negative and curves upward, so the surface is a bowl — a convex paraboloid. A bowl has a single bottom, which means gradient descent cannot get trapped in a false valley: any downhill path ends at the global minimum. Most models you meet later are not this friendly, which is why linear regression is the right place to learn the training loop.
Every point on this plane is a candidate line; the rings are equal-MSE contours and the dot marks the minimum. Click anywhere to drop a start, then watch gradient descent slide to the bottom.
Try lr = 0.005 (slow crawl), 0.03 (steady), then 0.08 (fast zigzag). The bowl shape is the same at every scale — only the step size changes how you travel it.
Model A misses by exactly 1 on all 10 points. Model B misses by 2 on 5 points and by 0 on the other 5. Both have MAE = 1. Which has the larger MSE?
Because MSE is a bowl, its bottom is exactly where both partial derivatives vanish. Solving those two equations gives a formula for w and b — no iteration, no learning rate, one leap.
A minimum of a smooth bowl has a flat tangent: the derivative is zero in every direction. So take the derivative of MSE with respect to b, set it to zero, and do the same for w. Two equations, two unknowns. The algebra (open the derivation below) collapses into a formula you can compute with a calculator.
Notice the two examples give the same recipe with different numbers: compute two means, two sums, one division. That is the entire training loop for a one-feature model, done by hand.
The dashed line is the exact least-squares answer, computed in one step. The moving line is descent walking downhill from a bad start. Watch the MSE curve fall as it chases the optimum.
Push lr toward 0.09 — past this bowl’s stability edge — and watch the loss climb instead of fall: the steps overshoot and the numbers explode. Shrink it to 0.005 and the curve crawls. The closed form never has this dial — but it pays O(d³) in features instead.
| Normal equation | Gradient descent | |
|---|---|---|
| Hyperparameters | none | learning rate, step count |
| Cost | one O(n·d² + d³) solve | O(n·d) per step, repeated |
| Answer | exact (up to floating point) | approximate, approaches the exact one |
| Large feature count d | fails — d³ inversion is brutal | fine |
| Huge row count n | builds all of XᵀX anyway | mini-batches keep memory small |
| Collinear features | XᵀX is singular; no inverse | still limps; regularization helps |
When a formula is out of reach, calculus still tells you which way is downhill. Gradient descent turns that direction into a simple loop — and the loop scales to models with billions of knobs.
The gradient of MSE answers two questions about the point where you currently stand: which way is uphill, and how steep is it. Training does the obvious thing: step the opposite way, by an amount proportional to the slope. For ŷ = wx + b with MSE, the two partial derivatives are almost embarrassingly simple — each one is an average of the errors, weighted by x for the slope:
Every point on this plane is a candidate line; the rings are equal-MSE contours and the dot marks the minimum. Click anywhere to drop a start, then watch gradient descent slide to the bottom.
Try lr = 0.005 (slow crawl), 0.03 (steady), then 0.08 (fast zigzag). The bowl shape is the same at every scale — only the step size changes how you travel it.
| Variant | Gradient from | Updates | Strength | Weakness |
|---|---|---|---|---|
| Batch | all n points | 1 update / epoch | smoothest, most stable | slow; memory grows with n |
| Stochastic (SGD) | 1 point | n updates / epoch | fast early progress, escapes shallow traps | noisy; can bounce around the minimum |
| Mini-batch | 32–256 points | n/batch updates / epoch | vectorizes well on GPUs; the default | one more dial to tune |
Two models can share an R² and fail completely differently. Plotting what the model missed — against x — is how you hear the data complain before you trust the number.
The residual of a point is y − ŷ: the vertical distance between the data and the line. In a strict statistical sense the “error” is the unobservable noise, and the residual is its observed stand-in — for this lesson the distinction matters less than the habit: plot the residuals against x and look at the shape. A model that has captured everything it can leaves residuals with no remaining pattern: roughly constant spread, centered on zero, no bends, no loners.
Any visible shape is a message. A curve means the trend bends and a straight line cannot follow. A fan means the model’s certainty should vary with x. A single distant residual means one row has more influence than the rest. None of these show up in a single summary number — which is exactly why the plot is not optional.
The left panel shows the data and the least-squares line; the right panel plots every residual against x. Switch datasets and read the shape: random noise, a curve, a funnel, or one point owning the fit.
A good fit does not mean a good model: the clean line and the fan both score high R², but only one has residuals you can trust to be independent. Look at the shape, not just the score.
| What the residual plot shows | What it means | What to do |
|---|---|---|
| Random cloud around zero | the model captured the trend; nothing structured is left | nothing — keep it |
| Curve (U or arch) | the true relationship bends; a line is the wrong shape | add x², or transform x/y |
| Fan (spread grows with x) | the variance is not constant (heteroscedasticity) | weighted least squares, log y, or report prediction intervals |
| One isolated large residual | an outlier or a high-leverage point owns part of the fit | check the row; use a robust loss if the outlier is real |
A residual plot shows a clear U shape: residuals are negative at both ends of the x-range and positive in the middle. What is the diagnosis?
MSE is what training minimizes. RMSE, MAE and R² are what people report — and they disagree in exactly the situations that matter. Knowing which question each one asks is the skill.
All four metrics start from the same residuals, but weigh them differently. MSE squares and averages: the training objective. RMSE = √MSE undoes the squaring, so it is back in the units of y — “the typical miss, with big misses getting extra weight.” MAE averages absolute misses: the plain-language “average error,” indifferent to whether a miss is small or huge. R² divides your squared error by the squared error of predicting ȳ, and turns the comparison into a unit-free fraction: how much of the target’s variation your model explains.
For a model with an intercept, the total squared deviation of y from its mean splits exactly into two piles: what the model explains and what it misses.
The decomposition needs the intercept. If you force the line through the origin, the budget stops adding up and R² can look terrible for a model that predicts fine. Another reason not to drop b casually.
Move the line and watch four scores disagree. Then switch on the outlier: one distant point can own MSE while MAE barely flinches. Every value is computed from the listed points.
| Metric | Value | Units | What it punishes |
|---|---|---|---|
| MSE | 0.800 | y² | Squares every miss — big misses dominate, outliers explode it. |
| RMSE | 0.894 | y | √MSE, back in the target's units — the typical-sized miss. |
| MAE | 0.800 | y | Every miss counts linearly; robust to a single outlier. |
| R² | 0.333 | — | 1 − SS_res/SS_tot: the share of variance your line explains. |
| baseline MSE | 1.200 | y² | Predicting ȳ = 4.00 every time — the R² = 0 reference. |
MSE is the squared scale, RMSE and MAE are in y-units, R² is unit-free. Never compare MSE across datasets with different y scales — compare R² or RMSE.
Linear regression predicts whenever the trend is roughly linear — but its confidence intervals and coefficients only mean what the textbook says when a short list of assumptions holds.
As a prediction machine, linear regression is happy with any roughly linear trend. As a statistical model — one whose coefficient estimates and confidence intervals you quote — it assumes more:
When the relationship bends, the fix is not a fancier optimizer — it is a better set of features. Polynomial regression is the smallest such fix: feed the model x, x², x³, … and it is still linear regression, because the model is linear in the weights. The curve below is the whole trade-off in one dial.
A polynomial is still linear regression — you just feed it x, x², x³ … as features. Turn the degree up: training fit always improves, but held-out points eventually get worse.
Degree 3 matches the true curve; degree 10 threads every training point and dives between them. More capacity is not more skill — it is more rope.
With multiple features, one practical failure is scale: if size runs 0–3,000 and bedrooms run 1–5, the MSE bowl is an elongated canyon and descent zigzags. Standardize features first (subtract mean, divide by standard deviation). Then the learned weights are comparable across features — a big |w| means that feature moves the prediction a lot per standard deviation, not just per raw unit.
The two listings below are the same model twice: once written out as the training loop, and once as the production call. Run both on the same data and the coefficients agree to several decimal places — the library is not doing different mathematics, it is doing the same mathematics with better numerical care and more edge-case handling. Use scikit-learn in production; write the loop once yourself to understand what it is doing.
import random
TRUE_W, TRUE_B = 3.0, 7.0
random.seed(42)
X = [random.uniform(0, 10) for _ in range(100)]
y = [TRUE_W * x + TRUE_B + random.gauss(0, 2.0) for x in X]
class LinearRegression:
def __init__(self, learning_rate=0.01):
self.w, self.b, self.lr = 0.0, 0.0, learning_rate
def predict(self, X):
return [self.w * x + self.b for x in X]
def compute_cost(self, X, y):
preds = self.predict(X)
return sum((p - t) ** 2 for p, t in zip(preds, y)) / len(y)
def compute_gradients(self, X, y):
preds = self.predict(X)
n = len(y)
dw = (2 / n) * sum((p - t) * x for p, t, x in zip(preds, y, X))
db = (2 / n) * sum(p - t for p, t in zip(preds, y))
return dw, db
def fit(self, X, y, epochs=1000):
for _ in range(epochs):
dw, db = self.compute_gradients(X, y)
self.w -= self.lr * dw
self.b -= self.lr * db
return self
def r_squared(self, X, y):
preds = self.predict(X)
y_mean = sum(y) / len(y)
ss_res = sum((t - p) ** 2 for t, p in zip(y, preds))
ss_tot = sum((t - y_mean) ** 2 for t in y)
return 1 - ss_res / ss_tot
model = LinearRegression(learning_rate=0.005).fit(X, y)
print(f"learned y = {model.w:.4f}x + {model.b:.4f}") # ≈ 3x + 7
print(f"R^2 = {model.r_squared(X, y):.4f}")from sklearn.linear_model import LinearRegression, Ridge
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error, r2_score
import numpy as np
rng = np.random.default_rng(42)
X = rng.uniform(0, 10, (100, 1))
y = 3.0 * X.squeeze() + 7.0 + rng.normal(0, 2.0, 100)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42
)
lr = LinearRegression().fit(X_train, y_train)
print(f"w = {lr.coef_[0]:.4f}, b = {lr.intercept_:.4f}") # close to 3, 7
print(f"test R^2 = {r2_score(y_test, lr.predict(X_test)):.4f}")
print(f"test MSE = {mean_squared_error(y_test, lr.predict(X_test)):.4f}")
# scaling matters for descent (sklearn's closed form already handles it)
scaled = StandardScaler().fit_transform(X_train)
ridge = Ridge(alpha=1.0).fit(scaled, y_train)
print(f"ridge coef: {ridge.coef_[0]:.4f} (shrunk toward zero)")The residuals get wider as x grows: a funnel opening to the right. Which assumption is being violated?
Answer before you look. The learning-rate, R² and overfitting questions are exactly the ones that decide whether a real training run succeeds.
0 / 6 answered · 0 correct
01What does the learning rate control in gradient descent?
02What does R² = 0 mean for a regression model?
03Why is feature scaling important for gradient descent in multiple linear regression?
04The normal equation gives optimal weights directly. Why would you prefer gradient descent instead?
05A degree-10 polynomial regression fits training data perfectly (R² = 1.0) but scores R² = 0.3 on test data. What should you do?
06Why does MSE square the errors instead of adding their absolute values?
Click a card to swap the lazy description for what it actually means.
Three problems with fully worked answers — the cubic one has every number you need to reproduce it.
Start MSE(0, 0) = mean(y²) = 17.2, and the true optimum is w = 0.6, b = 2.2 with MSE = 0.48. Batch takes the gradient over all five points: dw = (2/5)·Σ(ŷ−y)x = (2/5)(−66) = −26.4 and db = (2/5)(−20) = −8, so the first step lands at (0.264, 0.08) and MSE drops to 10.49. SGD uses one point at a time: if the first point happens to be (5, 5), the gradient is (2·(−5)·5, 2·(−5)) = (−50, −10) and the step jumps to (0.5, 0.1) — a big, noisy correction; starting with (1, 2) instead gives only (0.04, 0.04). Mini-batch of the first two points: gradients (−10, −6) → (0.1, 0.06). So per-update, SGD moves fastest (it sees 5 updates per epoch) but its cost curve zigzags; batch is the smoothest but pays for a full pass per update; mini-batch (32–256 examples in practice) is the usual compromise because it vectorizes well on GPUs. Run each to convergence and compare cost curves, not single steps: with lr = 0.05, batch descent reaches (0.600, 2.200) in a few hundred steps.
Train on these eleven noisy points: (−2.40, −9.94), (−1.92, −6.18), (−1.44, −1.64), (−0.96, 1.04), (−0.48, 1.04), (0.00, 2.60), (0.48, 1.25), (0.96, 2.21), (1.44, 2.54), (1.92, 1.60), (2.40, 3.37). Hold out seven test points from the same curve, e.g. (−2.1, −6.81), (−1.3, −1.62), (−0.5, 1.75), (0.3, 1.72), (1.1, 2.27), (1.9, 1.78), (2.35, 3.09). Scale x to [0, 1] before fitting (numerical stability), then solve the normal equations for each degree. Results: degree 1 → train R² = 0.66, test R² = 0.66 (underfit: a line cannot bend); degree 3 → train R² = 0.98, test R² = 0.99 (the true shape, noise aside); degree 10 → train R² = 0.9999, test R² ≈ −0.6 (negative: worse than predicting the mean). Overfitting is already obvious at degree 10: training error keeps falling while test performance collapses to worse-than-mean. The pattern, not the exact decimals, is the lesson — with 11 points, a degree-10 curve has enough knobs to wiggle through every training point, and the wiggle is noise.
Take one weight with no data pressure (∂MSE/∂w = 0), start at w = 0.3, lr = 0.1 and α = 1. Ridge subtracts 2αw each step: w ← w(1 − 0.2), so 0.3 → 0.24 → 0.192 → 0.154 → … — a geometric decay that approaches zero but never reaches it. Lasso applies the soft-threshold operator: shrink by lr·α = 0.1 and clamp at zero, so 0.3 → 0.2 → 0.1 → 0 exactly, and it stays there. With real data gradients the same thing happens: whenever a feature's gradient is smaller than α, L1's threshold snaps its weight to exactly zero, producing a sparse model that ignores features; L2's derivative 2αw vanishes as w approaches zero, so weak weights become tiny but stay alive. That is why L1 selects features and L2 just stabilizes them. (Code detail: use soft-thresholding or coordinate descent for L1; a naive subgradient step oscillates around zero instead of landing on it.)
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 02, Lesson 02) and the Math Foundations Notebook reference build. The loss-surface explorer, the closed-form-vs-descent race, the residual diagnostics lab, the metric console, the polynomial playground, every numeric check, and the worked exercise answers are original to this page. Every lab runs in your browser.