x · y is one engineered column built from two raw ones. No single raw axis separates these 14 points — their product separates every one.
A house price model with square footage and bedrooms beats one fed a raw address string, no matter how fancy the learner. Raw data hides the pattern; an engineered column exposes it. That is why feature work is the highest-leverage hour in classical ML.
BMI = weight / height²KNN, K-Means, SVMs and every gradient-trained model treat bigger numbers as more important. Standardization subtracts the mean and divides by the standard deviation so each feature gets one vote; min-max squeezes each into [0, 1].
z = (x − μ) / σCategories become columns, holes get a strategy (and often an indicator), interactions let a straight line bend, and selection throws away the columns that only add noise. The one unforgivable bug is a feature that saw the answer.
10 good + 90 noisy < 10 goodFeature engineering changes the question until a simple model can answer it, and feature selection plus a leakage check decide whether the answer is real — everything else is hyperparameter noise.
By the end you will be able to choose a scaling for a distance-based model, encode categories without inventing an order, impute missing values without discarding the missingness signal, build an interaction a linear model can use, rank features with filters, wrappers and embedded methods, and catch the classic ways a dataset leaks the answer.
A house price model with square footage will beat a model fed a raw address string, no matter how sophisticated the learner is. The algorithm can only work with what you hand it.
You pick a dataset, train the obvious model, and the results are mediocre. You try a fancier algorithm, then a week of tuning, and the numbers barely move. Then someone adds two transformed columns and a plain logistic regression beats the tuned ensemble. This happens constantly: in classical ML, the representation of the data is the highest-leverage decision, not the choice of model.
Feature engineering transforms raw data into columns that expose patterns. Feature selection throws away columns that add noise without adding signal. The source lesson frames it as a pipeline — and the ordering matters, because a scaler fitted before a train/test split quietly contaminates everything after it.
The lesson’s housing generator is a linear formula with a little noise. price = 50·sqft + 20,000·bedrooms − 1,000·age + neighborhood bonus + 15,000·has_pool, where neighborhoods pay 50,000 (downtown), 10,000 (suburbs) or 0 (rural). Plug in a 2,000 sqft, three-bedroom, ten-year-old suburban house with a pool:
Now imagine the raw table instead stored address as a string. No linear model can multiply a string by a weight. Replacing it with distance to city center and school rating turns unusable text into two numbers that carry the price signal — that is feature engineering, and it is worth more than a month of hyperparameter sweeps.
A second hand-check for the chapter’s favorite interaction: a 1.70 m, 68 kg person has BMI = 68 / 1.70² = 23.5. Height and weight alone cannot express “large for one’s frame”; the ratio can, and it is one column instead of two.
Square feet live in the thousands, rooms in the single digits. To a distance or a gradient, that imbalance is not a detail — it is a decision about which feature matters, made by accident.
Euclidean distance adds squared differences one axis at a time: two houses that differ by 1,000 square feet contribute 1,000² = 1,000,000; two houses that differ by 4 rooms contribute 4² = 16. The distance is effectively “square feet, with a rounding error from rooms”. Gradient descent inherits the same imbalance: the gradient for a feature is multiplied by that feature’s values, so a column 600× larger demands a 600× smaller learning rate.
Standardization replaces each value with z = (x − μ) / σ — subtract the mean, divide by the standard deviation. The result has mean 0 and standard deviation 1; a z of 2 means “two standard deviations above average”. Min–max scaling uses x′ = (x − min) / (max − min): subtract the smallest value, divide by the range, and every value lands in [0, 1].
The classes are separated mostly by rooms, the small-range feature, so raw distance is 638× more sensitive to square feet. Scale the same 16 rows three ways and watch the neighbourhoods — and the accuracy — change.
The raw panel answers “which house has similar square footage?”; the scaled panels answer “which house is similar overall?”. Standardize when a feature has a Gaussian shape, min–max when you need every value inside [0, 1] and can trust the extremes — and always fit the transform on training rows only.
| Algorithm family | Scale first? | Why |
|---|---|---|
| KNN, K-Means, SVM, PCA | Yes | They measure distance or dot product; units set the weight. |
| Linear / logistic regression, neural nets | Yes | Gradients scale with the feature, so one column sets the step size. |
| Decision trees, random forests, boosting | No | A split on x > 1000 is unchanged by any monotone rescaling. |
| Naive Bayes | Usually no | Each feature contributes through its own probability estimates. |
Take eight measurements: [2, 4, 4, 4, 5, 5, 7, 9]. The mean is 5. The squared distances from it are 9, 1, 1, 1, 0, 0, 4 and 16 — they sum to 32, so the variance is 32/8 = 4 and the standard deviation is 2 exactly. Standardization is then a subtraction and a division per value.
Gradient view. Fit a logistic model on a feature with values around 3,000; each gradient step changes its weight by about lr × 3,000. The other feature, around 4, moves by lr × 4 in the same step. The loss surface is a long narrow valley: gradient descent bounces across the steep wall while crawling along the floor. The formal number is the condition number — the ratio of the largest to smallest curvature of the loss. On the lab’s 8 training rows the raw feature covariance has eigenvalues 1,867,500 and 6.61 (ratio ≈ 282,700); after standardization they are 1.02 and 0.98 (ratio ≈ 1.05). One direction is no longer a thousand times stiffer than the other, so one learning rate serves both.
When min–max beats standardization. If the data has a hard boundary you must preserve — pixel values 0–255, a probability-like score — min–max keeps every value inside [0, 1] and never produces negatives. But a single outlier sets the range for everyone, so min–max is fragile exactly where standardization is merely insensitive. Robust scaling splits the difference: subtract the median and divide by the interquartile range (IQR). On [1, 2, 3, 4, 100] the median is 3 and IQR 2, so the outlier becomes (100 − 3)/2 = 48.5; standardization would have compressed it to (100 − 22) / 39.0 ≈ 2.0, hiding it entirely. If outliers are real, robust scaling keeps them visible; if they are errors, fix the errors first.
A feature has mean 100 and standard deviation 10. What is the standardized value of 120?
Models multiply and compare numbers. A category is not a number yet, and the encoding you choose decides what the model is allowed to believe — including false orders and leaked labels.
One-hot encoding creates one binary column per category: “neighborhood = downtown / suburbs / rural” becomes three columns that hold a single 1 per row. It invents no order, works with any learner, and explodes when a column has thousands of categories. A detail that bites linear models: keep all k columns and they sum to 1 in every row, which makes the design matrix collinear with the intercept — drop one column (the reference category) and keep k − 1. Trees do not care either way.
Label (ordinal) encoding maps each category to an integer: downtown = 0, rural = 1, suburbs = 2. One column instead of k, but the integers now carry a claim: suburbs (2) is twice rural (1), and the midpoint between them — 1.5 — is a neighborhood that does not exist. Trees, which split on individual values, can still use it; a linear model will happily multiply the invented order by a weight.
Target encoding replaces each category with the mean target for that category, smoothed toward the global mean: encoding = w · category_mean + (1 − w) · global_mean, with w = n / (n + smoothing). A category with many rows trusts its own mean; a rare category leans on the global mean. It is compact, handles high cardinality, and it is the most dangerous of the three, because the label is now a feature.
The same 10 houses and their prices, encoded three ways. Watch what each encoding claims about the categories — and how many columns it costs.
| # | neighborhood | price ($k) | downtown | rural | suburbs |
|---|---|---|---|---|---|
| 1 | downtown | 420 | 1 | 0 | 0 |
| 2 | downtown | 380 | 1 | 0 | 0 |
| 3 | suburbs | 300 | 0 | 0 | 1 |
| 4 | suburbs | 350 | 0 | 0 | 1 |
| 5 | downtown | 500 | 1 | 0 | 0 |
| 6 | rural | 310 | 0 | 1 | 0 |
| 7 | rural | 290 | 0 | 1 | 0 |
| 8 | suburbs | 480 | 0 | 0 | 1 |
| 9 | rural | 340 | 0 | 1 | 0 |
| 10 | downtown | 360 | 1 | 0 | 0 |
3 categories → 3 columns, exactly one 1 per row (or zero 1s if you dropped the reference category).
one binary column per category
Target encoding is the only one of the three that uses the label. Compute it inside the training folds and apply the frozen map to validation and test rows, or the model gets a peek at the answers.
The lab’s ten houses have prices 420, 380, 300, 350, 500, 310, 290, 480, 340 and 360 ($k). The global mean is 373.0. With smoothing = 10, a category’s own mean is weighted by n / (n + 10):
Follow the last line: the model is handed 384.5 for a category whose only example is a 500 house. Part of that 384.5 is the row’s own label, because the mean was computed with it included. That is the first leak in this lesson — and the reason target encoding must be computed inside the training folds, never on the full table. In practice: split first, compute the category means on the training part, apply the frozen map to the rest, and prefer K-fold or leave-one-out means inside training so a row never sees its own target.
Counting columns. One-hot with k categories costs k columns (or k − 1 after dropping the reference); label and target encoding cost 1. For a ZIP-code column with 20,000 categories that difference decides whether the design matrix fits in memory — and whether the model can learn anything about a category with four examples.
A missing value is not empty space — it is a decision waiting to be made. Drop it, fill it, or make the missingness itself a feature. Each choice quietly rewrites who is in your dataset.
Drop rows only when holes are rare and random. Every dropped row is data thrown away, and if the missing rows are not a random sample, you have changed the population the model learns from. Mean imputation fills with the average; median imputation fills with the middle value and is more robust when the feature is skewed or has outliers. Categorical columns use the mode, and time series often use forward/backward fill.
The strategy that is easy to forget is the indicator column: add a binary “was this value missing?” feature before filling. If missingness carries signal — the form nobody volunteers, the sensor that fails on hot days — the indicator preserves it while the fill keeps the row usable. Statisticians call the three situations MCAR (missing completely at random), MAR (missing depends on other observed columns) and MNAR (missing depends on the missing value itself). Dropping is only safe in the first case; indicator columns are most valuable in the last.
Three training houses and three test houses have no square footage on file. Pick a strategy and watch what it costs: rows retained, class balance, coverage, and accuracy on the rows the model can answer.
| split | # | sqft | age | premium | truth |
|---|---|---|---|---|---|
| train | 1 | 2400 (imputed) | 42 | no | ? |
| train | 2 | 2400 (imputed) | 47 | no | ? |
| train | 3 | 1100 | 38 | no | — |
| train | 4 | 2400 (imputed) | 45 | no | ? |
| train | 5 | 900 | 50 | no | — |
| train | 6 | 2600 | 12 | yes | — |
| train | 7 | 3400 | 8 | yes | — |
| train | 8 | 2400 (imputed) | 10 | yes | ? |
| train | 9 | 3000 | 15 | yes | — |
| train | 10 | 2200 | 20 | yes | — |
| test | 11 | 2400 (imputed) | 44 | no | — |
| test | 12 | 2400 (imputed) | 41 | no | — |
| test | 13 | 1200 | 39 | no | — |
| test | 14 | 2800 | 11 | yes | — |
| test | 15 | 3100 | 14 | yes | — |
| test | 16 | 2400 (imputed) | 9 | yes | — |
The 4 holes are not random: 3 of them belong to “no premium” houses (75%), versus 2 of the 6 complete rows (33%). Missingness itself is a feature — that is what the was-missing column records.
median fill plus a was-missing column
Every strategy here answers its rows correctly — the interesting damage is elsewhere: dropped rows change who the model sees, and mean vs median disagree (2200 vs 2400) because the present values are skewed low. When missingness carries signal, add the indicator column so the model can use it.
The lab’s six complete square-footage values are 1100, 900, 2600, 3400, 3000 and 2200. The mean is 2200; the median — the middle of 900, 1100, 2200, 2600, 3000, 3400 — is (2200 + 2600)/2 = 2400. The four holes have known true values 1200, 800, 900 and 2800, withheld from the model:
Dropping avoids imputation error entirely but costs coverage: only 6 of 10 training rows survive, the split goes from 5 no / 5 yes (50/50) to 33% / 67%, and 3 of the 6 test houses can no longer be scored at all. On the rows it can answer, the dropped model is right every time — which is exactly how dropping flatters itself. The holes here are also informative: 75% of them belong to “no premium” houses, versus 33% of complete rows. A model that never sees why the value is missing loses that signal; the indicator column keeps it.
A linear model draws straight boundaries — but it does that in whatever space you hand it. Interactions, logs, bins and text counts are ways of turning a curved problem into a straight one.
Interactions multiply features together. Weight and height are individually weak signals for health; BMI = weight / height² combines them into one strongly predictive column. The lab’s 14 points are the purest version: the label is positive exactly when x · y is large, so neither axis separates the classes and their product separates them perfectly. Polynomial features automate the search: for every pair (i, j) add xᵢ·xⱼ, and for every feature add xᵢ². A degree-2 expansion of n features grows to n(n + 3)/2 columns — 5 for 2 features, 9 for 3, and 5,150 for 100.
Log transforms compress right-skewed distributions — income, population, word counts — and turn multiplicative relationships into additive ones. Binning cuts a continuous feature into categories, which lets a linear model draw a step function and lets trees split coarse groups. Text features are counts: a count vectorizer tallies words, and TF-IDF reweights them by how distinctive they are (TF-IDF = TF × IDF, with IDF = ln(N / documents containing the word)).
The label is “premium” exactly when x · y is large. Neither raw axis separates the classes; their product does. Switch the feature and watch the strip below the scatter sort itself.
The product is not magic: it is the “AND” of two signed values turned into one number. Polynomial features, ratios (BMI), and attention’s Q · K all build the same kind of new column.
Polynomial features, source convention. A row [2, 3] keeps its originals, adds the squares, then the cross term:
Log transform. Incomes 10, 20 and 1,000 ($k) become log(1 + x) = 2.40, 3.04, 6.91. The gap between 20 and 1,000 is 50× in raw units but only 2.3× on the log scale — the transform trades raw distance for a fair comparison of orders of magnitude. (Log is undefined at 0, which is why every library ships log1p, log(1 + x).)
Binning. Ages 0.5, 12, 27, 44 and 49.9 into 5 bins over the observed [0, 50] range: width = (50 − 0)/5 = 10, so the bin index is floor(age / 10) capped at 4 → 0, 1, 2, 4, 4. The bin edges are a fitted statistic: compute them on training rows and reuse them, or test values fall into different bins than the model was trained on.
TF-IDF. Three documents: “cat sat mat”, “dog sat mat”, “cat dog run run”. The word sat appears in 2 of 3 documents, so IDF = ln(3/2) = 0.4055; its TF in document 1 is 1/3, giving 0.1352. The word run appears in 1 document, so IDF = ln(3/1) = 1.0986; its TF in document 3 is 2/4 = 0.5, giving 0.5493 — about 4× the weight of sat, from one extra occurrence, because it is the rarer word.
You apply a degree-2 polynomial expansion to 10 numeric features. How many columns come out (the lesson's convention: originals + squares + pairwise products)?
Irrelevant features add noise, training time and opportunities to overfit. Three families of methods disagree about how to rank a column — and their disagreements are the lesson.
Filter methods score each feature on its own, before any model trains. A variance threshold removes columns that barely move. Correlation keeps columns related to the target and drops one of any pair that are near-duplicates of each other. Mutual information measures how much knowing the feature reduces uncertainty about the target — it catches curves that correlation misses. Filters are fast and blind: no feature is ever judged in the company of the others.
Wrapper methods train a model on candidate subsets and score them with cross-validation. Recursive feature elimination trains once, removes the weakest feature, and repeats; forward selection starts with none and adds the best each round. They can detect redundancy and interactions because they measure the model that actually ships — at the price of training many times.
Embedded methods select while training. L1 (Lasso) regularization pushes irrelevant weights to exactly zero; tree ensembles report how much each feature reduced impurity. The choice of model becomes the choice of selector, which is efficient and makes the selection hard to inspect separately from the fit.
Twelve houses, six candidate features, one target. Correlation ranks features in one pass; cross-validation actually tries subsets; the engineered |offset| column is invisible to the first and obvious to the second.
| # | price | sqft_z | sqft_z_copy | review | listing_id | country | offset |
|---|---|---|---|---|---|---|---|
| 1 | 5.86 | 1.20 | 1.19 | 0.80 | -0.94 | 1.00 | -2.00 |
| 2 | 1.93 | 0.90 | 0.91 | -0.60 | -1.16 | 1.00 | 0.90 |
| 3 | 4.28 | 0.60 | 0.59 | 0.50 | 1.17 | 1.00 | 1.50 |
| 4 | 1.93 | 1.00 | 1.02 | -0.40 | 0.86 | 1.00 | 0.60 |
| 5 | 4.76 | 0.70 | 0.68 | 0.30 | 1.06 | 1.00 | 1.80 |
| 6 | 2.77 | 1.10 | 1.09 | -0.20 | -0.84 | 1.00 | -1.00 |
| 7 | 1.97 | -0.90 | -0.91 | -0.80 | -0.83 | 1.00 | -1.90 |
| 8 | 1.48 | -1.20 | -1.18 | 0.60 | -0.79 | 1.00 | -1.10 |
| 9 | 2.39 | -0.60 | -0.62 | -0.50 | 1.94 | 1.00 | -1.60 |
| 10 | 0.74 | -1.00 | -0.98 | 0.40 | -0.41 | 1.00 | 0.70 |
| 11 | 3.11 | -0.70 | -0.71 | -0.30 | -0.57 | 1.00 | -2.10 |
| 12 | 0.78 | -1.10 | -1.09 | 0.20 | 0.51 | 1.00 | -0.80 |
LOO RMSE fits the linear model on 11 rows, predicts the 12th, and repeats — one honest number for every candidate subset, computable by hand on a table this size.
Correlation is a one-feature-at-a-time score: it cannot see redundancy, and a non-monotone column (offset, where both extremes matter) correlates with nothing. The wrapper tries actual subsets and finds it; the engineered |offset| turns a U-shape into a straight line. That is feature engineering before feature selection.
Variance threshold. Column [1, 1, 1, 1] has mean 1, so every deviation is 0 and its variance is 0 — it carries no information and can never help. Column [1, 2, 3, 4] has mean 2.5 and variance (1.5² + 0.5² + 0.5² + 1.5²)/4 = 1.25, so it clears any sane threshold. A constant column is the one case where the answer is certain.
Correlation, worked. For x = [1, 2, 3, 4, 5] and y = [2, 4, 5, 4, 5]: both means are 3 and 4. The covariance is (4 + 0 + 0 + 0 + 2)/5 = 1.2, the standard deviations are √2 ≈ 1.414 and √1.2 ≈ 1.095, so r = 1.2 / (1.414 × 1.095) = 0.7746.
The lab in numbers. Correlation ranks sqft_z (0.635), sqft_z_copy (0.626) and review (0.369) on top, so the filter keeps those three — but sqft_z and its copy are 0.9999 correlated, so a redundancy rule drops the copy and the next candidate in line is listing_id, a random ID with |r| = 0.099. The filter’s three features score 1.568 leave-one-out RMSE. The wrapper tries every 3-feature subset and finds sqft_z + sqft_z_copy + review at 0.751 — better, but it kept the duplicate. Only after engineering |offset| does selection become easy: both filter and wrapper choose |offset| + sqft_z + review at 0.224. Keeping all seven features scores 6.791: with 12 rows and 8 parameters, the extra columns destroy generalization. More features, worse model.
A feature is useless by correlation (r ≈ 0) but a wrapper that tests it in a subset finds it valuable. What is the most likely explanation?
Data leakage is using, during training, information that will not exist at prediction time. It produces the most dangerous kind of model: one whose scores are excellent right up until the moment it matters.
Leakage has a small family of classic shapes. Target leakage: a feature is recorded after the outcome (a “cancellation email” flag, a refund row, an ICU death marker) and agrees with the label because it is the label in disguise. Train/test contamination: the same row (or a near-duplicate) appears on both sides of the split; preprocessing statistics, imputation values, scaling, target encoding or oversampling are computed on the full dataset before splitting; or cross-validation folds are shuffled when rows are not independent (several visits from one patient, several days from one store). Row-ID and time leakage: the row number is an index into a table that was often sorted by time, so neighboring IDs are neighboring events; a random split hands the model near-copies of its own training rows. And in temporal problems, a random split lets the model train on the future to predict the past.
The defense is a question you ask of every column: at the moment this prediction must be made, what would this field contain? If the honest answer is “nothing yet”, “the eventual outcome”, or “it depends on rows in the test set”, the column is leakage — regardless of how good the validation number becomes.
24 customers: 12 train, 4 validation, 8 test. The “cancellation email” column is recorded after the outcome, so it agrees with the churn label. Include it and validation looks perfect; deploy it and the column is blank.
| split | usage | cancellation email | churned |
|---|---|---|---|
| train | 5.4 | yes | yes |
| train | 2.6 | yes | yes |
| train | 5.2 | yes | yes |
| train | 3.0 | yes | yes |
| train | 4.8 | yes | yes |
| train | 2.2 | yes | yes |
| train | 4.8 | no | no |
| train | 4.2 | no | no |
| train | 3.6 | no | no |
| train | 4.4 | no | no |
| train | 3.9 | no | no |
| train | 5.0 | no | no |
| val | 5.0 | no | no |
| val | 3.7 | yes | yes |
| val | 4.6 | no | no |
| val | 2.9 | yes | yes |
| test | 4.5 | no | no |
| test | 3.3 | yes | yes |
| test | 5.1 | no | no |
| test | 2.7 | yes | yes |
| test | 4.9 | no | no |
| test | 3.1 | no | no |
| test | 5.3 | no | no |
| test | 2.4 | yes | yes |
A feature that is “too good” is usually a feature that should not exist yet. Ask what each column would contain at the moment of prediction — if the answer is “nothing”, the column is leakage.
The lab’s 24 customers use usage hours (honest) and a cancellation-email flag (recorded after the outcome). With the flag included, a 3-nearest-neighbour classifier scores 1.00 on validation, because every row’s neighbor on the leak axis is a row with the same outcome. Trained on honest columns only, the same classifier scores 0.50. At deploy the flag is blank, accuracy is 0.50, and the difference — 0.50 accuracy points — is the leakage premium. Nothing about the model changed; a column that cannot exist at prediction time was doing the work.
Detection checklist. (1) Sort every feature by how much it helps; the suspicious ones are usually the strongest. (2) Ask when each column is written and whether it exists before the outcome. (3) Split by entity and time, not randomly, when rows repeat or drift. (4) Fit every transform inside the training fold, ideally inside one pipeline, so no fit ever sees the test set. (5) Shuffle the labels and re-run: a model that still scores well on shuffled labels is reading something it should not.
A churn model reports 98% validation accuracy and 62% in production a month later. What is the most likely explanation?
Answer before you look. The target-encoding question and the correlation question are exactly the ones that separate a memorized definition from a working instinct.
0 / 5 answered · 0 correct
01Why is feature engineering often more impactful than choosing a fancier algorithm?
02What is one-hot encoding?
03What is the data leakage risk with target encoding?
04TF-IDF weights a word by its inverse document frequency. What is the effect?
05You have two features with correlation 0.98. Why might you remove one?
Click a card to swap the lazy description for what it actually means.
Four problems with exact numbers — including one leak to find. Try first; a worked answer is one click away.
Robust scaling computes (x − median) / IQR. For [1, 2, 3, 4, 100]: median = 3, Q1 = 2, Q3 = 4, IQR = 2, so the values become (−2/2, −1/2, 0, 1/2, 97/2) = (−1, −0.5, 0, 0.5, 48.5). Standardization uses mean 22 and std = √1522 ≈ 39.0, giving (−0.54, −0.51, −0.49, −0.46, 2.00). Standardization compresses the outlier ~24× more (2.0 vs 48.5) and squeezes the four ordinary values into a 0.08σ band, so a model barely distinguishes them. Robust scaling keeps both the bulk and the outlier visible. Choose standardization when the feature is roughly Gaussian and outliers are errors you will clean anyway; choose robust scaling when heavy tails are real and you cannot delete them.
Naive encoding for the category is mean([1, 0, 1]) = 0.667, and every row receives that same value — including each row's own label. Leave-one-out encoding for row i is (sum of the other labels) / (n − 1): row 1 gets (0 + 1)/2 = 0.5, row 2 gets (1 + 1)/2 = 1.0, row 3 gets (1 + 0)/2 = 0.5. The row's own label no longer sits inside its feature; in a category with exactly one row, LOO falls back to the global mean because there is nothing left to average. Averaging the effect: naive values are constant at 0.667, LOO values spread 0.5–1.0 and carry only the *other* rows' evidence. In production use K-fold encoding (compute the map on K−1 folds, apply to the held-out fold) — same principle, applied in batches, and it keeps a usable map for unseen rows.
Step 1, variance threshold at 0.01: `country` has variance 0 and is dropped. Step 2, correlation with the target: sqft_z 0.635, sqft_z_copy 0.626, review 0.369, listing_id 0.099, offset 0.036. Step 3, redundancy: sqft_z and sqft_z_copy correlate 0.9999, so the copy is dropped; with a top-3 budget the remaining picks are sqft_z, review and the random listing_id, scoring 1.568 leave-one-out RMSE. The wrapper (best 3 of six by LOO) finds sqft_z + copy + review at 0.751 — better, but it kept the duplicate. Now engineer |offset|: its correlation with price jumps to 0.693 and both filter and wrapper select |offset| + sqft_z + review at 0.224 RMSE. Keeping all seven features scores 6.791 — with 12 rows, the noise columns and the collinear pair overwhelm the model. The order matters: engineer, then select, then validate; selection cannot rescue information that the columns never represented.
The feature counts collections calls after 90 days past due — it is recorded after the outcome it is meant to predict, so it agrees with the default label and the model reads the answer. The signature is the gap: leakage inflates validation (0.99) while the live score reflects the honest signal (0.71). Fix: drop the column and rebuild the feature set from application-time data only, joined with point-in-time correctness (only fields that existed when the application was scored), then re-split by time rather than at random. Expect a realistic 0.78–0.82 AUC. Catch it with a label-shuffle test: shuffle the labels, re-run the pipeline, and a clean model collapses to ≈ 0.50 AUC; if it still scores well, something in the features or splits is leaking. Also audit for duplicate customers across the split and for any transform (scaling, imputation, target encoding) that was fitted before the split.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 02, Lesson 08) and the Math Foundations Notebook reference build. Interactive figures, the six labs, the pipeline figure and worked exercise answers are original to this page. Every score a lab prints is computed live from the table it displays.