Relevance without redundancy. Rank forty candidates, prune the tail, and watch validation rise — then level off. Three families of methods, one rule: select inside the fold.
More columns do not add information — they add volume. Distances stretch, data thins out, and with enough candidates some of them correlate with the target by pure chance. Redundancy is the other half of the problem: two copies of the same feature tell the model exactly one thing, but split the weight and double the variance.
relevance − redundancy = useful columnsFilter methods judge each feature alone with statistics — fast, but blind to interactions. Wrapper methods train a model, rank features, remove and repeat — they see interactions but pay for every round. Embedded methods fold selection into training: L1 drives weights to exactly zero and trees spend their best splits. Same goal, three price points.
statistic · model · penaltyChoosing features on the full dataset uses every label — including the labels you are about to be tested on. The model looks better than it is, and production pays the difference. Run the whole selection procedure inside each training fold, and judge the selected set on data that had no vote in choosing it.
selection is part of the modelFeature selection is modeling the feature set itself: score relevance, subtract redundancy, keep the smallest set that validates — and treat the selector as part of the model, so it never touches the test fold.
By the end you will be able to rank features with variance, correlation and mutual information; run RFE and read a Lasso path; choose a feature count with a validation curve and the one-standard-error rule; stress a selection with stability resampling; and explain why selecting before cross-validation quietly overstates every score.
Two things make a feature worth keeping: it is related to the target, and it is not a copy of a feature you already have. Everything in this lesson is a different way of measuring those two things.
Suppose you have 500 candidate features and 100 labeled rows. The training set fits beautifully — a flexible model can always find something to memorize — and then the model meets new data and collapses. It was not learning a pattern; it was learning which of the 500 columns happened to line up with the labels in those particular 100 rows. Adding features does not add information; it adds opportunities to be fooled.
The first failure is irrelevance: a feature that has nothing to do with the target still has some correlation with it in a finite sample, and a model will happily spend capacity on it. The second is redundancy: two columns that carry the same signal give the model one piece of information but two knobs to turn. The signal gets split between them, and small changes in the data swing the weights wildly — the coefficient for either one alone is nearly meaningless.
In a unit cube with side 0.9, the volume lands below 0.9^d. In 10 dimensions, 0.9^10 = 0.349 — a third of the space sits in the outer shell. In 50 dimensions, 0.9^50 = 0.005: 99.5% of the volume is shell, and “typical” points are all far apart. Data thins out exponentially while the number of columns grows linearly.
The numeric check is the whole first chapter: relevance filters look for the columns whose evidence beats that noise floor, and redundancy filters remove the copies that make the evidence look like it comes from more places than it does.
You add 200 random noise columns to a model trained on 100 rows. Training accuracy jumps to 99%. The test set behaves no better (and quietly worse). What happened?
Filter methods are statistics, not models: score every feature by how much it varies, how linearly it tracks the target, or how much uncertainty about the target it removes. Fast, simple, and blind to teamwork between features.
Variance threshold. If a column barely changes from row to row, it cannot separate anything. Drop features whose variance falls below a threshold — a common default is 0.01. This is the cheapest possible pass, and it never looks at the target at all.
Correlation. Pearson’s r measures how well a straight line explains the pair (feature, target). It is fast and interpretable, and it only sees straight lines.
Chi-squared. For categorical or count features — word counts, flags — compare each observed count with the count expected if the feature and class were independent. Large χ² means the counts lean toward one class.
Mutual information. MI asks a more general question: how many bits of uncertainty about the target does the feature remove? If feature and target are independent, the answer is exactly 0; unlike correlation, a curve or a cycle can score high.
| # | A | B | C | D | E | F | G | H | I | J | K | L | y |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | -0.26 | 0.22 | -0.23 | -0.34 | -0.93 | -0.57 | 0.97 | -0.57 | 2.49 | -1.78 | -0.82 | 0.52 | 0 |
| 2 | -0.68 | -0.16 | 0.03 | -0.55 | -0.53 | 0.65 | -1.21 | -0.31 | 2.50 | -4.92 | 0.69 | -0.30 | 0 |
| 3 | -0.03 | -1.01 | -1.33 | -0.16 | -1.00 | -1.87 | 0.38 | -0.43 | 2.50 | -0.64 | 0.26 | -0.40 | 1 |
| 4 | 1.26 | 1.73 | -1.12 | 1.48 | 0.58 | 1.93 | 1.01 | 0.73 | 2.58 | 0.89 | -0.77 | 1.39 | 0 |
| 5 | 0.83 | -0.42 | 1.30 | 0.70 | -0.31 | 0.23 | 0.69 | -0.09 | 2.47 | -3.99 | 1.11 | -0.95 | 1 |
| 6 | 0.89 | -0.10 | 0.85 | 0.86 | -0.21 | -1.11 | 1.58 | 1.73 | 2.51 | -0.10 | -0.36 | 1.87 | 1 |
| 7 | 0.21 | 0.24 | 1.48 | 0.02 | -0.95 | 0.21 | 0.80 | 0.82 | 2.57 | -2.00 | 0.73 | -0.09 | 1 |
| 8 | -1.61 | 1.00 | -1.08 | -1.81 | 1.60 | 0.23 | 0.27 | 0.70 | 2.55 | 4.78 | -1.09 | 0.18 | 0 |
| 9 | 1.28 | -0.18 | -2.47 | 1.30 | 0.63 | -0.70 | 1.65 | 0.27 | 2.51 | 2.90 | -0.21 | -0.57 | 1 |
| 10 | 0.12 | -0.89 | 0.30 | 0.19 | -0.98 | -0.61 | -1.32 | -0.54 | 2.53 | 0.50 | 1.88 | -0.37 | 1 |
| 11 | 0.18 | -0.81 | -0.28 | 0.31 | -0.97 | -0.25 | 0.89 | 1.17 | 2.56 | 2.46 | -0.33 | 0.41 | 1 |
| 12 | -2.31 | 0.12 | -0.10 | -2.11 | 4.34 | -0.84 | 0.06 | 1.07 | 2.46 | 3.43 | -0.29 | -0.28 | 0 |
| 13 | -0.65 | 1.39 | -2.18 | -0.78 | -0.57 | 1.25 | -0.64 | -0.02 | 2.49 | 1.26 | 1.48 | 1.25 | 0 |
| 14 | -1.69 | 1.92 | -0.69 | -1.76 | 1.85 | 2.10 | 0.51 | 1.00 | 2.54 | 2.76 | -1.31 | 0.58 | 0 |
| 15 | 0.21 | -0.12 | 0.08 | 0.29 | -0.96 | -0.04 | -1.53 | 0.19 | 2.56 | 2.19 | 0.75 | -0.84 | 1 |
| 16 | 2.13 | -0.56 | 0.06 | 2.03 | 3.53 | -0.26 | -1.13 | 0.86 | 2.46 | -0.33 | -0.06 | -0.23 | 1 |
| 17 | -0.15 | 0.01 | 1.01 | -0.20 | -0.98 | 1.26 | 1.72 | 0.31 | 2.51 | -2.26 | -0.92 | 1.78 | 1 |
| 18 | 0.44 | 0.08 | 0.56 | 0.35 | -0.81 | 0.33 | -0.81 | -0.19 | 2.50 | -3.35 | 0.24 | 0.41 | 1 |
| 19 | 0.58 | 1.07 | 0.40 | 0.55 | -0.67 | 0.83 | 0.08 | 2.17 | 2.56 | 2.42 | 1.35 | -0.58 | 0 |
| 20 | 2.19 | -1.34 | -0.39 | 2.15 | 3.81 | -0.83 | -0.38 | 1.66 | 2.45 | 0.00 | -0.37 | -1.26 | 1 |
| 21 | -0.47 | 1.89 | -0.86 | -0.35 | -0.78 | 1.03 | -0.23 | -0.28 | 2.53 | -3.66 | 0.04 | -1.12 | 0 |
| 22 | -1.12 | -0.65 | 0.33 | -1.04 | 0.26 | -1.03 | -0.14 | -0.57 | 2.52 | 7.81 | 0.12 | 1.38 | 0 |
| 23 | -0.82 | 1.07 | 0.31 | -0.69 | -0.32 | 0.79 | -1.91 | 0.33 | 2.57 | -2.28 | 0.96 | 0.29 | 0 |
| 24 | 0.89 | 0.11 | 1.04 | 0.84 | -0.21 | -0.78 | -0.09 | -0.70 | 2.52 | 1.48 | 0.18 | -1.19 | 1 |
Twelve candidates, three cheap statistics, no model. Switch the metric and watch the ranking change — the nonlinear feature falls off the correlation list but stays on the mutual-information list.
A filter never trains a model, so it cannot see that B makes F redundant. That interaction is what wrapper methods pay for.
The lab’s board recomputes these from all 24 rows; here is what the arithmetic says, and what each statistic misses.
The chi-squared check is a 2×2 word-count table over 100 emails: the word “free” appears in 20 of 25 spam and 10 of 75 inbox mails.
Both statistics agree the word is informative — and they disagree on nothing here because a 2×2 table is monotone. The disagreement appears with curves: E would score near zero correlation and a strong MI.
A feature has |correlation| = 0.02 with the target but mutual information of 0.41 bits. Which relationship is most likely?
Wrapper methods treat feature selection as a search problem: train a model, ask it which feature matters least, remove that one, and repeat. It costs a training run per round — and it is the only family that sees interactions.
Recursive Feature Elimination (RFE) is the canonical wrapper:
The reason to remove one at a time is that importances are relative. When two features carry the same signal, each gets roughly half the credit while both are present. Delete one and the other’s importance jumps — the model redistributes. Remove all the low-importance features in one sweep and you can delete a feature that would have become essential the moment its twin left.
Forward selection is RFE’s mirror image: start empty, add the feature that improves validation the most, stop when additions stop helping. It is often faster when the final set is small, and it can find features that only work in combination — at the price of never reconsidering an early choice. Both wrappers cost many training runs: stripping 500 features down to 10 is at least 490 fits, which is why practitioners often remove the bottom 10% per round, or pre-filter with statistics before calling a wrapper.
Each round trains a logistic model, drops the feature with the smallest coefficient, and re-scores. Step through the rounds and watch cross-validation accuracy rise while the tail disappears.
Round 1 removes H (|w| 0.145) — a noise column. Later rounds remove E, whose quadratic information a linear model cannot spend, and keep A B C D. In production the whole loop runs inside every training fold.
The model is logistic regression on the 24 displayed rows, all 12 features standardized. A coefficient of 4.124 means one standard deviation up in that feature moves the log-odds by 4.124.
Every value above is computed live in the stepper: the importances come from the same fits, and the accuracy numbers are 5-fold cross-validation on the displayed rows, not a training score. Training accuracy is 1.000 at every round — the model memorizes 24 rows trivially — which is exactly why the wrapper ranks by validation.
Why remove one feature per round instead of deleting every feature whose importance is below some cutoff?
Embedded methods do not select features as a separate step: the penalty on the weights, or the way a tree spends its splits, decides which features survive while the model is being fit.
L1 / Lasso. Add the absolute values of the weights to the loss:
The plain-English version: every weight now has to pay rent proportional to its size, and the rent is charged linearly — no volume discount. A weight that does not buy enough error reduction to cover its rent is pushed all the way to exactly zero, and a zero weight is a deleted feature. α sets the rent: larger α, fewer survivors.
Why exactly zero? The constraint region created by the absolute-value penalty is a diamond, and its corners lie on the axes. The first point where the loss contours touch the diamond is usually a corner, where one or more coordinates are zero. The L2 penalty (ridge) makes a circle, which has no corners — weights shrink smoothly toward zero but almost never land on it. That geometric difference is the entire reason Lasso selects and ridge does not.
Tree importance. Every split in a decision tree reduces impurity (Gini or entropy for classification, variance for regression). Sum, over all nodes splitting on a feature, the number of samples times the impurity decrease; average over the forest. Features that produce larger, more common reductions score higher — and trees capture curves and interactions automatically.
Permutation importance audits any trained model without touching its internals: record validation performance, shuffle one column, and measure how much the score drops. No drop means the model does not depend on that column. It avoids the tree bias toward high-cardinality columns (an ID column that perfectly tags every row), at the cost of one full evaluation per feature. Correlated columns share credit: shuffle one and the model quietly leans on its twin.
Drag λ down and coefficients leave the path one by one; drag it back up and they snap to exactly zero. The penalty is the sum of absolute weights — no model retraining between steps.
Soft-threshold: a coefficient with partial correlation ρ is set to sign(ρ)·max(|ρ| − λ, 0). At λ = 0.20, ρ = 0.35 shrinks to 0.15 and ρ = 0.12 becomes exactly 0.
Coordinate descent solves the Lasso one weight at a time with the soft-threshold rule — “subtract λ, but never cross zero”:
One Gini check by hand, on a 10-row node with 5 positives: parent impurity 1 − (0.5² + 0.5²) = 0.5. A split sends 6 rows (4 positive, 2 negative) left — 1 − (0.667² + 0.333²) = 0.444 — and 4 rows (1, 3) right — 1 − (0.25² + 0.75²) = 0.375. The gain is 0.5 − (6/10)·0.444 − (4/10)·0.375 = 0.083, and this node credits 0.083 × 10 = 0.83 to the feature it split on. Sum that over every node and tree, normalize, and you have the importance bar chart.
Selection methods rank features; they do not say where to stop. Answer that with a validation curve, and read it through a one-standard-error band instead of chasing the peak.
A validation curve plots model score against a knob — here, the number of features kept. Typical shape: accuracy climbs quickly, flattens once the real signal is in, then drifts down as noise features let the model start memorizing. The peak is a candidate, not a verdict.
The reason is that every point on the curve is an estimate with error bars, and the maximum of noisy estimates is biased upward: some k just got a lucky fold split. The one-standard-error rule cuts through this. Take the best mean, subtract one standard error, and choose the simplest model (fewest features) whose mean is still at or above that bar. You trade a fraction of a percent of measured score for a smaller, cheaper, more stable model — a trade that almost always pays out of sample.
Every k is scored honestly: mutual information ranks features inside each training fold, then logistic regression is tested on the held-out fold. The raw maximum is not the answer — the one-standard-error band is.
The curve peaks at k = 3–4 and falls off past k = 5: the extra columns are noise, and the model spends capacity fitting it. The one-SE rule asks “how much worse is a simpler model than the best, once fold noise is counted?”
The curve is honest in a second sense: the mutual-information ranking is recomputed inside every training fold, so the choice of which single feature to keep at k=1 differs from fold to fold — that is why k=1 is so unstable (0.610). The one-SE bar is not a formality; it is the smallest useful significance test on a curve made of noisy estimates.
Your curve peaks at k=12 with 0.94; the standard error is 0.03 and k=4 scores 0.92. Your colleague wants to ship k=12 because it is 'the best'. What does the one-SE rule say?
A selection that changes when you nudge the data is not a discovery. Stability selection reruns the selector on many resamples and counts how often each feature survives — and then the same discipline has to hold for your cross-validation split.
Stability selection turns a selector into a measurement. Run it — L1, RFE, MI, anything — many times, each time on a random subsample of the rows, with a slightly different hyperparameter. Count how often each feature is selected. A feature that survives 80% of runs is stable; one that appears in one run and vanishes in the next is a fluctuation, not a finding. The idea comes from genomics, where p is enormous, n is small, and reproducibility is the whole game.
The same logic applies to evaluation. Feature selection is part of the model. If the selector sees the test fold’s labels — because it was run once on the full dataset — the cross-validation score is contaminated. The honest protocol runs the entire selection procedure inside each training fold and evaluates the result on rows that had no vote in choosing the features.
Run L1 selection 40 times, each time on a random 80% subsample with a slightly different λ. A feature worth trusting survives most runs; a fluke does not.
Each bar is the fraction of 40 runs in which L1 kept the feature. ⧉ marks D, the copy of A (r = 0.994).
Lower λ admits more features (and more noise); higher λ makes only the strongest survive. A feature that appears at one λ and vanishes at the next is a feature you should not build a story around.
Same features, same folds, two protocols — one picks the top 5 using every label, the other picks inside each training fold. The gap between the lines is the optimism you get for free when the selector cheats.
Every dot is one random 5-fold split. Selection for the orange protocol happens once, on all 24 rows; the teal protocol re-ranks with only the training rows in view.
The naive protocol chose its set once, using all 24 labels. Cross-validating that choice cannot expose the mistake, because the labels that shaped it are inside the test folds. Drag k toward 8: the naive set starts admitting near-constant I and noise L, while the honest bars for those columns stay low. That is why selection belongs in the pipeline, not before it.
The gap is small in absolute terms because this dataset is small and mostly friendly. Its size is not the point: the naive protocol is structurally unable to detect its own mistake, and the gap grows with the number of candidates and shrinks with the number of rows — the exact regime — 500 features, 100 rows — where selection frameworks are usually deployed.
The families are tools, not rivals. A working recipe picks the cheap statistic first, the model-aware method second, and the honest evaluation protocol always.
The source lesson’s decision flow compresses to one idea: filters narrow, embedded or wrapper methods decide, and the validation protocol polices the whole thing. The table is the cheat sheet — use it by asking what your final model is and how many candidates you actually have.
| Method | Family | Speed | Nonlinear | Interactions |
|---|---|---|---|---|
| Variance threshold | filter | very fast | no | no |
| Correlation filter | filter | fast | no | no |
| Mutual information | filter | fast | yes | no |
| RFE / forward selection | wrapper | slow | model-dependent | yes |
| L1 / Lasso | embedded | fast | no | no |
| Tree importance | embedded | medium | yes | yes |
| Permutation importance | audit | slow | yes | yes |
from sklearn.feature_selection import SelectKBest, mutual_info_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
pipe = Pipeline([
("scale", StandardScaler()),
("select", SelectKBest(mutual_info_classif, k=5)),
("model", LogisticRegression(max_iter=2000)),
])
# Every fold re-fits the scaler, re-ranks the features,
# and trains only on that fold's rows.
scores = cross_val_score(pipe, X, y, cv=5, scoring="accuracy")
print(scores.mean(), scores.std())
# Compare against keeping everything, on the SAME folds:
full = Pipeline([("scale", StandardScaler()), ("model", LogisticRegression(max_iter=2000))])
print(cross_val_score(full, X, y, cv=5, scoring="accuracy").mean())
# For L1 selection instead:
# LogisticRegression(penalty="l1", solver="liblinear", C=0.5)
# For tree importance instead:
# from sklearn.ensemble import RandomForestClassifier
# select = SelectFromModel(RandomForestClassifier(n_estimators=300), threshold="median")Answer before you look. The mutual-information question and the leakage question are the two that separate a memorized definition from a working instinct.
0 / 6 answered · 0 correct
01Why can adding more features actually make a model perform worse?
02What is the key difference between filter and wrapper feature selection methods?
03Mutual information can detect relationships that Pearson correlation cannot. What kind?
04L1 (Lasso) regularization performs feature selection as part of training. How?
05RFE removes the least important feature and retrains. Why is this better than just removing all low-importance features at once?
06You rank features by correlation with the target using all 24 rows, keep the top 5, then run 5-fold cross-validation. The score looks excellent. What is the flaw?
Click a card to swap the lazy description for what it actually means.
Four problems with exact numbers — forward selection, stability selection, multicollinearity and a full pipeline. Try first; a worked answer is one click away.
On the lesson's 24 × 12 dataset the entry order is A (cv 0.870), then B (0.910), then E (1.000), then D (1.000 — the only add that keeps the score at 1.000). A fifth addition drops the score: the best candidate is I at 0.950, and every other add is worse, so the procedure stops at {A, B, E, D}. RFE on the same data stops at {A, B, C, D} — it keeps C and drops E, because a linear model's coefficients cannot spend E's quadratic information, while forward selection evaluated E in the presence of A and found the gain. Cost: forward selection evaluated 12 + 11 + 10 + 9 + 8 = 50 candidate sets, each with 5 folds, about 250 fits before the final scoring — RFE needed only 9 full-data fits plus its cross-validation. Forward selection is typically faster when the final set is tiny and the candidate pool is large; RFE is faster when you are cutting many features from a large pool. Neither is uniformly better, and both must run inside the evaluation fold.
The lesson's stability lab gives A 100%, B 100%, C 60%, D 0%, E 25%, F 8%, G 25%, H 0%, I 0%, J 25%, K 3%, L 13% (40 runs, 19-of-24 subsamples, λ between 0.214 and 0.285 of each subsample's λmax). Stable at 80%: A and B — the two strongest linear signals. C sits on the fence at 60%: real enough to appear often, not reliable enough to promise. D is selected 0% of the time: it is A's near-copy (r = 0.994), and whenever the solver can afford one of the pair it pays for A, so redundancy reads as instability. Single-run comparison: at λ = 0.114 the one-shot L1 solution keeps only {A, B}; at smaller λ it admits C and then E, G, J, L — so the single run understates the set at high λ and overstates it at low λ, and gives no way to tell which. That is the argument for resampling: stability is not a property any single run can observe.
The only pair at or above 0.9 is A~D with r = 0.994. Their MI scores tie at 0.420 bits (to three decimals), so the tie-break goes to the higher correlation with the target: A at 0.638 versus D at 0.611 — keep A, drop D. The next-largest pair is B~F at r = 0.789, below the threshold, so both stay; a stricter 0.75 threshold would drop F (MI 0.190) in favour of B (MI 0.420). After removing D the correlation matrix has no pair above 0.9. Two cautions: pairwise filtering only catches two-feature redundancy, not a feature reconstructed from three others (that is what variance inflation factors or a full correlation-matrix rank check are for), and the removal rule must be executed inside the training fold when it feeds a cross-validated pipeline.
Step 1: variance threshold 0.01 removes only I (variance 0.0014), leaving 11 candidates. Step 2: MI ranks the survivors B, A, D, E, C, F, G, J, L, K, H — keeping the top half leaves {B, A, D, E, C, F}. Step 3: RFE on those six eliminates F first, then E, then C, ending at {B, A, D} with 5-fold CV 0.910. RFE alone on all 12 stops at four features {A, B, C, D} with CV 0.880 and needed 9 full-data fits (plus cross-validation at each step); the pipeline needed 3 elimination fits after two statistics that cost no model training. So on this dataset the pipeline is both cheaper and marginally better — because the MI filter removed the obvious noise before the wrapper paid to discover it. The caveat is the filter's blind spot: if a feature were useful only in combination with another, the MI half-filter could drop it before RFE ever sees it. Pre-filtering trades a small risk of lost interactions for a large saving in training runs; on high-dimensional data that trade is almost always worth it, and it must all happen inside the training fold.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 02, Lesson 18) and the Math Foundations Notebook reference build. The displayed 24 × 12 dataset, the six labs (filter board, RFE stepper, Lasso path, validation curve with one-SE rule, stability selection and the leakage experiment), the chi-squared / MI hand check, the noise-ceiling arithmetic, the Gini split check and the rounded filter scores are original to this page. Every score, frequency, coefficient and accuracy shown is computed live from the displayed data.