Eleven straight cuts each get the wavy boundary right only in one strip — yet their majority vote tracks the curve almost perfectly. Ensembles are the most reliable trick in classical ML, and the reason is arithmetic, not vibes.
Five members at 60% each vote to 68%; twenty-one vote to 83%; a hundred and one reach 98%. The arithmetic only works because a wrong answer has to fool more than half the crowd — and because the members fail on different rows.
21 members × 60% → 82.6%Bagging trains independent models in parallel on bootstrap samples and averages away variance. Boosting trains models in sequence, each focused on what the ensemble still gets wrong, and drives down bias. Stacking feeds base-model predictions to a meta-learner that decides how to combine them.
parallel · sequential · learnedIf every member errs on the same rows, the vote repeats the error. Correlation is the tax: with ten members and pairwise error correlation 0.1, variance falls from 1.0 to 0.19; at correlation 0.5 it only reaches 0.55. Random forests buy decorrelation with bootstrap rows and random feature subsets.
ρ = 0.1, T = 10 → 0.19An ensemble converts many imperfect opinions into one better answer by exploiting the part of their errors that is independent: bagging averages it away, boosting corrects it step by step, and stacking learns which opinion to trust.
By the end you will be able to compute a majority-vote accuracy, tell hard from soft voting, explain bagging and pasting, walk through AdaBoost’s weights by hand, describe gradient boosting as residual-fitting, sketch a stacking pipeline without leaking the labels, and decide when XGBoost is worth its cost.
A single tree overfits; a single straight line underfits. Instead of hunting for one perfect model, combine several imperfect ones and let their independent mistakes cancel.
Suppose you have N independent classifiers, each right with probability p > 0.5. Let them vote. The vote is wrong only when more than half of them are wrong at once, so the accuracy of the crowd is an exact sum over the ways a majority can be correct:
The requirement is diversity. If every member makes the same errors, the vote just repeats them. Ensembles manufacture diversity four ways: different training subsets (bagging), different feature subsets (random forests), sequential error correction (boosting), and different model families (stacking). The next four chapters take them one at a time.
Twenty-one members each score about 60% on 100 test points. The slider makes them fail on the same points (correlated) or on different points (independent). Everything shown is measured from the simulated members — this is a simplified teaching model of error correlation, not a trained ensemble.
Same members, same individual accuracy — only the overlap of their mistakes changes. Independence is the whole product; a committee of clones is one person with extra paperwork.
Plug in p = 0.6 and increasing N. Each row sums the binomial terms k > N/2:
The original lesson quotes “about 74%” and “about 84%” for these two cases. Those figures correspond to members around 57% and 55% accurate, not 60% — the exact sum above is the honest version at p = 0.6. The lesson’s qualitative point stands and is even stronger: at a fixed member accuracy, more members help fast, then with sharply diminishing returns.
For a bagged average of T members, each with variance σ² and pairwise error correlation ρ, a standard decomposition gives:
Now put it in a full error budget. A single member with bias² = 0.16, variance = 1.00 and irreducible noise = 0.04 has MSE 0.16 + 1.00 + 0.04 = 1.20. Ten decorrelated members (ρ = 0.1) leave the bias and noise alone but replace the variance term: 0.16 + 0.19 + 0.04 = 0.39. Averaging attacks exactly one of the three terms — which is why bagging helps a jumpy model and does nothing for a systematically wrong one.
Five members each score 60%, but they always make the same mistakes together. What does majority voting score?
The simplest ensemble changes nothing about how the members are trained — it only changes how their answers are combined. There are exactly two standard ways, and they can disagree.
Hard voting asks every member for a label and takes the majority: five members say A, three say B, the answer is A. Soft voting asks for probabilities, averages them, and takes the larger average. For regression there is no vote at all — you simply average the predicted numbers. Soft voting is usually the better default because a member that is 90% sure should outweigh one that is 51% sure, and trained classifiers can supply those probabilities.
Both votes are defensible; they encode different assumptions. Hard voting is right when the members are peers and confidences are unreliable. Soft voting is right when the probabilities are calibrated — and it breaks when they are not, which is the trap at the bottom of this chapter.
Each numbered dashed line is one weak member: a straight cut fitted to a small window of the curve, right only near that window. Switch members on and off, or flip between the majority vote and the average confidence, and watch the shaded region morph. Every accuracy is measured on the 425 grid points drawn.
The vote is not a democracy of equals — it is a median. Straight cuts from many places cross each other; where they cross, a member’s opinion flips. Adding a member can even hurt for a moment: diversity matters more than count. The members here are hand-placed teaching models, not trained classifiers.
Three members give P(class A) = 0.90, 0.40, 0.40. What do hard and soft voting conclude?
Bagging attacks variance by training the identical algorithm many times on different slices of the data. Random forests, met in the trees lesson, are this recipe plus one extra twist.
Bagging (bootstrap aggregating) draws T bootstrap samples — each the same size as the original, drawn with replacement — trains one model per sample, and combines their predictions by vote or average. The models never interact and can be trained in parallel. The general recipe is only four lines:
Pasting is the same scheme with sampling without replacement. It is useful when the dataset is large and bootstrap duplicates waste training time; bagging’s duplicates are what break up correlated structure in small datasets, so pasting is the large-data variant, not the more accurate one.
Random forests add a second randomization: at each split only a random subset of features is considered (≈ √p for classification). Lesson 04 builds them; here the point is that feature subsampling lowers the error correlation ρ directly, which the variance formula rewards.
Boosting trains members in sequence, and each one is handed a reweighted dataset in which the rows the ensemble still gets wrong matter more. AdaBoost is the original, and its arithmetic fits on one line.
AdaBoost works with any base learner; the source uses decision stumps — a single split, two leaves. Every training row starts with weight 1/N. Each round fits a stump that minimizes weighted error, gives it a vote weight α, then multiplies the weights: rows it got right shrink, rows it got wrong grow. The next stump is forced to care about what the ensemble still misses.
Twenty-two points, three blocks of +1 labels. Fit stumps one round at a time: each cut is chosen against the current weights, its vote weight α = ½·ln((1−err)/err), and misclassified rows get heavier. Watch the red circles — the ensemble’s mistakes — move around.
This is the real algorithm on a tiny dataset: weighted error, α, weight update, normalize. The numbers in the readout are the same ones the chapter’s worked example computes by hand.
Take the lab’s 22 points, x = 1…22, with +1 labels in three blocks (x ≤ 4, 10–13, 17–19). All weights start at 1/22 ≈ 0.04545. Round 1’s best stump is x ≤ 4.5 → +1: it covers the left block and misses the other seven positives.
Round 2 picks x ≤ 19.5 → +1 against the new weights: it gets every positive block right, but mislabels the negatives at x = 5–9 and 14–16, eight rows at 1/30 each, so err₂ = 8/30 = 0.26667 and α₂ = ½·ln(0.73333/0.26667) = ½·ln 2.75 ≈ 0.50583. The raw accuracy actually dips after round 2 (14/22 = 63.6%) before later rounds repair it; the ensemble reaches 22/22 = 100% at round 6. The lab’s readout shows this exact sequence — a good reminder that a boosting curve is not monotone.
A base learner has weighted error exactly 0.5. What vote weight α does AdaBoost give it?
Gradient boosting is AdaBoost’s generalization: instead of reweighting rows, each new model is fitted to the negative gradient of the loss — for squared error, literally to the residuals.
Start with a constant prediction, usually the mean of y. Then repeat: compute how wrong the current ensemble is on every row, fit a small tree to that error, and add a fraction of its output to the prediction. The fraction is the learning rate, and it is the same shrinkage idea as in gradient descent — smaller steps, more trees, usually better generalization.
Start from the mean of y. Each round fits a two-leaf tree to the current residuals (the orange sticks), then steps a fraction of the way — the learning rate. Add trees and watch the staircase bend toward the wave and the sticks shrink.
Each tree is a single split with two constant leaves — a crude step function that cannot bend twice on its own. The sequence is the model; the learning rate decides how much of each fix to trust.
Four points with y = [10, 20, 30, 40] and a learning rate of 0.5. Start at the mean:
The same mechanism explains the learning-rate trade-off in the lab. After one tree the toy dataset’s MSE is 2.483 at lr = 0.1 and 1.201 at lr = 0.5; after ten trees it is 0.847 versus 0.208. The small rate lags early, but with enough trees it can catch up while making gentler corrections that usually generalize better. Typical production values run 0.01–0.3.
AdaBoost can only reweight examples; gradient boosting can optimize any differentiable loss, because the pseudo-residual is defined for every loss. For log loss, the negative gradient pushes predicted probabilities toward the labels; for ranking losses, it pushes pairs the right way. Each new tree is one step of gradient descent, but the step direction lives in function space — the tree itself is the step.
Voting and averaging fix the combination rule in advance. Stacking learns it: the base models’ predictions become features for a small meta-learner that decides how to combine them.
Train a handful of diverse base models — a random forest, a logistic regression, a KNN. Then build a new dataset whose features are the base models’ predictions and whose label is the original target, and train a meta-learner on it. The meta-learner does not see the raw features; it learns which base to trust, and in what proportion, from how each base has been doing.
The one rule that keeps stacking honest: the meta-features must be out-of-fold. Generate them with k-fold cross-validation — train bases on k−1 folds, predict the held-out fold, repeat — so that no row’s meta-features were produced by a model that memorized that row’s label. In-sample predictions would let the meta-learner read the answers.
The target is a diagonal band: +1 inside, −1 outside. B1 and B2 each cut one edge of the band (≈69% alone); B3 is near chance. The meta-learner is ridge least squares on the base scores — turn the bases on and off and watch which weights survive. All weights and accuracies are fitted on the 49 displayed points.
With B1 and B2 on, the meta puts equal positive weight on both and supplies a negative bias: its sum is positive only when both cuts say inside. That is an AND gate built from two weak models — the composition none of them could learn alone. B3 gets weight 0.00: the meta learned to ignore it.
The lab’s 49 points form a diagonal band: +1 when |x − y| < 0.45. B1 predicts + when x − y < 0.45 (69.4% alone) and B2 predicts + when x − y > −0.45 (69.4%). B3 is a near-chance horizontal cut (51.0%). Ridge least squares on their ±1 outputs finds:
That weighted sum is a logical AND: both cuts must say “inside” for the stack to say +1. This is the real reason stacking can beat every member — it composes boundaries that no single base model can represent. It also explains the cost: one more layer to train, validate and debug, which is why the source says stacking is for the last 1–2% of accuracy.
You build meta-features by predicting the training set with base models that were trained on that same data. What happens?
The three families attack different parts of the error. Pick by the problem, not by fashion — and remember that every extra member is extra training time, latency and code to maintain.
Bagging lowers variance without raising bias much; boosting lowers bias and can overfit; stacking combines both strengths at the cost of a second training layer. The table is the source’s, kept almost verbatim because it is the most reused artifact in the lesson.
| Method | Reduces | Best for | Watch out for |
|---|---|---|---|
| Bagging / random forest | Variance | Noisy data, many features, deep trees | Does not help with bias |
| AdaBoost | Bias | Clean data, weak stumps | Sensitive to outliers and label noise |
| Gradient boosting | Bias | Tabular data, competitions | Slow to train, easy to overfit untuned |
| XGBoost / LightGBM | Both (bias mainly) | Production tabular ML | Many hyperparameters |
| Stacking | Both | Squeezing out the last 1–2% | Complex; meta-learner can leak or overfit |
| Voting | Variance | A quick blend of diverse models | Only helps if the models are diverse |
For independent voters, the majority accuracy is an exact binomial sum. Slide the member accuracy and the committee size: gains come fast at first, then flatten, and the whole thing depends on members being better than a coin flip.
This is the independent-voter formula, not a claim about any particular dataset. Real members share data and architecture, so their errors correlate and the curve saturates lower — that is why the diversity lab matters.
XGBoost is gradient boosting plus a regularized objective and a second-order approximation. For one leaf, the best constant output has a closed form:
The same G and H drive split selection, so the algorithm chooses splits that reduce loss after regularization, not raw error. LightGBM keeps the math and changes the engineering: histogram-based split finding and leaf-wise growth, which is why it trains big datasets faster. Column subsampling and native missing-value handling make both behave like a well-tuned random forest and a careful gradient booster at once.
For most tabular problems the source’s order of operations is blunt and correct: start with LightGBM or XGBoost at default settings; tune n_estimators, learning_rate, max_depth and min_child_weight; only if you need the last half-percent build a stacking ensemble of 3–5 diverse models; cross-validate throughout. Neural networks on tabular data are usually worse than a tuned gradient booster — occasionally matching it with much more work. Where ensembles are not worth it: latency-critical inference, tiny datasets where a single regularized model already generalizes, and any setting where you must explain every prediction. An ensemble of 500 trees is not a story a regulator will accept.
Answer before you look. The AdaBoost weight question and the stacking cross-validation question are exactly the ones that separate a memorized definition from a working instinct.
0 / 6 answered · 0 correct
01Why does combining multiple weak classifiers into an ensemble improve accuracy?
02What is the main difference between bagging and boosting?
03In AdaBoost, what happens to the sample weight of a misclassified training point after each round?
04A random forest with 100 trees has the same test accuracy as 200 trees. Adding more trees to 500 also shows no improvement. Why?
05Gradient boosting fits each new tree to what quantity?
06When building meta-features for a stacking ensemble, why do the base-model predictions have to come from cross-validation?
Click a card to swap the lazy description for what it actually means.
Three problems with exact numbers. Try first; a worked answer is one click away.
On the lesson's 22-point, three-block dataset, the accuracies after rounds 1–7 are 68.2%, 63.6%, 68.2%, 86.4%, 86.4%, 100%, 100%. The dip at round 2 is real: round 1 picks the cut x ≤ 4.5 (weighted error 7/22, α = ½·ln(15/7) ≈ 0.381), which isolates the left block but mislabels the seven positive points in the middle and right blocks. Round 2 then picks x ≤ 19.5 to chase those rows; it fixes several of them but newly mislabels the negatives at x = 14–16, so the raw accuracy falls to 14/22 = 63.6% before the weights and later stumps repair it. Convergence is at round 6 (22/22), and the weighted errors stay between 0.26 and 0.35 while α shrinks slowly — the ensemble is done, and extra rounds only dilute it. This is why boosting tracks training accuracy per round instead of trusting the count: more rounds can hurt.
The loop is: fit tree t on the training residuals, add learning_rate × tree to the prediction, compute validation MSE, and remember the round with the lowest value. Suppose the recorded validation MSE is 1.20 at 10 trees, 0.82 at 25, 0.55 at 50, 0.41 at 75, 0.34 at 100, 0.315 at 125, 0.313 at 127 (the minimum), then 0.314, 0.316, 0.319, 0.322, 0.326, 0.330, 0.334, 0.339, 0.344, 0.350 over the next rounds. The best round is 127, and the tenth consecutive non-improvement is round 137 — early stopping returns the 127-tree model and saves all the training that would have followed. Two consequences to state: (1) the returned model is the checkpoint, not the last model, so keep a copy or the weights; (2) the patience window is a hyperparameter — patience 5 would stop at 132 and might miss a later recovery, patience 20 pays more compute before quitting. Typical tuned values land in the 50–300 tree range for learning rates 0.05–0.1.
Split 1,000 rows into 5 folds of 200. For each fold: train all three bases on the other 800 rows, predict the held-out 200, and write those three numbers into the meta-feature matrix; after the loop every row has features produced by models that never saw it. Train the meta-learner on the 1,000 × 3 matrix (plus bias) and score the untouched test set. A representative outcome: logistic 0.812, tree 0.834, KNN 0.798, stack 0.869 — the meta usually lands a couple of points above the best base and almost never far below it. Wrap the fold loop in a pipeline so scaling for KNN and the logistic model is fitted inside each fold; fitting the scaler on all 1,000 rows before splitting is the classic leak. The lesson's own 49-point band dataset makes the mechanism visible with exact numbers: two base cuts at 69.4% each and a near-chance third model at 51.0%; ridge stacking puts weights 1.00, 1.00, 0.00 with bias −1.00 and scores 100.0%, because that weighted sum is exactly 'both cuts say inside'.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 02, Lesson 11) and the Math Foundations Notebook reference build. The six labs, the worked AdaBoost and gradient-boosting checks, the exact majority-vote table, the stacking AND-gate example, worked exercise answers and the jury/relay/coach memory hook are original to this page. Every accuracy a lab prints is computed live from the members and points it displays; the diversity curve is a labelled simulation.