grid · random · Bayesian are three ways to search the same surface, priced by evaluations. On this one the peak is 10.000: the 4 × 4 grid stops at 9.961, sixteen random draws reach 9.986, and ten guided steps find 10.000.
Weights, biases and split thresholds are learned — the optimizer moves them every step. The learning rate, number of trees, depth, regularization, batch size and dropout are chosen before training starts. There is one learned solution per configuration; there is no such guarantee for the configuration itself.
5⁶ = 15,625 combinationsEvery configuration comparison you make on the test set consumes it. Choose features, models and knobs on validation; touch the test split once for the final report. Run 100 trials and the winner looks about 2.5σ better than average before any real signal is counted — that is selection luck, not skill.
tune on validation · report on testStart at defaults, run 20–50 wide random trials with early stopping, read which knobs matter, then refine 50–100 trials with random or Bayesian search in the narrowed range and retrain on all training data. Grids are for two knobs you can afford to exhaust; beyond that, breadth beats precision.
random breadth → Bayesian focus → retrainHyperparameter tuning is one more layer of fitting: the search fits configurations to the validation folds, so the winner’s validation score is optimistic — the honest number is a locked test set or nested cross-validation, and the cheapest way to search is wide random first, Bayesian after.
By the end you will be able to tell parameters from hyperparameters, price a grid before running it, choose random search over grid search for the right reason, explain the surrogate and acquisition split of Bayesian optimization, use early stopping and schedules to buy the same answer for less, and report a score that includes the cost of tuning.
Two kinds of numbers live in a model. Training writes one kind; you write the other — and there are far more ways to write it than you can ever try.
Parameters are the numbers the optimizer updates during fit(): weights, biases, and the split thresholds a tree chooses. Hyperparameters are the numbers set before training starts that control how that learning happens. A rule of thumb: if it changes while training runs, it is a parameter; if you choose it to start training, it is a hyperparameter.
The split matters because the two kinds of numbers have completely different costs. There is exactly one set of learned parameters for a given configuration — the optimizer finds it. There is no such guarantee for the knobs, and the space is large enough that how you search it decides what you find.
| Hyperparameter | What it controls | Typical range |
|---|---|---|
| Learning rate | Step size of each update | 0.001 – 1.0 · log scale |
| Trees / epochs | How long to train | 10 – 10,000 |
| Max depth | Model complexity | 1 – 30 |
| Regularization λ | Overfitting prevention | 0.0001 – 100 · log scale |
| Batch size | Gradient-estimate noise | 16 – 512 · linear |
| Dropout rate | Fraction of units dropped | 0.0 – 0.5 |
A logistic regression on 8 features learns 9 parameters (8 weights + 1 bias) no matter how you configure it. Now count the knobs instead. If each has 5 reasonable values, the grid of combinations is 5 raised to the number of knobs:
Nothing about the model got smarter; the search just got deeper. The rest of this lesson is about spending those hours where they buy the most.
Which of these does training write by itself, with no choice from you?
Searching configurations is fitting. Do it on the test set and the number you report is the maximum of many noisy guesses, not what the model will do next.
The three-way split from Lesson 01 exists for exactly this reason. Training data fits the parameters. Validation data chooses everything else: features, architecture, and every hyperparameter. The test data measures the final result — once. Tuning on the test set folds the validation role into it, so the reported score is the best of N looks at the same rows.
That is not a small distortion. When you compare many configurations, the winner is selected partly for genuine quality and partly for luck on those specific validation rows. Report the winner’s validation score as if it were its true skill and you are reporting the luck too. The fix is structural: keep a test set that no choice ever touched, or wrap the whole tuning procedure in nested cross-validation (Chapter 06).
Imagine 20 configurations whose true quality is identical — every difference you see is noise on the validation split. Model each estimate as a standard normal draw around the shared truth. What does the winner look like?
A 100-trial sweep guarantees that its winner looks about 2.5σ better than an average configuration before any real signal is counted. Validation is still the right place to make those choices — it just means the winner’s score is optimistic, and the honest number comes from data the search never saw.
A pipeline tunes 40 configurations on validation and reports the best validation score as the expected test score. What did it actually measure?
With the same budget, the two strategies buy completely different things: coverage of every combination, or resolution in each knob. Usually you want the second.
Grid search evaluates every combination in a list of values. It is exhaustive and trivially reproducible: a 3 × 3 grid of learning rate and depth runs exactly 9 fits. But the cost multiplies with every knob — the same 5 values across 6 knobs is 15,625 fits — and that exponential blow-up is the curse of dimensionality showing up in a search space. The coverage is worse than it looks, too: if only the learning rate matters, those 9 fits test 3 distinct learning rates.
Random search draws each knob from a distribution, one draw per trial. The same 9 trials test 9 distinct learning rates, and every other knob gets 9 values too. This is why random search so often wins at equal budget: it follows the Bergstra & Bengio observation that most problems have low effective dimensionality — one or two knobs do most of the work, and the rest are nearly flat. A grid wastes trials varying the flat directions; random spends them all on every direction.
The one place random beats grid for free is the search scale. Draw the learning rate and regularization strength log-uniform, so 0.001–0.01 gets as many samples as 0.1–1.0. Draw batch size and depth linearly. Match the distribution to how the knob actually behaves.
One fixed surface. The grid spends its evaluations on graph paper; random spends them everywhere. Drag the reveal slider to watch each strategy’s best-so-far score climb.
Change the budget and re-run: with few evaluations, breadth in the knob that matters beats precision in the knobs that do not.
Coverage. A grid of m values per knob over k knobs is mᵏ evaluations, but if only one knob matters, the grid offers only m distinct values of it. Random search offers all mᵏ distinct values at the same price.
The lab in numbers. The displayed surface peaks at 10.000 at lr ≈ 0.0182. A 4 × 4 grid (16 evaluations) tops out at 9.961 — its best point is lr 0.01, the nearest grid line. Sixteen random draws (seed 7) reach 9.986 by their fourth evaluation, finding lr 0.025. Over 10 seeds, random won all 10 times, scoring 9.964–10.000 (mean 9.989). Only when the grid is dense enough to land beside the optimum does it catch up: a 6 × 6 grid (36 evaluations) hits 9.9975 because 0.4 is 0.02 away from the true peak. The lesson is about budget: spend small budgets on breadth, not precision.
Random search never notices that high learning rates diverge. A surrogate model does — and an acquisition function turns that knowledge into where to look next.
Bayesian optimization replaces guessing with two pieces. First, a surrogate model — usually a Gaussian process — is fitted to every (configuration, score) pair observed so far. At any candidate point it returns a prediction μ(x) and an uncertainty σ(x). Plain English: “the score here is probably around this, and here is how surprised I could be.”
Second, an acquisition function scores candidates using both numbers and picks the next evaluation. That score is where the exploration/exploitation trade-off lives. The classic choice is Expected Improvement (EI): the average amount by which the next evaluation is expected to beat the current best. UCB is simpler: prediction plus κ times uncertainty. Whichever you use, the rule is the same — a point wins by being promising or by being unknown.
Four starting points, then each step fits the surrogate and evaluates where Expected Improvement is largest. Watch the uncertainty band collapse onto the peak.
Mean alone says “stay at x = 0.55 where the best score is”; EI says “x = 1.0 is unexplored, that is where the information is.” It visits the edge first, then closes in on the peak.
The surrogate. With observations y at points X, the Gaussian process prediction at a new point x* is a weighted average of the observed scores, weighted by proximity — plus an uncertainty that shrinks near observed points:
The acquisition. With best score f_best so far, EI is
Worked comparison. Best so far is 0.7. Candidate A has μ = 0.8, σ = 0.25; candidate B has μ = 0.6, σ = 0.6. Which does EI evaluate?
The lab in numbers. The four starting points leave the best at 0.9218 (x = 0.55). EI at that high-mean point is only 0.0146, while the untouched edge x = 1.0 has EI 0.0967 — so the first step explores the edge (score 0.5033). The next step exploits: x = 0.653, score 0.9812, then x = 0.633, 0.9970. Five EI steps in, the best is 0.9989; ten steps reach 0.9992. Ten guided evaluations did what random search typically needs 25–50 draws to match — the 2–5× fewer evaluations the source reports.
After the four starting points, the surrogate's highest mean sits at x = 0.55, yet the first EI step goes to x = 1.0. Why?
Not every configuration deserves the full training budget. Stopping bad runs early, and letting the learning rate move during a run, are the two cheapest tuning tricks in the book.
Early stopping ends a run when validation performance has not improved for patience consecutive epochs, and keeps the best checkpoint rather than the last one. Inside a search, this is not just a time-saver: a configuration that is clearly bad after 10 epochs does not need 200. Median pruning goes further and compares a trial’s intermediate score with the median of completed trials at the same step. Hyperband turns pruning into a tournament: start many configurations on a tiny budget, keep the best fraction, and triple the budget of the survivors.
The other cheap trick is a learning-rate scheduler. Instead of tuning one fixed constant, you choose a shape for it. A high rate early makes fast progress; a small rate late settles the solution. Schedules often beat every fixed learning rate because no single value has to serve both phases of training.
Three configurations, 20 epochs each. Raise the patience and watch bad runs die earlier — the best checkpoint never changes.
The good run’s best epoch is 10 whatever the patience: stop markers end the search, they do not change the checkpoint you keep. The divergence run is killed at epoch 5 — 15 epochs of compute returned to the search budget.
| Scheduler | Rule | When to reach for it |
|---|---|---|
| Step decay | lr × 0.1 every N epochs | Classic CNN training |
| Cosine annealing | lr × 0.5 × (1 + cos(πt/T)) | Modern default |
| Warmup + decay | linear rise, then cosine fall | Transformers |
| One-cycle | rise and fall over one cycle | Fast convergence |
| Reduce on plateau | × factor when the metric stalls | Safe default |
Patience, computed. The good run in the lab bottoms out at 0.340 on epoch 10, then drifts up as it overfits. With patience 3 the stop lands on epoch 13 — 7 of 20 epochs returned. With patience 5 it lands on 15 — 5 of 20 saved. The divergence run bottoms out at 1.05 on epoch 2 and is killed at epoch 5 with patience 3: 15 epochs of compute handed back. Across all three runs the ledger is 38 of 60 epochs at patience 3 (36.7% saved) and 42 of 60 at patience 5 (30%).
Hyperband, computed. Start 81 configurations at 1 epoch; keep the top third (27) and give them 3 epochs; keep the top third again (9) and give them 9 epochs.
Cosine, computed. With lr₀ = 0.1 and T = 10 epochs, lr(t) = 0.1 × 0.5 × (1 + cos(πt/10)) gives 0.100 at t = 0, 0.050 at t = 5, and 0.000 at t = 10. One tuned constant became a schedule that starts fast and lands softly.
Ordinary cross-validation tells you how good the winner looks. Nested cross-validation tells you how good the whole procedure — tuning included — actually is.
Tuning on a single validation split risks overfitting that split: the winning configuration may simply be the one that got lucky on those rows. Nested cross-validation separates the two jobs with two loops. The outer loop splits the data into a test part and a training part, five times. Inside each outer fold, the inner loop runs its own cross-validation on the training part to pick hyperparameters. The chosen configuration is then scored once, on the outer test part that the inner loop never touched.
Each outer fold finds its own best configuration independently — they are allowed to disagree. The mean of the outer scores is an unbiased estimate of the procedure: feature pipeline, tuning budget and all. It is not the score of one model, and it is not how you pick the final configuration for deployment; for that you tune once on the full training data. Nested CV answers the harder, more honest question: if I ran this whole workflow on fresh data, what should I expect?
Pick an outer fold. The inner loop tunes λ on that fold’s training rows only; the outer score is measured once on the held-out rows. The optimistic score below is what happens when the same rows do both jobs.
held-out rows: 1, 5, 9 · tuned on rows 2, 3, 4, 6, 7, 8, 10, 11, 12
The optimistic number is not a lie anyone tells on purpose — it is what you get when the same data chooses λ and reports the score. Nested CV separates the two jobs.
| λ | inner CV MSE (fold 1 training rows) | verdict |
|---|---|---|
| 0.01 | 0.1743 | rejected |
| 0.1 | 0.1739 | tuned here → evaluated once on the held-out rows |
| 1 | 0.1936 | rejected |
| 10 | 1.4955 | rejected |
The model is one ridge weight, w = Σxy / (Σx² + λ), so the two loops cost almost nothing here. With a real model each inner fit is a full training run — the 5 × 5 × 27 = 675 fits the lesson prices below.
The two loops multiply. With 5 outer folds, 5 inner folds, and a 3 × 3 × 3 = 27-point inner grid, one full nested run costs:
The lab in numbers. On the 12-row ridge example, the optimistic approach — choose λ by scoring every λ on all rows, then report that score — gives 0.1191 MSE. Nested CV, where each fold tunes on its own training rows and is judged on its held-out rows, gives 0.1575: an optimism of 0.0384 MSE, and in RMSE terms 0.345 versus 0.397 — the optimistic protocol understates the error by about 13%. The four folds even disagreed about λ (0.1, 0.1, 0.01, 0.01), which is exactly why one split’s winner is not a truth. Use nested CV when the number will be published, compared across teams, or used to make a high-stakes decision — and plain validation splits for ordinary iteration.
The inner loop picks λ = 0.1 in two outer folds and λ = 0.01 in the other two. What is the correct reading?
The workflow that survives contact with a real budget: start at the defaults, search wide with random trials, learn which knobs matter, then refine and retrain.
1. Start with library defaults. They encode years of practitioner experience and are often 80% of the way there. The first sweep is looking for the last 20%, so do not spend it re-deriving the first 80%.
2. Coarse random search, wide ranges, 20–50 trials. Use log-uniform distributions for the learning rate and regularization, linear for depth and batch size, and early stopping to kill bad runs fast. 3. Analyze what correlated with performance — Optuna’s parameter-importance plot, or just the best 10 trials side by side — and narrow the ranges. 4. Fine search with random or Bayesian methods, 50–100 trials inside the narrow space. 5. Retrain on all training data with the winning configuration, and evaluate once on the test set.
Spend the budget where the variation is: put 60% of trials on the top two knobs (almost always the learning rate plus one more) and 40% on everything else. When in doubt, run twice as many trials as you have hyperparameters — six knobs means at least 12 trials, and 50 is a strong default. Random search at 50 trials beats a carefully designed small grid more often than intuition suggests.
Drag the probe anywhere on this validation-score heat map (or use the two sliders). The bright band runs almost straight up: learning rate sets the score, the second knob barely moves it.
This is why importance-aware budgets exist: spend 60% of the trials on the top two knobs, leave the flat directions at their defaults.
Take the budget lab’s defaults: 27 grid configurations, 5-fold validation, 200 epochs per fit, 1.5 seconds per epoch.
The arithmetic is not pro-search or anti-search; it is pro-order. A 5-hour random sweep plus a 1-hour Bayesian refinement and a one-day nested validation costs a fraction of the blind grid and answers a better question. And the flat directions in the surface above get their defaults, not their own experiments.
Configurations × folds × epochs × seconds: change any number and the bill updates. This is the arithmetic that decides random versus grid.
The chapter’s six-knob grid is the same formula at a bigger scale: 15,625 configs × 5 folds × 200 epochs × 1.5 s = 271 days. A 40-trial random search with early stopping finishes in 5 h.
| Model | Tune first | Recommended search | Budget |
|---|---|---|---|
| Random forest | n_estimators, max_depth, min_samples_leaf | Random, ~50 trials | Low |
| Gradient boosting | learning_rate, n_estimators, max_depth | Bayesian, ~100 + early stopping | Medium |
| Neural network | learning_rate, weight_decay, batch_size | Bayesian or random, 100+ | High |
| SVM (RBF) | C, gamma | Grid on log scale, 25–50 | Low |
| Lasso / ridge | alpha | 1D log-scale search, ~20 | Very low |
| XGBoost | learning_rate, depth, subsample, colsample | Bayesian, 100–200 + early stopping | Medium |
Importance tiers to guide step 3: high — learning rate, regularization, and the iteration count (replace the last one with early stopping); medium — depth or layer count, minimum samples or weight decay, subsample; low — max features, activation choice, and batch size within a reasonable range. Never search batch size on a log scale; always search learning rate and regularization on one.
Answer before you look. The random-versus-grid question and the test-set question are exactly the ones that separate a memorized definition from a working instinct.
0 / 5 answered · 0 correct
01What is the difference between a parameter and a hyperparameter?
02Grid search over 4 hyperparameters with 5 values each requires how many evaluations?
03Why does random search often outperform grid search with the same evaluation budget?
04In Bayesian optimization, what does the acquisition function balance?
05You tune hyperparameters on the test set and report the best test performance. What is wrong with this approach?
Click a card to swap the lazy description for what it actually means.
Four problems with exact numbers — grid versus random, a Hyperband bracket, an EI computation and a budget. Try first; a worked answer is one click away.
The 4×4 grid evaluates the same 16 points every time and tops out at 9.961 — its best point is (lr 0.01, depth 4), while the true maximum is 10.000 at (lr 0.0182, depth 5). The grid cannot see between its own lines. Sixteen random draws reached 9.986 with seed 7 and beat the grid in 10 of 10 seeds, with results between 9.964 and 10.000 (mean 9.989). The margin comes from resolution in the one knob that matters: 4 unique learning rates for the grid versus 16 for random. Grid catches up only when it is dense enough that a grid line lands near the optimum — on this surface a 6×6 grid (36 evaluations) scores 9.9975 because x = 0.4 is 0.02 away from 0.42, while 36 random evaluations score about 9.988. At small budgets, spend on random breadth, not grid precision.
Round 1: 81 configs × 1 epoch = 81 epochs; keep 27. Round 2: 27 × 3 = 81 epochs; keep 9. Round 3: 9 × 9 = 81 epochs; the winner is the best of those. Total = 81 + 81 + 81 = 243 epochs versus 81 × 9 = 729 for the full grid — exactly 3.00× cheaper, and no single run is longer than 9 epochs. Add a fourth rung (keep 3, give them 27 epochs): 324 epochs versus 81 × 27 = 2,187, a 6.75× saving. Finally retrain the winner for the full budget when you need the absolute best model: 243 + 9 = 252 epochs, still 2.9× under budget.
For A: z = (0.8 − 0.7)/0.25 = 0.4, Φ(0.4) = 0.6554, φ(0.4) = 0.3683, so EI = 0.1 × 0.6554 + 0.25 × 0.3683 = 0.0655 + 0.0921 = 0.1576. For B: z = (0.6 − 0.7)/0.6 = −0.1667, Φ = 0.4338, φ = 0.3934, so EI = −0.1 × 0.4338 + 0.6 × 0.3934 = 0.1927. B wins despite the lower mean: its uncertainty is the opportunity. UCB with κ = 2 agrees (A: 0.8 + 0.5 = 1.30, B: 0.6 + 1.2 = 1.80). In the lab, after four starting points the best is 0.9218 at x = 0.55; EI at that high-mean point is only 0.015, while the untouched edge x = 1.0 has EI = 0.097 — so step 1 explores the edge (score 0.5033). Step 2 lands at x = 0.653 (0.9812), step 3 at x = 0.633 (0.9970), and by step 6 the best is 0.9989 — the surrogate is now refining around the peak. Exploration first, exploitation after.
Full bill: 5⁶ = 15,625 configurations × 5 folds = 78,125 fits; × 200 epochs = 15,625,000 epoch-fits; × 1.5 s = 23,437,500 seconds = 6,510 hours ≈ 271 days. Under-6-hour plan: fix the four least important knobs at defaults, keep learning rate plus n_estimators, and run 40 random trials with patience-based early stopping. 40 trials × 5 folds = 200 fits; average 60 epochs per fit (bad runs die earlier) = 12,000 epoch-fits; × 1.5 s = 18,000 seconds = 5.0 hours. Spend the last hour on a 20-trial Bayesian refinement inside the narrowed range, and reserve the test set for a single final report.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 02, Lesson 12) and the Math Foundations Notebook reference build. Interactive figures, the six labs, the response surface shared by the hero and labs, the worked EI and Hyperband numbers, and the worked exercise answers are original to this page. Every score a lab prints is computed live from the surface or table it displays.