Machine learning flips programming: instead of writing the logic by hand, you hand an algorithm labelled examples, and it returns a model — rules encoded as numbers that generalise to data it has never seen.
Traditional code takes rules plus data and produces output. Machine learning takes data plus expected outputs and produces the rules — a model whose parameters were tuned by the examples themselves.
rules + data → output ⇄ data + answers → modelEach row is one example. The feature columns are what the model may look at. The label column is the answer it learns to predict — and it must never appear among the inputs.
X (features) → y (label)Hold out part of the data before training starts: train for learning, validation for tuning, test for the final honest score. A model scored on its training data is being graded on the answers it memorised.
train 70% · validation 15% · test 15%Machine learning is programming by example: you provide rows of features and their answers, the algorithm finds the rules, and generalisation is whether those rules still work on rows it has never seen.
By the end you will classify a problem as supervised, unsupervised or reinforcement; point at the features and the label in a table; split honestly; walk the eight-step workflow; recognise underfitting and overfitting; read accuracy without being fooled by it; and know when the right answer is to skip ML entirely.
A spam filter written by hand is a pile of if-statements that breaks the moment spammers change wording. A model trained on labelled emails learns the pattern instead — and can be retrained when the world changes.
Traditional programming and machine learning solve problems in opposite directions. In the traditional route you write the logic yourself: if the subject contains “FREE MONEY”, mark it spam. In the learning route you supply examples — emails with their known labels — and the algorithm discovers the rules. What comes out of training is the model: the rules, stored as numbers (called parameters or weights) that map inputs to outputs.
The hard part is not fitting the examples you show it. The hard part is generalising: making good predictions on rows the model has never seen. That single idea — measure on unseen data — is why the rest of this lesson spends so much time on splitting data and on scores that can be trusted.
The smallest complete example is the nearest-centroid classifier. Training computes the average of each class; prediction asks which average is closest. No gradient descent, no hyperparameters, no iteration — yet it has the same three steps every learning algorithm follows: learn a representation, predict from it, evaluate against the truth.
import numpy as np
class NearestCentroid:
"""The smallest possible "learning" algorithm: one mean per class."""
def fit(self, X, y):
self.classes = np.unique(y)
self.centroids = np.array([
X[y == c].mean(axis=0) for c in self.classes # learn: two means
])
def predict(self, X):
distances = np.array([
np.sqrt(((X - c) ** 2).sum(axis=1))
for c in self.centroids # compare distances
])
return self.classes[distances.argmin(axis=0)] # predict: closest mean
def score(self, X, y):
return np.mean(self.predict(X) == y) # evaluate: % correct
# fit = compute means, predict = nearest mean, score = fraction correct.
# Every algorithm you meet later keeps this three-step shape.Here is a one-dimensional spam filter. The single feature is the overlap score with a spam-word list; the label is spam (1) or legitimate (0).
Notice what the algorithm never did: nobody wrote a rule such as “score above 4.5 means spam”. The 4.525 boundary fell out of the two means. Change the examples and the boundary moves.
A training table has a rigid anatomy: each row is one example, each feature column is something measurable, and exactly one column holds the answer the model must learn to predict.
The features (often written X) are the columns the model is allowed to look at. The label (written y) is the answer column. When the label is a category — spam/ham, cat/dog, benign/malignant — the task is classification. When it is a number — a price, a temperature, a demand — the task is regression. Both are supervised: both train on examples whose answers are known.
Models do not read text or pictures; they read numbers. A categorical feature such as sender_in_contacts = yes/no becomes 1/0 before training, and a raw email becomes counts or overlaps. This translation is feature engineering, and the same translation must run again at prediction time — production does not know anything the training table did not encode.
Every row is one example. Click each column header to cycle its role — input feature or the answer column — then check yourself. Exactly one column is the label.
| meeting notes | 0.90 | 0 | yes | ham |
| WIN FREE MONEY NOW!!! | 0.10 | 5 | no | spam |
| invoice attached | 0.40 | 1 | yes | ham |
| URGENT: claim your prize | 0.20 | 4 | no | spam |
| lunch tomorrow? | 0.70 | 0 | yes | ham |
| exclusive offer 4 u | 0.15 | 3 | no | spam |
If is_spam slipped in as a feature, the model would score perfectly in training and be useless in production, because at prediction time nobody has told it the answer yet. The label is what training is trying to predict.
| Supervised task | Classification | Regression |
|---|---|---|
| Output | Discrete categories | Continuous number |
| Question it answers | Which category? | How much? |
| Example | Is this email spam? | What will this house sell for? |
| Output space | {spam, not spam} | Any real number |
| Typical loss | Cross-entropy | Mean squared error |
| What training draws | Boundaries between classes | A curve through the data |
Which of these is a regression task?
The clearest question to ask about any ML problem is what the data gives you: known answers, inputs alone, or a score that arrives after you act.
Supervised learning has labelled examples: photos with cat/dog tags, houses with sale prices. The model learns the mapping, and the label’s type decides the task — categories make it classification, numbers make it regression. Unsupervised learning has inputs only. With no answer column, the model can only look for structure: groups of similar customers (clustering), or a compressed version of 1,000 columns that keeps the important variation (dimensionality reduction). Reinforcement learning has neither a dataset nor labels — an agent acts, receives rewards and penalties, and adjusts its strategy to earn more reward over time. Game-playing, robot control and RLHF for language models all live here.
Most day-to-day work is supervised, unsupervised is the standard tool for exploration and preprocessing, and reinforcement learning is powerful but data-hungry and hardest to make safe. The proportions in practice are lopsided: the vast majority of production models start with a labelled table and a supervised objective.
Ask one question first: is there an answer column? If yes it is supervised. If not, is the model chasing structure or a reward?
10,000 photos are labelled “cat” or “dog”. Learn to tell them apart.
SCENARIO 1 OF 6The three buckets are not the whole story: semi-supervised learning mixes a few labels with many unlabelled rows, and self-supervised learning mints its own labels from the data.
The three buckets are clean; real projects blur them. Semi-supervised learning mixes a few labelled rows with many unlabelled ones — label propagation, pseudo-labelling and consistency training all use the unlabelled data to stretch a small label budget. Self-supervised learning goes further and creates the labels from the data itself: hide 15% of words and predict them (BERT), or predict the next word (GPT). No human labels; the supervision is manufactured from the text.
A model hides 15% of the words in a sentence and learns to predict them. Which paradigm is this?
A score means one of two things: how well the model memorised its examples, or how well it will handle new ones. The split is what decides which question you are answering.
Hold out part of the data before training starts. The model learns on the training set. You compare feature sets and settings on the validation set, which absorbs all your trial and error. The test set stays locked until the end, and is scored exactly once. That final number is the only honest estimate of generalisation you will get.
The order matters more than the ratio. A quick way to remember the sizes: 70/15/15 or 80/10/10. For small datasets, use k-fold cross-validation: rotate which chunk is held out, train k times, and average the k scores. Every row gets to be held out once, and the average is more stable than a single split.
| Split | Purpose | When used | Typical size |
|---|---|---|---|
| Training | The model learns from this data | During training | 60–80% |
| Validation | Tune settings, compare models | After each training run | 10–20% |
| Test | One final unbiased score | Once, at the very end | 10–20% |
The class means are computed from the filled training points only. Watch what happens to the test score when the boundary is allowed to see the held-out points — that is what “peeking” buys you, and why it is a lie.
In a real project the true test set is locked away. This simplified lab lets you break the rule so you can see the size of the lie. Test accuracy computed on data that influenced the fit always looks better than it is.
One more piece of arithmetic worth doing before the split: look at the class balance. If 90 of 1,000 rows are positive and you split at random, a 15% test set holds about 13.5 positives — round to 13 or 14. Stratify the split (keep the class ratio in every pile) so a small test set does not accidentally contain no positives at all.
You try 40 different models, pick the one with the best test-set accuracy, and report that score. What went wrong?
The algorithm is one step of the job. Collecting, cleaning, splitting, evaluating and monitoring are the other seven — and they decide whether the model is useful.
Every project follows the same pipeline: collect raw data, clean and explore it, engineer features, split train/validation/test, train the model, evaluate against a trivial baseline, deploy it, and monitor it while the world drifts underneath. Evaluation sends you backward: a score that is not good enough means better features, not automatically a fancier model. Monitoring sends you all the way back to collecting fresh data.
Cleaning and exploration are not a warm-up; they are the job. The source material puts it plainly: this step often takes 60–80% of total project time. A model trained on duplicated rows, impossible values and mislabelled examples will faithfully learn the mess. No algorithm choice undoes that.
Every ML project runs this loop — the algorithms change, the steps do not. Click any step, or walk it one at a time.
Gather raw examples — labelled if you plan to learn with supervision. More data is usually better, but relevant data beats more data.
Ask: Does the data actually contain the pattern I want to predict?
Common mistake: Collecting whatever is cheap, then discovering the label is missing.
→ The arrows only point forward until evaluation or monitoring sends you back.
This is the same pipeline for a two-dollar regression and a two-million-dollar language-model run: data, cleaning, features, split, fit, evaluate, ship, watch.
A model can fail in two opposite directions. Underfitting is not learning enough; overfitting is learning too much — including the accidents of the training rows.
Underfitting is a model too rigid to capture the real pattern: a straight line through a curved relationship. Training and held-out scores are both poor. Overfitting is a model flexible enough to fit the training rows exactly, noise included: the training score keeps climbing while the held-out score stalls or falls. Good fit is the middle — it captures the real pattern and ignores the accidents.
The classical explanation is the bias–variance trade-off. Bias is error from wrong assumptions (too simple); variance is error from sensitivity to which particular rows happened to land in the training set (too complex). Total error decomposes as bias² + variance + irreducible noise. You cannot remove the noise; you choose a complexity where the other two terms sum to their smallest.
k is how many neighbours vote. Small k bends the boundary around every point; large k flattens it. Watch the two accuracy curves and find the k where held-out accuracy peaks.
k = 1 fits the training data perfectly — including the four mislabelled points — and pays for it on held-out data. That gap is overfitting in its purest form. Note that this simple U-shape is a classical picture; very large modern models can behave differently.
Signs of overfitting in a real run: training accuracy far above validation accuracy, the validation curve turning upward while training loss keeps falling, and performance improving noticeably when you add data. Signs of underfitting: both curves are flat and high, and more data barely helps — the model simply cannot represent the pattern.
A model scores 58% on training data and 56% on held-out data. What is the diagnosis?
Accuracy is the first metric everyone reaches for, and the easiest to be fooled by. And before any metric matters, one question comes first: should this be machine learning at all?
Accuracy is the fraction of predictions that are correct: correct predictions divided by total predictions. If a model labels 90 of 100 emails correctly, its accuracy is 90/100 = 0.9 = 90%. That sentence is the whole formula. The problem is what it hides: a dataset where 99% of rows are one class lets a model score 99% by saying that class every time.
The fix is to count the four possible outcomes separately. For a rare positive class — fraud, disease, a failing machine — write a confusion matrix: true positives (caught), false positives (false alarms), false negatives (missed), true negatives (correctly cleared). Then read the metrics that name your real concern. Recall is the fraction of real positives the model caught; precision is the fraction of alarms that were real. When missing a positive is expensive, recall matters most; when false alarms are expensive, precision does. Always compare against a trivial baseline — the majority class, or random guessing — before celebrating any score.
Fraud is rare: 10 of these 1,000 transactions. Compare a model that always says “legitimate” with one that actually catches most fraud — and watch which one wins on accuracy.
The cost of the two mistakes is not equal: a missed fraud can cost thousands, a false alarm costs a phone call. When classes are rare or mistakes are asymmetric, report recall and precision, not just accuracy.
Different domains weight these differently: a cancer screen would rather over-refer (lower precision) than miss a case (lower recall), while a spam filter that silences real mail is worse than one that lets a few ads through. The metric is a statement about which mistake you refuse to make — choose it deliberately.
ML is a tool with a cost: data collection, cleaning, training, monitoring, and a stream of probabilistic answers that are sometimes wrong. Before reaching for a model, check whether the problem actually wants one.
The decision order that follows from this: Do I have data? Can I write the rules? Is being sometimes wrong acceptable? Do I need full explainability? Do I have enough labels? — and only then, which algorithm. Saying “this should be a rule, not a model” is a senior answer, not a failure.
Answer before you look. The split, overfitting and labelling questions are exactly the ones that decide whether a real project succeeds.
0 / 5 answered · 0 correct
01In supervised learning, what does the model receive during training?
02What is the purpose of splitting data into training and test sets?
03A model gets 98% accuracy on training data but 55% on test data. What is this an example of?
04An e-commerce site wants to group customers into segments based on purchase behaviour, with no predefined labels. Which type of ML is this?
05Which scenario is NOT a good use case for machine learning?
Click a card to swap the lazy description for what it actually means.
Four problems with fully worked answers — the nearest-centroid exercise has every number you need to reproduce it by hand.
Concretely: of 1,000 rows, 700 become train, 150 validation and 150 test. Train the model on the 700; compare feature sets and settings on the 150 validation rows; only when you have stopped changing anything, score once on the 150 test rows. You must not tune on the test set because every comparison you make against it leaks information back into your choices — after a hundred tweaks, the test set has effectively become training data, and the reported score measures how well you fitted the test rows, not how well the model will generalise. The validation set exists to absorb that trial and error; the test set is the one honest exam you take at the end. If 150 rows feels too small to trust, use k-fold cross-validation on the train+validation portion and keep the test set untouched.
(1) Predicting next quarter's sales from past months is regression: the output is a number, and the past is a labelled dataset because the true values are recorded. (2) Grouping grocery baskets into shopping segments without any predefined segment names is clustering — unsupervised, because there is no answer column; you only have the baskets. (3) A warehouse robot learning to pick items, rewarded +1 per success and −0.01 per second, is reinforcement learning: there are no labelled examples, only actions and rewards. Two follow-up habits worth building: name the metric before naming the algorithm (mean absolute error for sales, a within-cluster distance for segments, successful picks per hour for the robot), and write down a trivial baseline (last quarter's sales, one giant cluster, a random policy) so the learned model has something to beat.
The 39-point gap — training far above held-out — is the signature of overfitting: the model has enough capacity to memorise the training rows, noise included. Three fixes: (1) Get more training data, so memorising one quirk costs accuracy on many others. (2) Reduce capacity — a simpler model (fewer features, shallower tree, smaller network) or add regularisation, which penalises large weights; early stopping is a cheap version when you can watch the validation curve. (3) Re-check the split and the features before blaming the model: is the test set from a different time period or a different distribution, and did any answer-like column leak into the features? A model that scores 60% may also be underfitting on a harder test distribution rather than overfitting; the training score alone cannot tell you which. Diagnose with a validation set, fix the data problem first, then adjust capacity.
Legitimate mean = (1 + 2 + 3)/3 = 6/3 = 2.00. Spam mean = (7 + 8 + 9 + 4.2)/4 = 28.2/4 = 7.05. The nearest-centroid boundary sits halfway between the means: (2.00 + 7.05)/2 = 4.525 — left of it is legitimate, right is spam. Training predictions: 1, 2, 3 → legitimate ✓; 7, 8, 9 → spam ✓; the sneaky 4.2 lands left of 4.525, so it is predicted legitimate ✗. Training accuracy = 6/7 ≈ 85.7%. The new email at 4.5 is also left of 4.525 → predicted legitimate; if it were truly spam, that is the same mistake the boundary already makes for points in the overlap zone. Refitting on all 7 points before scoring would not change this training measure, but on a real dataset refitting on the points you are about to score makes them influence the boundary — that is leakage, and the number it produces is optimistic. Fit on train, score on test; report one honest number.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 02, Lesson 01) and the Math Foundations Notebook reference build. The split lab, paradigm sorter, workflow stepper, flexibility playground, accuracy-trap lab, label-anatomy table, every numeric check, and the worked exercise answers are original to this page. Every lab runs in your browser.