fit on train · transform at serve is the whole discipline. One object carries imputation, scaling, encoding, the model and its versions — and the split is the gate that keeps test rows out of every fitted statistic.
A pipeline is an ordered sequence: ingest, validate, split, transform, train, evaluate, package. Each stage has an input and an output contract, and the whole sequence is fitted once as a single object. Reordering the stages is how leaks and impossible dependencies sneak in.
raw → validate → split → fit → train → score → packageA median, a mean, a standard deviation and a category list are all learned parameters. Compute them before the split and the test rows are baked into training; compute them inside the pipeline and every fit — and every cross-validation fold — sees only its own training rows.
fit on train · transform testSerialize the whole object — imputers, scaler, encoder, model — and serving calls transform and predict on it, never fit. Version that artifact with its code, data and config, and “the model from last month” becomes reproducible instead of folklore.
pipeline + model + versions = one artifactA pipeline is a function with fitted parts: prediction = model(encode(scale(impute(row)))), where every part that learns a number — median, mean, standard deviation, category list, weight — learns it on training rows only, and the fitted whole is serialized, versioned and reused unchanged at serving time.
By the end you will be able to name every stage and its contract, spot leakage by looking at the order of the code, route numeric and categorical columns through one ColumnTransformer, read a sklearn Pipeline call sequence, explain why a seed and a data version belong next to every score, and describe what monitoring should watch before accuracy falls.
A pipeline is an ordered sequence of stages, fitted once as a single object. The order is not style — it is the part that keeps test data out of the model.
You have a notebook that loads data, fills missing values with the median, scales the features, trains a model and prints accuracy. It works. You ship it. A month later someone retrains and gets different results: the median was computed over all rows including the test set (data leakage), the scaling parameters were never saved so inference uses different statistics, the feature engineering was copy-pasted between training and serving and the copies diverged, and a city the encoder never saw arrives in production. None of these are hypothetical — they are the most common reasons ML systems fail after launch.
The fix is structural, not a matter of discipline. Package every step into one object and the failure modes disappear together. The canonical stages are:
Two of those stages are the whole lesson. Split is the gate: every later fit happens behind it. Transform is the stage that must remember its fitted state — the median, the mean, the standard deviation, the category list — because serving replays those exact numbers on new rows.
Move the seven stages with the arrows. The checker flags impossible dependencies and the one order that leaks test data into training.
The leaky preset is the mistake from the source lesson: standardize first, split later. Every stage after split may look at test rows only through transform, never fit.
Each stage is a function; the pipeline is their composition. Take a churn row with a missing income and walk it through the fitted stages:
The toy model gives the premium plan a weight of +1.2 and a bias of −0.3; the arithmetic is what matters: every number fed to the model was produced by a stage that stored its fitted constants. At serving time the same composition runs with the same constants, so the same raw row produces the same vector and the same prediction — that is the pipeline contract: f = model ∘ encode ∘ scale ∘ impute.
Data leakage is information from the test split — or from the future — reaching the training process. It inflates the score and hides until launch. The order of two lines of code decides whether it happens.
A scaled feature is z = (x − μ) / σ: take the column, subtract its mean μ and divide by its standard deviation σ. In plain English, the scaler learns the column’s centre and spread, and stores them as fitted parameters. Imputation learns a median; an encoder learns a category list. Any of those parameters computed before the split has been fitted on test rows too — the model then trains in a coordinate system shaped by data it will be graded on.
The mistake hides behind perfectly ordinary code. Compare the two snippets below: both train a model, both score the same rows. Only one of them is an estimate of future performance.
# leaky: fit the scaler on everything
X = df.drop("target", axis=1)
y = df["target"]
scaler = StandardScaler()
X = scaler.fit_transform(X) # test rows are in μ and σ
X_train, X_test = X[:400], X[400:]
# honest: split first, fit on train only
X_train, X_test = X[:400], X[400:]
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train) # learns μ, σ
X_test = scaler.transform(X_test) # reuses themThe same 16 training rows and 10 test rows. Solid links show the selected test row’s neighbours when the scaler is fitted on train; dashed links show them when it is fitted on everything.
Three test rows report the score as 50, 60 and 70 instead of 0–10. Their units inflate the leaked standard deviation from 2.84 to 18.44, which shrinks the score column’s weight and turns two fails into passes. The metric moved; the protocol was already wrong.
The lab’s table has 16 train rows and 10 test rows, with the score column (x2) on a 0–10 scale — except three test rows that arrive from a feed reporting 50, 60 and 70. Watch the mean move first, then the standard deviation.
That shift changes which training rows a test row is closest to. With 3-nearest-neighbours, fitting on train gives 10/10 = 1.000; the full-data scaler gives 8/10 = 0.800. Rows (6, 0) and (7, 0) are true fails whose neighbours switch to passing rows once the score column is compressed. At k = 1 the same comparison is 1.000 versus 0.900. The leaked scaler effectively tuned the distance metric to the test rows’ scale — the model got to peek at the exam while the syllabus was being set.
One honest caveat: a leak does not always move the metric. With plain least squares and an intercept, an affine rescaling of the features leaves predictions unchanged, so an OLS score can be identical under both protocols. Lowering the score is not what makes leakage wrong — using information that would not exist at prediction time is. That is why the fix is structural: split before any statistic, and validate with cross-validation, where each fold fits its own copy of every transformer.
A team median-imputes, one-hot encodes and scales the full dataset, then splits it, trains a model with 5-fold cross-validation and reports the CV score. What is wrong?
A real table mixes numbers, categories and holes. One transformer cannot treat them all the same way — so a ColumnTransformer routes each group through its own fitted path and glues the results back together.
Numeric columns live on different scales: age spans 18–80, income spans four figures to six. Left alone, income dominates every distance-based model. So the numeric path is impute, then scale: missing values become the training median, then each column becomes a z-score, (x − μ) / σ, centred at 0 with spread 1.
Categorical columns cannot be averaged or multiplied. The categorical path is impute, then one-hot encode: missing values become the most frequent category, and each category becomes a column of zeros with a 1 where the row matches. The category list is learned from the training rows, and handle_unknown="ignore" turns a category it has never seen into an all-zero row instead of a crash. Missingness is itself information: a MissingIndicator column (“income was absent”) can be fed to the model alongside the imputed value.
Two fitted paths, one output matrix. That is exactly what sklearn’s ColumnTransformer is: a router that sends each column subset through its own Pipeline and concatenates the numeric outputs in order.
Pick a row and watch it route: numeric columns are imputed then scaled, categorical columns are imputed then one-hot encoded, and the serve row carries a city the encoder never saw.
| row | age | income | city | plan |
|---|---|---|---|---|
| 30 | 50000 | new_york | premium | |
| 40 | — missing | chicago | free | |
| 45 | 90000 | la | premium | |
| 50 | 30000 | new_york | basic | |
| 60 | 70000 | houston | free | |
| 58 | 96000 | seattle | premium |
Toggle drop="first": each categorical stream loses one column because the dropped category is implied by all zeros. Width falls from 9 to 7.
Widths are checkable by hand on the lesson’s table. Five training rows, two numeric columns and two categorical columns:
The two per-row checks are the whole discipline in miniature: the imputed 60000 makes the mean exactly 60000 and the std exactly 20000, and the unseen city contributes no signal instead of an exception. The imputation value and the μ/σ are all fitted on the five training rows — the serve row is transformed, never fitted.
Adding a polynomial branch. Suppose the two numeric columns deserve interaction terms. PolynomialFeatures(degree=2, include_bias=False) turns [age, income] into [age, income, age², age·income, income²] — 2 → 5 columns. It belongs inside the numeric pipeline, after imputation and before scaling: the row (45, 60000) produces 2025, 2.7 million and 3.6 billion, and those three magnitudes must be standardized like any other feature. The output grows from 9 to 5 + 4 + 3 = 12 columns, and every new column is still fitted and transformed inside the same train-only pipeline.
sklearn’s Pipeline is a chain with one rule: fit runs the whole chain forward on training data, predict re-runs the stored transformations on new rows. The fitted object — not the code — is the model.
The whole apparatus reduces to two verbs. fit learns every step’s constants and transforms the training data along the way. transform applies the stored constants without learning anything. fit_transform is both at once, and it is only ever called by the pipeline on the rows behind the split. At prediction time the pipeline walks the same steps with transform — never fit — which is what makes training and serving use identical statistics.
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
import joblib
numeric = Pipeline([
("impute", SimpleImputer(strategy="median")),
("scale", StandardScaler()),
])
categorical = Pipeline([
("impute", SimpleImputer(strategy="most_frequent")),
("encode", OneHotEncoder(handle_unknown="ignore")),
])
preprocess = ColumnTransformer([
("num", numeric, ["age", "income", "score"]),
("cat", categorical, ["city", "plan"]),
])
pipe = Pipeline([
("preprocess", preprocess),
("model", LogisticRegression(max_iter=1000)),
])
pipe.fit(X_train, y_train) # fit_transform each step, then fit model
predictions = pipe.predict(X_test) # transform each step, then predict
joblib.dump(pipe, "pipeline.joblib")Nothing here is magic, and the from-scratch version in the source lesson shows it. A transformer is any object with fit, transform and fit_transform — fit stores the constants, transform applies them. A pipeline is a list of such objects plus a model, and its two methods are loops:
class PipelineFromScratch:
def __init__(self, steps): # [(name, step), ..., (name, model)]
self.steps = steps
def fit(self, X, y):
X_current = X
for name, step in self.steps[:-1]:
X_current = step.fit_transform(X_current) # learn + apply
name, model = self.steps[-1]
model.fit(X_current, y)
return self
def predict(self, X):
X_current = X
for name, step in self.steps[:-1]:
X_current = step.transform(X_current) # apply stored state
name, model = self.steps[-1]
return model.predict(X_current)Every number below is computed live from the same 500-row table. Change the seed to reshuffle the split; turn imputation off to watch the NaN sanity check fail.
The 2017 rules-of-ML advice in one screen: add checks between stages, keep the split reproducible, and never let a transform see rows it did not fit on.
Trace the two calls on the lesson’s 400/100 split. The sequence is fixed, and so is the direction of learning:
pipe.fit(X_train, y_train): the imputer learns three medians from 400 rows, the scaler learns three μ/σ pairs, the encoder learns two category lists (5 cities, 3 plans), and both transform the training matrix. Then the model fits its weights on the transformed rows.pipe.predict(X_test): the same imputer, scaler and encoder call transform on the 100 test rows using the stored constants. The model predicts. No statistic is recomputed.cross_val_score(pipe, X, y, cv=5) clones the empty pipeline five times. Each clone fits on its own 320-row training slice and scores its 80-row validation slice, so each fold has its own medians, μ/σ and category lists.The last line is the point of the whole exercise: 55 constants fitted inside the CV loop all come from training rows only, so every one of the five fold scores is a clean estimate. Fit once on everything and you have 11 constants that quietly know the test set.
A service loads pipeline.joblib and calls pipeline.fit(new_requests) before predicting, so it can adapt to fresh data. What just happened?
Reproducibility is four fixed things — seed, dependencies, data, configuration — plus one habit: log them next to every number the pipeline prints, and version the fitted pipeline as one artifact.
A reproducible experiment needs four inputs pinned. Seeds: random.seed, np.random.seed, the framework’s seed — the split, the initialization and the batch order all move without them. Pinned dependencies: an exact requirements.txt or lock file, because a library upgrade can change a default or a floating-point path. Versioned data: DVC stores the bytes in object storage and the hash in git, so checking out a commit plus dvc checkout restores the exact rows a run saw. Config files: every hyperparameter in YAML or JSON instead of buried in cells, so “the run with depth 5” can be rebuilt.
On top of that sit the logs. Experiment tracking (MLflow, Weights & Biases) records parameters, metrics, artifacts, tags and code revisions for every run, which turns “the good one from last month” into a searchable row. A model registry adds versions and stage labels — staging, production, archived — plus explicit promotion and instant rollback. The unit that gets versioned is the fitted pipeline, not the notebook: one artifact carrying its imputer, scaler, encoder, model and schema.
Build a run from its four tags, log it, then replay the whole log. A run is reproducible when the same tags return the same score — but a matching score alone proves nothing.
Runs #5 and #6 score the same 0.7333 from different changes — proof that equal metrics cannot identify a configuration. Only the tags plus a replay can.
The lab’s six runs are the same tiny experiment under different tags. Read the columns as the receipt for each score:
Two lessons fall out of the table. First, the seed is not a detail: changing it moved the score more than the data version did, so a single split can sell a story that is mostly luck. Second, matching scores do not mean matching runs — runs #5 and #6 agree to four decimals for completely different reasons. The audit is mechanical: recompute from the recorded tags and require the same numbers. If the replay differs, something not in the tags changed, and that something is the bug.
One honest limit: bit-identical replay is easy on a single CPU and harder on GPUs, where parallel reductions can reorder floating-point additions. Frameworks offer deterministic modes (torch.use_deterministic_algorithms, cuDNN deterministic flags) that trade speed for exactness. When exact replay is impossible, record the tolerance and the hardware, and compare within it — but never pretend a tag list is complete when it omits the machine.
You rerun last week's training with the same code and data and get 0.8421 instead of 0.8406. Nothing in the pipeline changed. Which is the most useful next step?
Train/serve skew is a mismatch in code. Data drift is a mismatch in the world. The first is removed by one pipeline object; the second can only be monitored, detected and retrained against.
Two different things are called “it broke in production”. Train/serve skew is the pipeline problem: the median is computed at training time but a different constant is used at serving time, because the preprocessing was reimplemented, the scaler was never serialized, or one code path fills missing values and the other drops the row. One fitted pipeline used in both places removes this class entirely.
Data drift is the world problem: the inputs move. A sensor is recalibrated, a marketing campaign changes who signs up, a partner starts reporting a column in different units. The model is still the model; its inputs now land somewhere else in feature space. Concept drift is deeper still — the relationship between inputs and target changes, so even a correctly measured feature no longer predicts what it used to. The pipeline cannot remove drift, but it can make it visible: log input distributions, missing rates and prediction distributions at serving time, and compare them with the training snapshot.
Retraining has three triggers: a schedule (nightly, weekly), a monitored signal (drifted inputs, changing prediction mix), and — when labels arrive late — a drop in measured performance. Ship the new pipeline behind a comparison: shadow it on live traffic, or run champion versus challenger, then promote through the registry and keep the rollback ready.
The model is trained once on 60 rows. The slider simulates a calibration drift that lowers every serve-time attendance value; watch the accuracy curve in the lower panel.
At drift 20 accuracy is 0.775, at 30 it is 0.700, and at 40 it is 0.425. The model never retrained — only the world moved.
The lab trains once on 60 rows and scores the same 40 serve rows with the attendance column subtracted by an increasing drift. Nothing is retrained; only the input distribution moves.
Two numbers in that table deserve to be circled. First, the metric’s early plateau is a cancellation effect: rows flip in both directions when drift is small, so accuracy barely moves while the model is already seeing different data. An input monitor catches this months before a metric does. Second, the cliff is steep — once a critical mass of rows crosses the decision boundary, accuracy collapses from 0.775 to 0.425 in 20 points of drift. Waiting for the dashboard to turn red means waiting until the damage is already visible to users.
The practical loop: log the serve-time distribution of every input and of the prediction, compare with the training snapshot on a schedule, alarm when a population-level statistic moves beyond a threshold, and retrain with fresh labelled data — through the same pipeline, so the new model arrives with a receipt.
A ranking model's live accuracy has not moved in a month, but the share of its predictions in the top class has grown from 30% to 55%. Inputs look stable. What is the most likely explanation?
A pipeline is a graph of tasks. An orchestrator schedules the graph, passes artifacts between tasks, retries failures, records every run, and pages a human when the result cannot be trusted.
Training once is easy. Retraining every night, on fresh data, with a score gate, a registry entry and an alert if validation fails — that is an orchestration problem. Tools like Airflow, Prefect, Dagster and Kubeflow Pipelines express the pipeline as a DAG: each node is a task, edges are dependencies, and the orchestrator decides what to run, in what order, with what retries and what history.
Three properties make a DAG production-grade. Artifact passing: the split task outputs the exact train/test tables the transform task consumes, so no step re-reads mutable storage halfway through. Idempotency: rerunning a task must produce the same artifact, not a duplicate row — which is also what makes backfills (reprocessing last month with today’s code) safe. Isolation: each run pins its container or environment, so “which Python was this?” has an answer. Add CI tests of the pipeline itself — schema in, shapes out, no NaNs — and the 3 a.m. failure becomes a failed test instead of a silent deployment.
| tool | what it is good at |
|---|---|
| Airflow | Python-defined DAGs and the widest ecosystem; the default choice for scheduled batch pipelines. |
| Prefect / Dagster | Dynamic flows and asset-aware scheduling: tasks declare the data they produce, not only when they run. |
| Kubeflow Pipelines | Container-native ML pipelines on Kubernetes, with per-step resource limits and artifact lineage. |
| GitHub Actions / cron | CI tests plus lightweight scheduled retrains; perfect until you need retries, backfills and run history at scale. |
Answer before you look. The scaler question and the unknown-category question are the two that separate a memorized definition from a working instinct.
0 / 5 answered · 0 correct
01What is data leakage in the context of an ML pipeline?
02Why is fitting a scaler on the full dataset before splitting into train and test considered leaky?
03In sklearn, what is the difference between calling fit_transform and transform on a pipeline step?
04Why is a ColumnTransformer necessary for real-world datasets?
05A production model receives a categorical value it never saw during training ('new_category'). What should the pipeline handle?
Click a card to swap the lazy description for what it actually means.
Four problems with exact numbers — a mixed column transformer, a deliberate leak, a serialization identity check and a reproducibility audit. Try first; a worked answer is one click away.
The lesson's console does exactly this on 500 rows (400 train / 100 test). Numeric: [age, income, score] → impute medians 34.2 / 38,805 / 571.7 → scale with means 34.76 / 52,674 / 574.57 and stds 10.56 / 43,410 / 160.18 → 3 columns. Categorical: city (5 known values) and plan (3) → 5 + 3 = 8 one-hot columns. Combined matrix: 3 + 8 = 11 features, shapes 400 × 11 and 100 × 11. Code: ColumnTransformer([("num", Pipeline([("impute", SimpleImputer(strategy="median")), ("scale", StandardScaler())]), ["age", "income", "score"]), ("cat", Pipeline([("impute", SimpleImputer(strategy="most_frequent")), ("encode", OneHotEncoder(handle_unknown="ignore"))]), ["city", "plan"])]) then Pipeline([("preprocess", preprocessor), ("model", LogisticRegression(max_iter=1000))]). cross_val_score with cv=5 refits the medians, means, stds and category lists on each fold's 320 training rows and scores the held-out 80, so each of the 5 reported numbers is honest.
On the lesson's 26-row leakage table (16 train, 10 test) with 3-nearest-neighbours: clean scaler fitted on the 16 train rows gives test accuracy 10/10 = 1.000; the leaky scaler fitted on all 26 gives 8/10 = 0.800. The leaky fit moves the score column's mean from 3.69 to 10.15 and its standard deviation from 2.84 to 18.44, because three test rows report scores of 50–70 instead of 0–10. The inflated σ shrinks the score column's weight, so rows (6, 0) and (7, 0) — both fails — get classified as passes: the model was tuned, in effect, on the test rows' scale. With k = 1 the same comparison is 1.000 versus 0.900. The size of the gap depends on the data and model — kNN and regularized models move, plain least squares can be invariant to affine rescaling — so a small gap is never proof that the split was clean; only the protocol is.
joblib.dump(full_pipe, "pipeline.joblib") writes the whole object — imputer statistics, scaler mean/std, category list, model weights. In the serving script: pipe = joblib.load("pipeline.joblib"); new_predictions = pipe.predict(df). Check with np.array_equal(predictions, new_predictions) — it returns True because no transform is recomputed: the same medians, the same μ/σ, the same one-hot columns run in the same order. What could still differ: a different row order or dtype in df (align columns by name), a library version that changes an algorithm's default (pin versions), or a transformer fitted again after loading (never call fit at serving time — transform only). A versioned artifact plus pinned dependencies is what makes the check repeatable months later.
Run 1 (seed 42, data v3, code 2.1.0, lib 1.3.0): 0.900. Run 2, identical tags: 0.900 — bit-identical, so the configuration is reproducible. Run 3 (seed 7): 0.967. Run 4 (data v2): 0.867. Run 5 (code 2.0.0, one feature): 0.733. Run 6 (lib 1.2.0, no standardization): 0.733. Runs 5 and 6 land on the same score with completely different tags — equal metrics are not evidence of equal configurations, and worse, the score alone cannot tell you which change caused a result. The audit answer is tag-by-tag comparison plus a replay: recompute each run from its recorded seed, data version, code version and library version, and require the numbers to match exactly. Then promote the best run to the registry's production stage, keep the rest archived, and remember that “best” here means 0.967 from a single split; confirm it with cross-validation before shipping.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 02, Lesson 13) and the Math Foundations Notebook reference build. The six labs, the 26-row leakage table, the mixed column table, the run log, the accuracy-versus-drift curve, the worked exercise answers and the added numeric checks are original to this page. Every score a lab prints is computed live from the table or curve it displays; the datasets are small teaching stand-ins, not measurements of any real model.