margin = 2 / ‖w‖ is the whole idea: widen the buffer until only the nearest points touch it. Then C prices the mistakes, and a kernel bends the street into any shape.
Scale w and b so the nearest points sit at y·(w·x + b) = 1. Then the distance from the boundary to each rail is 1/‖w‖ and the street width is 2/‖w‖. Maximizing that width is minimizing ½‖w‖² — a convex problem with one global answer.
margin = 2 / ‖w‖The points touching the rails are the support vectors; they are the only ones with nonzero dual weight α. Delete a point outside the street and the boundary does not move. At prediction time the model is just the support vectors.
αᵢ > 0 ⟺ support vectorReal data overlaps, so each point may break its constraint for a price: slack ξ = max(0, 1 − y·f(x)), multiplied by C. Swap the dot product for a kernel K(x, z) and the same linear machine draws curved boundaries without ever visiting the feature space.
½‖w‖² + C·Σ max(0, 1 − yᵢ·f(xᵢ))An SVM draws the widest street it can between the classes, charges a fine C·ξ to any point that wanders into the street, and at prediction time remembers only the points standing on the curb — or, with a kernel, only their pairwise similarities.
By the end you will be able to compute a margin from w, say which points are support vectors and why, read the C and slack trade-off off a plot, parse hinge loss, and choose between a kernel SVM, a linear SVM and logistic regression for a given dataset.
Two classes, one separating line — and an infinite family of candidates. The SVM adds one tie-breaker: leave the largest possible buffer to the nearest points of each class.
A classifier is a hyperplane: w·x + b = 0. The vector w points perpendicular to it, and b shifts it. For any data point xᵢ, the signed distance to the hyperplane is (w·xᵢ + b) / ‖w‖; with labels yᵢ ∈ {−1, +1}, the quantity yᵢ·(w·xᵢ + b) is positive on the correct side and measures confidence up to the scale of w.
Since multiplying w and b by any positive number leaves the boundary unchanged, we are free to fix the scale. The standard choice pins the nearest points at exactly 1: yᵢ·(w·xᵢ + b) ≥ 1. With that convention the distance from the boundary to each margin rail is 1/‖w‖, so the margin width is 2/‖w‖. Maximizing it is the same as minimizing ½‖w‖² — a convex quadratic program with exactly one global solution.
Drag any point, or pick one and use the sliders. The boundary is recomputed from scratch — it is the unique line that maximizes the width of the band while keeping the classes outside it.
The solver sweeps 720 directions and refines the winner, so the printed margin matches the drawing. In the solver’s scaling, every support vector satisfies y·(w·x + b) = 1 and every other point has y·(w·x + b) > 1.
Distance. Take a point x and the hyperplane through the origin with normal w. The projection of x onto w is (w·x / ‖w‖²)·w. Slide a copy of the hyperplane to touch x: its offset is the length of that projection, |w·x| / ‖w‖. Adding a shift b changes w·x to w·x + b, so the distance is |w·x + b| / ‖w‖.
Margin. Choose the scaling so every point satisfies yᵢ(w·xᵢ + b) ≥ 1. The rail on the + side is the line w·x + b = 1; the rail on the − side is w·x + b = −1. Both are parallel to the boundary, and each is 1/‖w‖ away from it, so the street width is 2/‖w‖. Maximizing 2/‖w‖ is equivalent to minimizing ½‖w‖² (same minimizer, friendlier derivative), giving the primal problem:
Every constraint is linear and the objective is a convex bowl, so this is a convex quadratic program: one global optimum, no local minima, no random restarts. That guarantee is a large part of why SVMs were trusted long before neural networks were.
A trained SVM ignores almost all of its training data. The points sitting exactly on the margin rails are the support vectors, and they alone determine w and b.
Look again at the street: the nearest + point and the nearest − point touch the rails. Those touching points satisfy y·(w·x + b) = 1 exactly and are called the support vectors. Every other point has y·(w·x + b) > 1: it sits strictly outside the street and contributes nothing. Move it, delete it, add ten more like it — the boundary does not budge. In the language of the dual problem, support vectors are exactly the points with a nonzero coefficient αᵢ; everything else has αᵢ = 0.
This sparsity has two consequences. At prediction time an SVM only needs the stored support vectors and their α values — not the full training set. And the count of support vectors is a bound on generalization error: the smaller the fraction of training points that end up supporting the boundary, the more confidently the model has separated the classes.
Four points, two per class, all on the line y = x. Uncheck points and watch the exact max-margin solution, the support-vector list and the dual weights α update. Non-support vectors have α = 0: uncheck them and nothing moves.
Checkbox edits change the dataset; the solver recomputes w, b and the margin exactly for the displayed points.
Take a tiny dataset, all on the line y = x: two + points at (1, 1) and (2, 2), two − points at (−1, −1) and (−2, −2). By symmetry the widest street is perpendicular to that line: direction u = (1, 1)/√2.
Now recover the dual weights from w = Σ αᵢyᵢxᵢ and Σ αᵢyᵢ = 0. Only the two support vectors can have αᵢ > 0: w = α₁(1, 1) + α₃(1, 1) = (α₁ + α₃)(1, 1), and Σαᵢyᵢ = α₁ − α₃ = 0, so α₁ = α₃ = 0.25. Check primal = dual: ½‖w‖² = ½(0.25 + 0.25) = 0.25, and the dual objective Σαᵢ − ½ΣΣ αᵢαⱼyᵢyⱼ(xᵢ·xⱼ) works out to 0.5 − 0.25 = 0.25 as well — strong duality, verified numerically.
Delete the outer point (2, 2) and nothing changes. Delete the inner point (1, 1) instead and the street must stretch to reach (2, 2): w = (1/3, 1/3), b = −1/3, margin = 3√2 ≈ 4.243, with α = 1/9 on the two remaining support vectors. Same data, one deletion, a completely different boundary — that is the definition of a support vector.
You delete a training point that lies far from the boundary, well outside the margin. What happens to the SVM's decision boundary?
A hard margin demands a clean separation and gives up when the data overlaps. The soft margin keeps the widest street it can and pays a penalty for every point that steps inside.
Hard-margin constraints yᵢ·(w·xᵢ + b) ≥ 1 are unsolvable the moment a + point and a − point are close enough that no line separates them, and nearly unsolvable with one noisy outlier. The fix is one new variable per point: a slack ξᵢ ≥ 0 that relaxes its constraint to yᵢ·(w·xᵢ + b) ≥ 1 − ξᵢ. A point outside the street has ξᵢ = 0; a point inside it has ξᵢ equal to how far past the rail it stands; a misclassified point has ξᵢ > 1.
Training now minimizes two things at once: the width term ½‖w‖² and the total slack, with C deciding the exchange rate:
Because ξᵢ = max(0, 1 − yᵢ·f(xᵢ)) at the optimum, the two-line problem above is equivalent to the unconstrained objective ½‖w‖² + C·Σ max(0, 1 − yᵢ·f(xᵢ)) — hinge loss plus L2 regularization, which is the subject of the next chapter. One warning before you tune anything: C is the inverse of a regularization strength. Large C means less regularization (the model contorts to fit), small C means more (the model stays smooth and may underfit).
One noisy + point at (1.0, −0.8) sits inside the − cluster. Slide C from cheap fines (left) to expensive fines (right) and watch the SVM choose between a wide margin and a clean training set.
Small C → wide margin, many violations (underfitting). Large C → margin collapses, zero violations, risk of overfitting. The objective printed in the footer is the exact quantity the solver minimizes.
These are the lab’s 11 points and the exact output of its dual SMO solver (same algorithm as Platt’s Sequential Minimal Optimization, run to KKT convergence). Only the noisy + point at (1.0, −0.8) changes story.
Follow the margin column: 4.586 → 0.367, a 12× collapse, while violations drop 6 → 0. At C = 0.05 the model shrugs at the noisy point and keeps a broad, safe boundary; at C = 50 it rotates the boundary until that single point is respected, and the street squeezes into a sliver. Both are optimal — for different prices of slack. That is why C is a hyperparameter you tune with validation data, not a setting with one right answer.
You switch an SVM from C = 0.1 to C = 100 on a noisy dataset. What do you expect?
The soft-margin problem can be written as an unconstrained sum of per-point penalties. Those penalties are hinge-shaped: linear while you are in trouble, exactly zero once you are outside the margin.
Define the margin score of a point as z = y·f(x) where f(x) = w·x + b. Its hinge loss is max(0, 1 − z). Three regimes, one formula:
Compare it with the logistic loss log(1 + e^(−z)) from Lesson 03. Logistic loss is smooth and never exactly zero — every point keeps voting forever. Hinge loss is zero with zero gradient on the whole half-line z > 1, so safely classified points drop out of the solution entirely. That is the loss-level explanation of support vectors: training with hinge loss produces sparse models.
Move the vertical line along z = y·f(x). Outside the margin (z ≥ 1) the hinge is flat at zero; inside it falls with slope −1. The 0-1 loss is flat almost everywhere — no slope, no direction, no learning.
The 0-1 loss counts mistakes, which is what we care about — but it is flat (zero gradient) on both plateaus, so gradient descent gets no signal. Hinge and logistic are stand-ins with usable slopes.
In the constrained problem, the best feasible slack for a point is its shortfall below the rail: ξᵢ = max(0, 1 − zᵢ). Substituting that into ½‖w‖² + CΣξᵢ removes every ξ and every constraint, leaving the unconstrained objective the source code actually minimizes:
Numeric check. Take z = 0.15 from w = (0.2, −0.3), b = 0.1 and the point x = (1, 0.5), y = +1. Compute z = 0.2·1 + (−0.3)·0.5 + 0.1 = 0.15, so the hinge is 0.85. With C = 1 and learning rate 0.1:
One step moved the point’s margin from 0.15 to 0.37. If z had already been 1.5, both gradients would have been zero for this point and nothing would have changed — the difference between hinge and logistic loss in a single example.
Which training points contribute a nonzero gradient under hinge loss?
The same SVM has a second form in which the weights disappear and the data appears only as pairwise dot products. That one change is what lets a linear method draw nonlinear boundaries.
Every constrained problem has a companion dual, and Lesson 18 built the machinery: introduce one multiplier per constraint, form the Lagrangian, and — for convex problems — the two optima coincide (strong duality). Applying it to the soft margin, with multipliers αᵢ for the margin constraints and μᵢ for ξᵢ ≥ 0, gives:
Read the prediction line carefully: x never appears alone — it appears only dotted with training points. The weight vector is not needed at prediction time at all. And wherever the primal used a dot product xᵢ·xⱼ, the dual may replace it with any kernel function K(xᵢ, xⱼ) that behaves like a dot product in some feature space. That substitution is the kernel trick: a linear boundary in a richer space becomes a nonlinear boundary in the original one, and the richer space is never built.
Ten points at radius 1 (squares, class −) and ten at radius 2 (circles, class +). In the input plane no line separates them. Add a third coordinate z = x² + y² and the two rings become two flat levels: z = 1 and z = 4.
The lift is a teaching example of a feature map, not what an SVM literally computes: a kernel SVM reaches this decision boundary by evaluating similarities, never the third coordinate.
Write the Lagrangian with multipliers αᵢ ≥ 0 on the margin constraints and μᵢ ≥ 0 on the slacks:
Why the dot products are all the dual needs. Look at the one place training data appears in the dual: the pairwise product xᵢ·xⱼ. If those numbers can be produced by a similarity function K, the optimizer never asks what φ looks like. Concretely, with the polynomial kernel of degree 2, K(x, z) = (x·z + 1)² = φ(x)·φ(z) for the explicit feature map
The RBF kernel is the extreme case: exp(−γ‖x−z‖²) = exp(−γ‖x‖²)·exp(2γ·x·z)·exp(−γ‖z‖²), and the middle factor expands as a polynomial of every degree. Its feature space is infinite-dimensional — and it is still evaluated in a handful of floating-point operations.
The RBF kernel measures closeness. Gamma decides what “close” means: a wide neighbourhood that blends many points together, or a narrow one where every point is its own island.
K(x, z) = exp(−γ‖x − z‖²) equals 1 when two points coincide, 0.6065 at squared distance 0.5/γ, and 0.000335 at squared distance 8/γ. For γ = 0.5 that means 0.6065 at squared distance 1 and 0.000335 at squared distance 16 (distance 4). So γ is a zoom knob on distance: small γ makes distant points count as similar (the boundary is smooth and nearly global), while large γ makes only immediate neighbours count (the boundary grows spiky and can carve a private island around each point). Same kernel, same data, wildly different geometry.
The polynomial kernel trades locality for degree: (x·z + c)^d encodes feature interactions up to order d, which is why d = 1 is the linear kernel and d = 2 already draws conic boundaries (as the lift lab showed for x² + y²). The default advice is unglamorous and correct: start with a scaled linear SVM, and only reach for RBF when the linear boundary is clearly wrong.
The same machinery works for regression. Support Vector Regression fits a tube of width ε around the data: points inside the tube cost nothing, points outside pay linearly, and the support vectors are the points on or outside the tube wall. Wider tube → fewer support vectors → smoother fit. And the situations where SVMs still beat the crowd are concrete: hundreds to a few thousand training points, very high-dimensional sparse features, binary problems with a clean margin, one-class anomaly detection, and any setting where you want a training-time guarantee rather than a hope. Deep learning owns the other end — images, audio, and millions of rows with learnable features.
This trains a real RBF-kernel SVM (C = 1) with the same dual solver as the C lab. Slide γ from smooth to spiky and watch the decision regions change shape.
Rule of thumb: γ small → each point influences a wide area (smooth, underfits). γ large → each point only recognizes its own neighbourhood (spiky, overfits). scikit-learn’s default gamma="scale" is 1/(features · variance of X).
Similarity values at three distances for γ = 0.5 (the lab defaults to γ = 0.5 and lets you move it):
The scaling rule. σ of the features sets the unit of ‖x − z‖², so γ and feature scale are entangled: doubling every feature quadruples its contribution to distance, which is equivalent to multiplying γ by four. That is why the standard pipeline is standardize → SVM → tune C and γ together, never tune one without the other.
from sklearn.svm import SVC, LinearSVC
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
# small data, nonlinear boundary: RBF kernel, dual solver
clf = Pipeline([
("scaler", StandardScaler()),
("svm", SVC(kernel="rbf", C=1.0, gamma="scale")),
])
clf.fit(X_train, y_train)
print(f"accuracy: {clf.score(X_test, y_test):.4f}")
print(f"support vectors: {clf['svm'].n_support_}")
# tall data (n large, d small): primal solver, O(n) per epoch
fast = Pipeline([
("scaler", StandardScaler()),
("svm", LinearSVC(C=1.0, max_iter=10000)),
])You train an RBF SVM and it gets 100% training accuracy, but the decision regions are ragged islands around individual points. What is the most likely cause, and what should you try first?
Answer before you look. The C question and the support-vector question are exactly the ones that separate a memorized answer from a working understanding.
0 / 5 answered · 0 correct
01What are support vectors in an SVM?
02What does the SVM maximize when finding the decision boundary?
03What happens when you increase the C parameter in an SVM?
04How does the kernel trick enable SVMs to learn nonlinear boundaries?
05Hinge loss is zero when y·f(x) ≥ 1. What does this mean in terms of classification?
Click a card to swap the lazy description for what it actually means.
Four short problems with exact numbers. Try first; a worked answer is one click away.
All four points lie on the line y = x, so the widest street runs perpendicular to it. Take the unit direction u = (1, 1)/√2; the projections u·x are √2 and 2√2 for the positives, −√2 and −2√2 for the negatives. The gap between the nearest classes is √2 − (−√2) = 2√2, so half the gap is g = √2 and the boundary sits at the midpoint, 0. Rescaling gives w = u/g = (0.5, 0.5) and b = 0, with margin 2/‖w‖ = 2√2 ≈ 2.828. Support vectors: (1, 1) and (−1, −1), where y·f(x) = 1 exactly; the outer points have y·f(x) = 2. Deleting (2, 2) changes nothing (y·f = 2 > 1, α = 0). Deleting (1, 1) forces the boundary to move toward the remaining positive point: g = 3√2/2, w = (1/3, 1/3), b = −1/3, margin = 3√2 ≈ 4.243, and now (2, 2) and (−1, −1) are the support vectors. Exact dual weights for the four-point problem: α = (0.25, 0, 0.25, 0), and the dual objective is 0.25 = ½‖w‖².
The C-lab solver (dual SMO on the same 11 points) gives: C = 0.05 → w = (−0.307, 0.310), b = 0.139, margin 4.586, 6 violations, noisy-point margin −0.416 (slack 1.416). C = 0.5 → w = (−0.469, 0.542), b = 0.249, margin 2.791, 2 violations, noisy-point margin −0.653 (slack 1.653). C = 5 → w = (−1.381, 1.208), b = 2.020, margin 1.090, 1 violation, noisy-point margin −0.327. C = 50 → w = (−4.103, 3.590), b = 7.974, margin 0.367, 0 violations, noisy-point margin exactly 1.000 (it became a support vector, sitting on the margin). The pattern is the whole trade-off: as C grows, the street narrows from 4.59 to 0.37 and the model contorts itself to stop paying fines. At C = 0.05 the model has 6 violations and still gets 10/11 training points right; at C = 50 it gets all 11 but the margin is 12× narrower.
A line through the origin has equation ax + by = 0; restricted to the x-axis it is ax = 0, so its sign is constant for all x > 0 and cannot put (1, 0) and (2, 0) on opposite sides. Kernel values: K((1,0), (1,0)) = exp(0) = 1; K((2,0), (2,0)) = 1; K((1,0), (2,0)) = exp(−0.5·1) = e^−0.5 ≈ 0.6065. Use the signed vote f(x) = Σ αᵢyᵢK(xᵢ, x) with αᵢ = 1 for both points: at x = (1, 0), f = 1·(−1)·1 + 1·(+1)·0.6065 = −0.3935 < 0 → predict −1 ✓. At x = (2, 0), f = 1·(−1)·0.6065 + 1·(+1)·1 = +0.3935 > 0 → predict +1 ✓. Being close in input space makes K near 1 and far makes it near 0; the kernel's similarity geometry, not the original coordinates, separates the classes. A real RBF SVM would also learn the αᵢ, but the direction of the fix is exactly this.
z = −1: hinge 2.000, logistic 1.3133, hinge slope −1 (subgradient), logistic slope −0.7311. z = 0: hinge 1.000, logistic 0.6931, slopes −1 and −0.5000. z = 0.5: hinge 0.500, logistic 0.4741, slopes −1 and −0.3775. z = 1: hinge 0.000, logistic 0.3133, hinge slope 0 (subgradient lies in [−1, 0]), logistic slope −0.2689. z = 2: hinge 0.000, logistic 0.1269, slopes 0 and −0.1192. Only the first three contribute to the hinge gradient: the points with z ≥ 1 are done, and their weights in the solution are exactly zero. Logistic loss gives every point a nonzero slope forever, so every training point keeps its vote. That is why an SVM's prediction cost depends on the number of support vectors rather than the size of the training set.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 02, Lesson 05) and the Math Foundations Notebook reference build. Interactive figures, the exact hard-margin and SMO solvers, the kernel-lift and gamma labs, the dual console, worked exercise answers and the numeric checks are original to this page. Every lab runs in your browser.