Keypoint detection looks like six different problems — body, face, hands, animals, anatomy, a gripper’s object frame — and it is one: detect K named points and return their coordinates. Learn why every serious model paints a Gaussian per point, how a two-channel direction field decides which elbow belongs to which shoulder, and how OKS turns “how close” into a single number. Then read the three tools you will actually ship.
Pose, face landmarks, hands, animals, anatomy, a robot's object frame: the names change, the interface does not. K ordered (x, y, score) triples per instance, and the ordering is the contract — index 9 is always the left wrist, which is why a horizontal flip needs a keypoint swap and not just a mirror. Human body: 17 (COCO) or 33 (MediaPipe). Face: 68 or 478. Hand: 21.
K = 17 · 33 · 68 · 21 · 478 — one problem, five shapesOne H × W map per joint, painted with a Gaussian at the true location: target = exp(−d²/2σ²) with σ ≈ 2 cells at H/4, and per-pixel MSE as the loss. The target's mass is 2πσ² = 25.13 at σ = 2, and the peak-cell cost of a miss jumps 54× between 1 and 4 cells. At inference the decode is an argmax — 1.61 image px off on the lab's off-grid peak at stride 4 — refined to 0.061 px by a parabola through the peak cell.
σ = 2 cells → 8 image px · argmax 1.61 px → parabola 0.06 pxTop-down detects people and runs a pose model per crop — accuracy leader, cost 8 + 3N ms. Bottom-up predicts every keypoint and a 2-channel direction field per limb in one pass, then ranks candidate connections by the line integral of that field — constant 26 ms for any crowd, and the parse is a greedy matching. Then OKS scores the result: a rigid 10 px shift is 0.497 on a 50 × 100 person and 0.776 on a 100 × 200 one, because the score divides by the box area.
8 + 3N vs 26 ms · ∫PAF·u: 1.00 true vs 0.03–0.42 wrong · OKS 10 px → 0.50A pose model never outputs a skeleton: it outputs K heatmaps that answer where and 2C field channels that answer which one, and everything you ship after that — argmax or parabola, greedy matching or a crop, PCK or OKS — is the bookkeeping that turns those numbers into the figure on screen.
By the end you will be able to say when top-down (8 + 3N ms) beats bottom-up (a constant 26 ms) — and the arithmetic says the crossover is 6 people; write the Gaussian target by hand and predict a decoder’s error in image pixels from its σ and its stride; explain how a PAF line integral separates a true limb (≈1.00) from the closest wrong pair (0.03–0.42) and where occlusion breaks it; compute OKS for a rigid shift (0.4967 at 50 × 100, 0.7764 at 100 × 200) and explain the 11.72× strictness ratio between the nose and the ankle; and pick between MediaPipe’s 33 normalised-plus-world landmarks, MMPose’s configurable 17, and a single-shot detector head — with the output format you will actually parse.
Pose, face landmarks, hand tracking, animal pose, anatomy, a gripper’s object frame: detect K named points on an object and return their (x, y) — or (x, y, z). The names change, the number of points changes, the interface does not.
A keypoint is a specific, ordered point on an object: a joint, a corner, an anatomical landmark. A pose is a set of keypoints that belong to one instance, in a fixed order. That order is the whole trick. The model never outputs “a skeleton” — it outputs numbers, and the numbers only mean something because everyone agreed on the index map in advance.
Here is the map that this lesson uses, COCO’s 17 body keypoints — the annotation standard behind almost every pose benchmark you will read. Note which indices are left and which are right: everything a horizontal flip touches depends on that.
The same task appears at very different scales. A hand is 21 points at arm’s length; a face mesh is 478 points on a 200-pixel head; a whole-body annotation is 133 points. The engineering question is not “which architecture” first — it is how many instances, how fast, and how precise. A single-image, single-person pose is a ~20 ms problem — the latency model in the next chapter puts a one-person top-down pipeline at 11 ms and a bottom-up pass at 26 ms — while a crowd of fifty at 30 fps is a different problem with a different architecture. The next chapter is exactly that split.
| Task | K | What the points are |
|---|---|---|
| Human body · COCO-17 | 17 | The benchmark body skeleton: nose, eyes, ears, shoulders, elbows, wrists, hips, knees, ankles. |
| Human body · MediaPipe Pose | 33 | The production skeleton: COCO-17 plus feet, hands and a six-point face. |
| Face · iBUG 300-W | 68 | Jaw line, brows, eyes, nose, mouth — the 68-point annotation standard. |
| Face · MediaPipe Face Mesh | 478 | A full 3D face mesh with iris points; used for AR filters and gaze. |
| Hand · MediaPipe / COCO-WholeBody | 21 | Wrist plus four joints per finger — the basis of gesture control. |
| Whole body · COCO-WholeBody | 133 | 17 body + 6 feet + 68 face + 42 hands in one annotation set. |
| Animal · AP-10K / Animal-Pose | 17–20 | Quadruped skeletons; the same heatmap regressor, a different ordering. |
| Object · BOP line | 8 corners | The 3D bounding-box corners a robot needs to grasp a known object. |
Pose is also the invisible layer under a surprising amount of software: motion capture for games and film turns a performer into joint angles; fitness apps count reps and score form; sports analytics measures stride and joint angles frame by frame; gesture control and AR try-on track hands and bodies in real time; robotic grasping needs the corners of an object; medical imaging measures anatomy — a hip angle, a spine curve — from landmarks. One interface, many products.
Top-down detects people first and runs a keypoint model on each crop: the accuracy leader, with a cost that grows one person at a time. Bottom-up predicts every keypoint in one pass and then groups them: constant time, crowd-proof, and one association problem harder.
Top-down — detect person boxes, crop each one, run a per-person keypoint model on the crop, map the coordinates back into the image. The box hands the model a huge prior: a person is roughly centred, roughly upright, and occupies most of the crop, so the network can spend all its capacity on joints instead of on search. That is why top-down owns the accuracy leaderboard: HRNet (the reference architecture that keeps a high-resolution stream alive through the whole network) and ViTPose (a plain vision transformer as the backbone) are both top-down. The price is arithmetic: one detector pass plus one pose pass per person, so a stadium full of people is a queue of crops.
Bottom-up — one forward pass predicts every keypoint’s heatmap for every person in the frame plus an association field, and a separate grouping step decides which elbow belongs to which shoulder. OpenPose made this work with Part Affinity Fields; HigherHRNet and modern single-shot models (YOLOv8-pose, which puts keypoints in the detector’s head) continue the line. Cost is constant in the number of people — the frame is expensive, the crowd is free — at the price of a harder association problem and, historically, about ten points of AP behind top-down on COCO.
Read that table the way an engineer does: the decision is not a preference, it is a crowd size. Below the crossover, top-down is both more accurate and fast enough; above it, no amount of tuning makes a per-person queue fit in 33 ms. The practical hybrid is what production systems actually do — bottom-up to find everyone, top-down for the individuals the application cares about.
Two failure modes are worth memorising because they show up as bugs, not as bad numbers. Top-down inherits the detector’s mistakes: a missed box is a missing person, and two people merged into one box produce one skeleton with limbs from both. It also inherits the crop’s aspect ratio: the standard 256 × 192 window truncates a tall person’s feet and a raised hand, so keypoints outside the crop come back missing or fabricated. Bottom-up inherits occlusions: when two people overlap, the association field is ambiguous exactly where the integral is weakest — chapter 04 is the mechanism.
One frame, two pipelines, a shared clock. Top-down pays one detector pass and then a fixed cost per person; bottom-up pays one big pass and a constant grouping step. Move the crowd size and watch the crossing point: below it the two-stage pipeline wins on latency, above it only the single pass survives.
Latency is a teaching model, not a benchmark: a person detector ≈ 8 ms, a 256×192 pose head ≈ 3 ms, one bottom-up pass ≈ 22 ms, grouping ≈ 4 ms on a mid-range GPU at 720p. The accuracy half of the trade is the source’s claim — top-down (HRNet, ViTPose) leads the COCO leaderboard, bottom-up (OpenPose, HigherHRNet) wins on throughput — and it is not modelled here.
A stadium camera sees 60 players at once and the broadcaster wants skeletons at 30 fps. Which pipeline family do you reach for first, and why?
A keypoint model outputs a heatmap per joint: an H × W grid whose peak sits on the joint. Training is per-pixel MSE against a Gaussian painted at the true location; inference is a single argmax — or a sub-pixel refinement if you care.
The obvious approach is to regress (x, y) directly: a small head, two numbers per joint, MSE loss. Every modern pose model refuses. The reasons are structural, not stylistic. A convolutional stack builds a spatial feature map — its values are indexed by where they came from. Asking it to collapse that map into two scalars at the end throws away the alignment between the features and the answer. A heatmap keeps it: every output cell is a small classifier asking “is the joint here?”, and the loss is applied where the joint actually is.
The second reason is the loss landscape. With direct regression, a prediction that is 3 px off gets the same gradient direction as one that is 300 px off — the error is a distance, not a location. With a Gaussian target, being 1 px off costs almost nothing and being 10 px off costs a lot; the loss is smooth and local, which is exactly what a first-order optimiser wants.
The target is a picture, and the loss is per-pixel MSE against it: L = (1 / K·H·W) · Σ (pred − target)². Read the numbers and the loss stops looking arbitrary. A one-cell miss on a σ = 2 target moves the peak cell from 1.000 to 0.882, so that cell contributes (1 − 0.882)² = 0.0139. A four-cell miss moves it to 0.135 and contributes (1 − 0.135)² = 0.748 — 54× the error for 4× the distance, because the loss is the square of a quantity that decays like a bell. That, and not fashion, is why the model learns to put a blob in the right place before it learns to make the blob sharp.
A one-hot target (1 at the true cell, 0 everywhere else) is an extreme classifier: it gives a gradient signal on exactly two cells (the truth and the current prediction) and nothing anywhere else. Worse, it claims a precision the labels do not have. COCO’s annotators place a wrist within a few pixels, not to the pixel, and the benchmark publishes exactly that uncertainty as a per-keypoint sigma — 0.026 for the nose, 0.089 for the ankle, the same constants the OKS metric uses in chapter 05. A Gaussian target with σ ≈ 2 px says “the joint is near here and I am not sure to the cell”, which is true, and it turns that uncertainty into a smooth loss landscape. Annotation variance and training-target width are the same idea seen from two sides.
Resolution is a budget decision. Heatmaps are predicted at H/4 — one output cell per 4 × 4 input pixels — because the last layers’ compute scales with the map area and H/4 keeps the memory sane while leaving joints several cells apart. A 640 × 480 frame becomes a 160 × 120 map; 17 keypoints of float32 is 17 × 160 × 120 × 4 bytes = 1.31 MB per image, and a single 256 × 192 crop — the top-down standard — is a 64 × 48 × 17 tensor of 52,224 values, 209 KB. At H/4 the whole decoding problem is: which of 160 × 120 cells holds the peak?
import numpy as np
def gaussian_heatmap(size, cx, cy, sigma=2.0):
yy, xx = np.meshgrid(np.arange(size), np.arange(size), indexing="ij")
return np.exp(-((xx - cx) ** 2 + (yy - cy) ** 2) / (2 * sigma ** 2)).astype(np.float32)
hm = gaussian_heatmap(64, 32, 32, sigma=2.0)
print(f"peak {hm.max():.3f} at ({hm.argmax() % 64}, {hm.argmax() // 64})")
# peak 1.000 at (32, 32) — the index arithmetic is (i % W, i // W)At inference the decode is one line — hm.argmax(), then (i % W, i // W) — and it is exactly as precise as the heatmap grid: an argmax can be up to half a cell off (exact when the peak sits at a cell centre), and at H/4 half a cell is 2 image pixels. Two refinements recover the continuous peak:
The lab makes the difference concrete on a perfect Gaussian with its peak at cell (6.35, 11.2), all errors below quoted as 2-D image pixels at stride 4: argmax returns (6, 11), 0.403 cells = 1.61 px off; the parabola returns (6.339, 11.190), 0.015 cells = 0.061 px; soft-argmax at β = 10 returns (6.395, 11.200), 0.045 cells = 0.18 px; and the source’s 0.25 × difference nudge returns (6.038, 11.022), 1.44 px — better than the integer it started from, but by only 0.17 px. That last number is worth keeping honest: the 0.25 coefficient is a conservative heuristic that always under-corrects on a Gaussian (for σ = 2 the exact first-order coefficient is ≈2.27), so it is fine as a nudge and wrong as a measurement. The parabola fit is the version to ship.
Drag the joint inside the grid (or use the sliders) to move the true location, then watch the four decoders read it back. This is the training target — a perfect Gaussian — so the error you see belongs to the decoder alone. A real network adds its own error on top.
The target is not normalised: its total mass is 2πσ², so a σ = 4 target carries 4× the mass of σ = 2 while the peak stays at 1.000 — which is why the MSE number moves when you change σ.
| d (cells) | target value | peak-cell MSE contribution | what the model is being told |
|---|---|---|---|
| 0 | 1.000 | 0.0000 | this is the joint — keep the peak at 1.000 |
| 1 | 0.882 | 0.0138 | close: a small nudge in the right direction |
| 2 | 0.607 | 0.1548 | close: a small nudge in the right direction |
| 3 | 0.325 | 0.4561 | wrong location: the peak itself is missing |
| 4 | 0.135 | 0.7476 | wrong location: the peak itself is missing |
| 5 | 0.044 | 0.9141 | far away: this cell should be near zero |
| 6 | 0.011 | 0.9779 | far away: this cell should be near zero |
A model outputs H/4 heatmaps for a 640 × 480 frame. Its predicted peak sits 3 cells to the left of the ground-truth peak. How far off is the joint in the image?
A bottom-up model predicts 17 heatmaps for a whole frame. Two people produce 34 peaks and nothing that says which 17 belong together. Part Affinity Fields answer that question with a direction, and a line integral turns the answer into a number you can sort.
Bottom-up pose has one extra problem compared to top-down: the crops are gone. Instead of a per-person window that implies “these joints belong together”, the model sees every person at once, and 34 candidate points arrive in a bag. Association is the step that turns a bag of peaks into instances — and the naive version (nearest neighbour) fails the moment two people stand close, because the nearest elbow is often the other person’s.
Part Affinity Fields (Cao et al., OpenPose) solve it by predicting, alongside the K heatmaps, a 2-channel unit vector field for every limb type — one channel for x, one for y, pointing from the source joint to the target joint of that limb. For the 16 body edges of COCO-17 that is 32 extra channels; unsurprisingly the PAF head is the bigger half of the network. The field is trained on the same idea as the heatmaps: at every pixel along a true limb, the target is the limb’s unit direction; elsewhere it is whatever the annotation says (usually zero), and the loss is per-pixel MSE.
The matching step is then almost anticlimactic. For each limb type, score every candidate source × target pair with E, sort descending, and greedily accept a pair while both its endpoints are still free. OpenPose describes this as a relaxation of maximum-weight bipartite matching: the greedy version is within a hair of the optimal assignment and runs in polynomial time, which is why the whole parse is a few milliseconds instead of a research project. After all limb types are parsed, the accepted edges connect keypoints into components — each component is an instance.
The elegance is that the association is a graph problem on top of a regression. There is no per-person crop, no iteration over detected people, no NMS over skeletons. The same forward pass handles two people or twenty, and the only cost that grows with the crowd is the matching, which is polynomial and tiny compared to the convolutions.
Two honest caveats. First, the integral is weakest exactly where the task is hardest: two people crossing put two limb directions in the same corridor, and the field’s average points somewhere between them, so a low threshold can link the wrong pair. Second, the modern alternative to PAFs is associative embedding (Newell et al.), which gives every keypoint a scalar “tag” and groups points whose tags are close — cheaper to parse, less robust in dense crowds. Both ideas appear in HigherHRNet-class models; a from-scratch pipeline that you can debug by hand still starts with PAFs.
Two people, one limb type at a time. Every source × target pair gets a line integral through the field; the greedy parse walks the scores from the top and accepts a pair only if both endpoints are free. Slide the two people together and watch the closest wrong answer climb.
The field is a teaching model: a unit vector inside each limb’s corridor, zero outside it. A trained PAF is a 2-channel convolution regressed per pixel and is nonzero almost everywhere — but the ranking it produces on these four pairs behaves the same way.
What does the line integral E(a, b) actually measure?
Two metric families dominate. PCK is a threshold on the distance to each joint, normalised by the person’s size. OKS replaces the threshold with a per-joint exponential and is what COCO’s mAP@OKS 0.5:0.95 reports.
PCK — Percentage of Correct Keypoints. A joint counts as correct if its distance to ground truth is at most α × L, where L is a normaliser: the person’s box, the torso, or the head diameter (MPII’s “PCKh”). PCK is then the fraction of visible joints that passed, and PCK@0.2 means α = 0.2. The normaliser is doing real work: on a 50 × 100 box the threshold is 0.2 × 100 = 20 px, so a wrist predicted 12 px off is correct; give the same model a 25 × 50 person in the distance and the bar drops to 10 px and that same absolute error fails. Always read which L a reported PCK uses before comparing two numbers.
OKS — Object Keypoint Similarity. The keypoint analogue of IoU, and the reason COCO pose numbers are comparable at all. For every visible joint, form a squared distance, scale it by the person’s box area and by a per-joint constant κi, and exponentiate:
Read the two terms: after the same 5 px error the ankle keeps 2.3× more of its score than the nose, and because the penalty is exponential the gap widens fast — at 20 px the nose term is already ≈0 while the ankle still holds 0.28. The reason is annotation mechanics, not anatomy: a nose is a crisp, visually unambiguous point, while an ankle is a joint observed through clothing, from one side, at an angle that changes with the foot. The constants encode how sure the annotators were, and the metric inherits that judgement. It is also why “the model is good at 0.9 AP” needs the per-joint breakdown before you trust it.
Because OKS is a similarity in [0, 1] exactly like IoU, the AP machinery carries over unchanged: sort predictions by instance score, match them greedily to ground-truth poses in score order, and sweep 10 OKS thresholds from 0.50 to 0.95 in steps of 0.05. The mean over those thresholds is AP@[.5:.95] — the number in every pose leaderboard — with AP@0.5 (is there a skeleton?) and AP@0.75 (is it precise?) reported separately because they disagree interestingly. A model that finds everyone but jitters at the wrists has high AP@0.5 and low AP@0.75.
The scale term is the part beginners miss: OKS divides by the box area, so it is resolution- and zoom-invariant. A rigid 10 px translation of a perfect skeleton is a disaster on a small person and a mild irritation on a large one. That invariance is what makes a single number meaningful across a dataset where people range from 30 to 900 pixels tall — and it is exactly the number the lab below lets you compute by hand.
| uniform shift of the whole skeleton | 50 × 100 person | 100 × 200 person | AP thresholds passed |
|---|---|---|---|
| 2 px on every joint | 0.9541 | 0.9881 | 10 / 10 at 50 × 100 |
| 5 px on every joint | 0.7764 | 0.9304 | 6 / 10 at 50 × 100 |
| 10 px on every joint | 0.4967 | 0.7764 | 0 / 10 at 50 × 100 |
| 20 px on every joint | 0.1633 | 0.4967 | 0 / 10 at 50 × 100 |
| joint | κ | OKS term at 5 px (50 × 100 box) | relative strictness (κ / 0.026)² |
|---|---|---|---|
| 0 · nose | 0.026 | 0.3967 | 1.00× |
| 1 · left eye | 0.025 | 0.3679 | 0.92× |
| 3 · left ear | 0.035 | 0.6004 | 1.81× |
| 5 · left shoulder | 0.079 | 0.9047 | 9.23× |
| 7 · left elbow | 0.072 | 0.8864 | 7.67× |
| 9 · left wrist | 0.062 | 0.8499 | 5.69× |
| 11 · left hip | 0.107 | 0.9469 | 16.94× |
| 13 · left knee | 0.087 | 0.9207 | 11.20× |
| 15 · left ankle | 0.089 | 0.9241 | 11.72× |
OKS is the pose version of IoU: for every visible joint, an exponential penalty scaled by that joint’s constant and the person’s box area, then averaged. Shift the whole skeleton by a fixed number of pixels and watch the score — the same 10 px on a bigger person costs less, and the same 10 px on the nose costs 11.7× more than on an ankle inside the exponent.
The sigmas are COCO’s published annotation-variance constants (.026 nose … .089 ankle); the reference evaluator squares 2κ in the denominator, which is what this calculator does. Re-estimate the constants on your own labels or the metric will punish the wrong joints.
| joint | κ | exponent at 10 px | exp(−e) | |
|---|---|---|---|---|
| 0 · nose | 0.026 | 3.6982 | 0.0248 | |
| 1 · left eye | 0.025 | 4.0000 | 0.0183 | |
| 2 · right eye | 0.025 | 4.0000 | 0.0183 | |
| 3 · left ear | 0.035 | 2.0408 | 0.1299 | |
| 4 · right ear | 0.035 | 2.0408 | 0.1299 | |
| 5 · left shoulder | 0.079 | 0.4006 | 0.6699 | |
| 6 · right shoulder | 0.079 | 0.4006 | 0.6699 | |
| 7 · left elbow | 0.072 | 0.4823 | 0.6174 | |
| 8 · right elbow | 0.072 | 0.4823 | 0.6174 | |
| 9 · left wrist | 0.062 | 0.6504 | 0.5219 | |
| 10 · right wrist | 0.062 | 0.6504 | 0.5219 | |
| 11 · left hip | 0.107 | 0.2184 | 0.8038 | |
| 12 · right hip | 0.107 | 0.2184 | 0.8038 | |
| 13 · left knee | 0.087 | 0.3303 | 0.7187 | |
| 14 · right knee | 0.087 | 0.3303 | 0.7187 | |
| 15 · left ankle | 0.089 | 0.3156 | 0.7293 | |
| 16 · right ankle | 0.089 | 0.3156 | 0.7293 |
Same model, same 5-pixel error. Which joint drags the OKS of a 50 × 100 person down more — the nose or the ankle?
MediaPipe on the edge, MMPose in the research loop, YOLOv8-pose for real-time crowds: the same 17-or-33-number answer in three different json shapes. Read the format before you read the paper.
MediaPipe Pose Landmarker is the one that runs in your phone. Google’s pipeline is a two-model cascade with a person detector (224 × 224) followed by a landmark model (256 × 256) on the detected crop — the top-down pattern from chapter 02, compiled for on-device use with a MobileNetV2-class trunk and a 3D body model (GHUM) for the depth channel. It predicts 33 landmarks — COCO’s 17 body joints plus feet, hands and a six-point face — and returns each one in two coordinate systems: normalised image coordinates (x and y in [0, 1], z relative to the hips) and world landmarks in metres with the origin between the hips. Pick the lite / full / heavy bundle, set num_poses and the three confidence thresholds, and choose one of three running modes: IMAGE, VIDEO or LIVE_STREAM. Sub-10 ms per frame on a phone is the number that makes fitness apps possible.
import mediapipe as mp
from mediapipe.tasks import python
from mediapipe.tasks.python import vision
base = python.BaseOptions(model_asset_path="pose_landmarker_lite.task")
options = vision.PoseLandmarkerOptions(
base_options=base,
running_mode=vision.RunningMode.IMAGE,
num_poses=1,
)
landmarker = vision.PoseLandmarker.create_from_options(options)
result = landmarker.detect(mp.Image.create_from_file("runner.jpg"))
nose = result.pose_landmarks[0][0] # index 0 is the nose
print(nose.x, nose.y, nose.z, nose.visibility) # x, y in [0, 1] of the image
hip = result.pose_world_landmarks[0][23] # metres, origin between the hips
print(f"hip at ({hip.x:.3f}, {hip.y:.3f}, {hip.z:.3f}) m")MMPose (OpenMMLab) is the toolbox you use when you need a specific backbone, a custom keypoint set, or a number that a reviewer will check. It ships the full zoo — HRNet, ViTPose, HigherHRNet, associative-embedding models — as configs, with the training loop, augmentation and evaluation attached. The high-level entry point is an inferencer that takes a path or a frame and returns a prediction dict per instance.
from mmpose.apis import MMPoseInferencer
inferencer = MMPoseInferencer(pose2d="hrnet_w48_coco_256x192")
result = next(inferencer("crowd.jpg", return_vis=False))
person = result["predictions"][0][0]
print(person["keypoints"]) # 17 × [x, y] in image pixels
print(person["keypoint_scores"]) # 17 peak values — not calibrated probabilities
print(person["bbox"]) # the box the crop came fromYOLOv8-pose is the third corner: keypoints inside the detector’s head, one forward pass, no separate pose model and no association step — 17 COCO joints per person with a confidence each, NMS to thin the anchors, done. It is the fastest multi-person option in practice and the usual choice when the pipeline has to run at 30 fps on a modest GPU. The trade is the one chapter 02 described: a single-shot head has less capacity per person than a dedicated top-down crop network, so it lands between the two classics on accuracy and near the bottom-up models on latency.
Whatever you choose, the last accuracy knob is free: flipped-image test-time augmentation. Run the frame, run its mirror image, mirror the second prediction back with x′ = W − 1 − x, swap the left/right labels, and average the two heatmaps before the argmax. Averaging heatmaps before the decode beats averaging decoded coordinates, because the naive average of two noisy peaks can land in the valley between them. It is worth roughly a point of AP and costs exactly one extra forward pass — the standard trick on every leaderboard entry.
The swap is where the bookkeeping lives, and it is pure index arithmetic: eight pairs of indices for COCO-17, and the nose does not move. Left and right are conventions, not physics — the model has no idea which is which until you tell it with the flip pairs.
| flip pair | left | right | after mirroring |
|---|---|---|---|
| 1 ↔ 2 | left eye | right eye | x′ = W − 1 − x, then swap 1 and 2 |
| 3 ↔ 4 | left ear | right ear | x′ = W − 1 − x, then swap 3 and 4 |
| 5 ↔ 6 | left shoulder | right shoulder | x′ = W − 1 − x, then swap 5 and 6 |
| 7 ↔ 8 | left elbow | right elbow | x′ = W − 1 − x, then swap 7 and 8 |
| 9 ↔ 10 | left wrist | right wrist | x′ = W − 1 − x, then swap 9 and 10 |
| 11 ↔ 12 | left hip | right hip | x′ = W − 1 − x, then swap 11 and 12 |
| 13 ↔ 14 | left knee | right knee | x′ = W − 1 − x, then swap 13 and 14 |
| 15 ↔ 16 | left ankle | right ankle | x′ = W − 1 − x, then swap 15 and 16 |
| 0 | nose | — | mirror only: x′ = W − 1 − x |
Everything above is 2D pose: keypoints in image coordinates, at production quality for years. Estimating where the joints are in the world from one camera is the open problem. Three families of solutions, in increasing cost:
| route | example | input → output | when to use it |
|---|---|---|---|
| Lift 2D to 3D | VideoPose3D | a sequence of 2D joints (17 × 2) plus a time window → 17 joints × 3, camera-relative, in metres only if the input had scale | you already have a good 2D model and want cheap 3D; the 3D part is a small MLP, not a big network |
| Direct 3D regression | PyMAF, MHFormer | the RGB frames themselves → 3D joints, usually with a parametric body model attached | you need end-to-end accuracy and can afford a heavy model; the body model supplies the metric scale |
| Multi-view triangulation | CMU Panoptic rigs | several synchronised cameras → 3D joints with real, measurable accuracy — the ground truth others are trained against | you control the studio; also how the datasets that the other two routes learn from were built |
Four axes decide almost every pose deployment: how many people, how much latency, 2D or 3D, and whose keypoints. Pick a use case and read the tool, the pipeline it implies, the output format you will actually parse, and the one thing that breaks it.
landmarks[33] = { x: 0.51, y: 0.33, z: -0.41,
visibility: 0.99, presence: 0.99 }
world_landmarks[33] = { x: -0.02, y: -0.31, z: 1.14 } # metresThe whole model is a U-Net-shaped regressor with K output channels and an MSE loss against Gaussians. The whole inference is an argmax. What is left is the part nobody warns you about: the coordinate bookkeeping around those two facts.
You have already built every ingredient: a convolutional encoder and decoder from Phase 4 Lesson 07 (U-Net), a per-pixel target from chapter 03, and a metric from chapter 05. The network below is deliberately tiny — two stride-2 convolutions down, two transposed convolutions up, one channel per keypoint — because the point of the build is the shape of the problem, not the capacity of the model. Stack it deeper and add skip connections and you have the actual architecture family (HRNet’s job is to keep a high-resolution path alive instead of losing it in the bottleneck).
import torch.nn as nn
import torch.nn.functional as F
class TinyKeypointNet(nn.Module):
def __init__(self, num_keypoints=4, base=16):
super().__init__()
self.down1 = nn.Sequential(nn.Conv2d(3, base, 3, 2, 1), nn.ReLU(inplace=True))
self.down2 = nn.Sequential(nn.Conv2d(base, base * 2, 3, 2, 1), nn.ReLU(inplace=True))
self.mid = nn.Sequential(nn.Conv2d(base * 2, base * 2, 3, 1, 1), nn.ReLU(inplace=True))
self.up1 = nn.ConvTranspose2d(base * 2, base, 2, 2)
self.up2 = nn.ConvTranspose2d(base, num_keypoints, 2, 2)
def forward(self, x):
return self.up2(self.up1(self.mid(self.down2(self.down1(x)))))
# input (N, 3, H, W) → output (N, K, H, W); loss is per-pixel MSE
# against the Gaussian targets, upsampled to the target's resolutionimport torch
def heatmap_to_coords(heatmaps, stride=1):
"""heatmaps: (N, K, H, W) → (N, K, 2) coordinates in input pixels."""
N, K, H, W = heatmaps.shape
flat = heatmaps.reshape(N, K, -1)
index = flat.argmax(dim=-1)
xs = (index % W).float()
ys = (index // W).float()
return torch.stack([xs, ys], dim=-1) * stride
def subpixel_refine(heatmaps, coords, stride=1, weight=0.25):
"""A 1-D parabola per axis around the argmax cell."""
N, K, H, W = heatmaps.shape
refined = coords.clone() / stride
for n in range(N):
for k in range(K):
x, y = int(coords[n, k, 0] / stride), int(coords[n, k, 1] / stride)
if 0 < x < W - 1 and 0 < y < H - 1:
hm = heatmaps[n, k]
# parabola vertex through h(x-1), h(x), h(x+1)
left, mid, right = hm[y, x - 1], hm[y, x], hm[y, x + 1]
denom = left - 2 * mid + right
dx = 0.5 * (left - right) / denom if abs(denom) > 1e-9 else 0.0
up, down = hm[y - 1, x], hm[y + 1, x]
denom = up - 2 * mid + down
dy = 0.5 * (up - down) / denom if abs(denom) > 1e-9 else 0.0
refined[n, k, 0] = x + dx
refined[n, k, 1] = y + dy
return refined * strideimport numpy as np, torch, torch.nn.functional as F
def make_synthetic_sample(size=64, rng=None):
"""Four black dots on a white canvas + their Gaussian targets."""
rng = rng or np.random.default_rng()
img = np.ones((3, size, size), dtype=np.float32)
kps = rng.integers(10, size - 10, size=(4, 2))
for cx, cy in kps:
img[:, cy - 2:cy + 2, cx - 2:cx + 2] = 0.0
hms = np.stack([gaussian_heatmap(size, cx, cy) for cx, cy in kps])
return img, hms, kps.astype(np.float32)
model = TinyKeypointNet(num_keypoints=4)
opt = torch.optim.Adam(model.parameters(), lr=3e-3)
for step in range(200):
batch = [make_synthetic_sample() for _ in range(16)]
imgs = torch.from_numpy(np.stack([b[0] for b in batch]))
hms = torch.from_numpy(np.stack([b[1] for b in batch]))
pred = model(imgs)
pred = F.interpolate(pred, size=hms.shape[-2:], mode="bilinear", align_corners=False)
loss = F.mse_loss(pred, hms)
opt.zero_grad(); loss.backward(); opt.step()
if step % 40 == 0:
print(f"step {step:3d} mse {loss.item():.4f}")What should the numbers look like when it works? The loss starts near the energy of the target itself and falls; you are looking for the decoder’s L2 error, not the MSE. Two floors are worth computing before you are disappointed:
In the wild the loss is usually not plain MSE. HRNet trains COCO with online hard keypoint mining: compute the per-keypoint losses, keep the hardest K, and average only those — the pose version of hard-example mining, which stops the easy joints (nose, eyes, both highly stable) from drowning out the wrists and ankles that actually decide the metric. If your loss plateaus at a value that looks fine while the per-joint errors are ugly, that asymmetry is the reason.
That is the lesson in one sentence: a pose is a set of ordered keypoints, a keypoint detector is a heatmap regressor, and everything else is bookkeeping. The model is the solved part — a U-Net that paints K blobs, trained with a Gaussian target. The score you get in production is decided by the three decisions around it: top-down or bottom-up (chapter 02), how you decode the peak (chapter 03), and which metric you optimised and report (chapter 05). The source lesson asks you to ship two artefacts — a prompt that picks the pose stack for a given latency, crowd size and 2D/3D need, and a skill that writes the sub-pixel heatmap-to-coordinate routine. They are this page’s pose-stack chooser and the decode block above.
Answer before you look. The heatmap question, the crowd-size question and the PAF question are the three that separate a memorised pipeline diagram from a mechanism you can debug on a new model.
0 / 5 answered · 0 correct
01Why do pose models regress heatmaps instead of (x, y) coordinates directly?
02Top-down pose estimation versus bottom-up: which scales better with crowd size, and why?
03What are Part Affinity Fields?
04Why does sub-pixel refinement around the argmax meaningfully lift keypoint accuracy?
05OKS (Object Keypoint Similarity) is the pose-estimation analogue of what object-detection metric?
Click a card to swap the lazy description for what it actually means.
Three problems with exact numbers — train the tiny model, beat the argmax with a parabola, and build a two-person bottom-up pipeline that has to earn its matches. Try first; a worked answer is one click away.
The dataset is 4 black dots on a 64 × 64 white canvas with Gaussian targets at σ = 2, batch 16, Adam at 3e-3: it converges in about a minute on a CPU, with the MSE falling from roughly the target's own energy to a plateau. Measure what matters — the decoded L2 in pixels — and expect two things. First, the integer-argmax floor: a Gaussian peak lands at a random sub-cell offset, so each axis carries ≈0.25 cells of error on average, which is ≈0.35 cells of L2 and, at stride 4, ≈1.4 image pixels. A converged model on this toy problem should land near that floor, not at 0. Second, the refinement: the parabola decode cuts the sub-cell error to a few hundredths of a cell, so the measured L2 drops toward the network's own prediction error (usually 0.5–1.5 px at full resolution for a model this small). Sanity checks that catch real bugs: the target's total mass is 2πσ² = 25.133 per joint; the peak of every target is exactly 1.000; and a model that reports a suspiciously perfect L2 is probably decoding with the same coordinates it trained on (leaking the target) or forgetting the stride somewhere.
Pick a peak whose true centre sits at cell 6.35 with σ = 2 and read the three cells around the argmax: h(5) = exp(−1.35²/8) = 0.7963, h(6) = exp(−0.35²/8) = 0.9848, h(7) = exp(−0.65²/8) = 0.9486. The parabola vertex is δ = 0.5(h₋ − h₊)/(h₋ − 2h₀ + h₊) = 0.5(0.7963 − 0.9486)/(0.7963 − 1.9696 + 0.9486) = −0.0762/−0.2247 = 0.339 — within 0.011 cells of the true 0.35 offset (0.045 image px at stride 4). The 0.25 rule returns 0.25 × (0.9486 − 0.7963) = 0.0381, an error of 0.31 cells, which is barely better than the integer argmax it started from. The reason is algebra, not luck: near the peak the first difference h₊ − h₋ grows linearly with the offset with coefficient 2·exp(−1/2σ²)/σ², which at σ = 2 equals 0.4413 — so the exact coefficient is 1/0.4413 ≈ 2.27, not 0.25. The 0.25 rule is a conservative nudge that always under-corrects a Gaussian target; it is safe, it is cheap, and it is not a measurement. Report the gain as an L2 reduction: on the synthetic set the parabola should take the mean error from ~1.4 px (integer, stride 4) to well under 0.5 px, and the honest comparison is always decoder-vs-decoder on the same heatmaps.
Double the target tensor: K = 4 heatmap channels plus 2 channels per limb type. With 4 keypoints there are 3 natural limbs (say 0→1, 1→2, 2→3), so the PAF head has 6 channels; the target at each pixel along a limb is that limb's unit vector, zero elsewhere, and the loss is plain MSELoss over heatmaps and fields together (weight the two terms 1:1 to start). At inference, argmax the heatmaps to get 8 candidate points, then for each limb type score every candidate source × target pair with the sampled line integral E(a, b) = (1/n)Σ PAF(p_s)·u, sort descending, and accept greedily while both endpoints are free; the accepted edges are your instances. Two checks tell you whether association worked. First the field itself: on a held-out image, the integral of the true pair should be ≈1.0 and the best wrong pair should be far below it — with two patterns 100 px apart and 24 samples, a wrong pair that leaves the corridor lands at 0.03–0.07, and even at heavy overlap the worst cross-pair we measured (torso, corridors fully mixed) only reaches 0.42. Set the accept threshold at 0.5 and the margin does the work. Second the metric: on a 100 × 200 synthetic figure with ~2 px errors, a correct match scores OKS ≈ 0.97–0.99, because the κ-weighted terms are all close to 1 (e.g. the ankle at 2 px, area 20,000: e = 4/(8 × 0.089² × 20,000) = 0.0032 → 0.997). A single swapped association is unmistakable — the mis-linked wrist gets its partner's position, tens of pixels away, and its term collapses to ≈0.003, dragging the instance mean toward 0.75 — which is exactly why association errors, not regression errors, are what a bottom-up pipeline is judged on.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 04, Lesson 21) and the Math Foundations Notebook reference build. The five labs — the heatmap playground with its four decoders, the PAF associator with the line-integral table, the top-down/bottom-up Gantt race, the OKS calculator with the live 17-joint table, and the pose-stack chooser — are original to this page, as are the numeric checks they compute: the target's mass Σ ≈ 2πσ² (25.133 at σ = 2, 100.53 at σ = 4), the peak-cell MSE ladder (0.0139 at one cell, 0.748 at four), the H/4 memory arithmetic (326,400 values = 1.31 MB for a 640 × 480 frame; 52,224 = 209 KB for a 256 × 192 crop), the decoder comparison on one asymmetric peak (argmax 1.41 image px, parabola 0.045, soft-argmax 0.18 at β = 10, the source's 0.25 nudge 1.44, with the exact linear coefficient 2.27 at σ = 2), the soft-argmax temperature band, the latency model with its crossover at 6 people and the 33.3 ms budget fitting 8 people top-down, the PAF corridor integrals (true 1.000, cross pairs 0.03–0.42), the OKS worked examples (0.4967 and 0.7764 for a rigid 10 px shift; 0.397 against 0.924 for 5 px on nose against ankle; the 11.72× strictness ratio) and the flip-TTA swap table. Every number shown is computed live by the labs or verified by hand in the prose.