Detection gives you boxes. Association gives you names.
A tracker is a detector plus four small ideas: predict each track forward, cost every track-detection pair with 1 − IoU, solve the one-to-one assignment, and run a lifecycle — birth, update, coast, death. Learn the loop by building it, meet the family that scales it (SORT, DeepSORT, ByteTrack, BoT-SORT), then see why the newest trackers throw the loop away and keep a memory of the object instead.
Every tracker in production is one loop: a per-frame detector, a Kalman filter predicting each track forward, an assignment between predicted tracks and detections on cost = 1 − IoU with everything below the gate made infeasible, and a lifecycle — birth from unmatched detections, update on matches, coast on the prediction, death after max_age missed frames. The only step a detector does not already give you is the assignment.
cost = 1 − IoU · gate 0.2–0.5 · Hungarian O(n³) · 100 tracks × 100 detections = 10,000 cells at 30 fps02 / THE CLASSICAL FAMILY
Each generation adds one idea.
SORT: Kalman plus IoU Hungarian, gate 0.3. DeepSORT: add an appearance embedding and a Mahalanobis motion gate. ByteTrack: keep the weak boxes — split detections at 0.5, match the leftovers against 0.1–0.5 boxes with the stricter 0.5 overlap gate. BoT-SORT: subtract the camera's global motion first, plus optional ReID. Read the list as fixes to one fragile assumption at a time.
ByteTrack: high ≥ 0.5 · low 0.1–0.5 · stage 1 IoU ≥ 0.2 · stage 2 IoU ≥ 0.5 · births ≥ 0.603 / THE MEMORY FAMILY
Stop matching. Start remembering.
SAM 2 prompts an instance once, encodes it into a memory bank (seven slots: the prompted frame plus the six most recent) and cross-attends to it on every new frame; the mask decoder returns the same instance. No Kalman filter, no cost matrix, no Hungarian — association is implicit in the attention, and identity survives occlusions that would kill any box-based track.
Tracking is detection plus association: the detector is memoryless, so identity is created every frame by matching this frame’s boxes to last frame’s tracks — with a motion model for the gaps, a gate for the impossible pairs, and a memory bank for the objects that disappear entirely.
By the end you will be able to write the tracking loop from scratch — IoU, the cost matrix, gating, Hungarian, birth/update/coast/death — and explain what each knob changes; read a cost matrix and predict which track a greedy pass would kill; say what ByteTrack’s second stage actually gates on (confidence down to 0.1, overlap up to 0.5) and why that recovers identities; compute the stale-box IoU that decides whether a 3- or 4-frame occlusion breaks an ID, and explain why a memory bank does not have that failure mode; and pick between MOTA, IDF1 and HOTA depending on whether your product counts boxes, keeps identities, or needs to know which half of the tracker is broken.
01
DETECT, THEN ASSOCIATE
A detector names what is in a frame. A tracker says who is who.
Detection answers “what is there?” once per frame. Tracking answers “which of these is the same one as last frame?”, and that second question is the whole product: a line count that never double-counts, a ball followed through a scrum, a car that has been in lane 2 for 8.4 seconds.
The difference is easiest to see in what breaks without it. A detector on a fixed camera already gives you a box per car per frame — but nothing that says the car at (312, 204) this frame is the same car as the box at (286, 205) two frames ago. Count boxes crossing a line and you count every car many times. Follow one animal through a bush and you cannot, because for four frames there is no box at all. Ask “how long has this truck been queuing?” and there is no “this truck” to ask about — only a fresh box every 33 ms.
Tracking adds exactly one thing to the per-frame detector: association — the bookkeeping that decides which detection continues which existing track. Everything else in this lesson follows from taking that step seriously. The standard recipe has four parts, and every tracker in production is a variation on it:
A per-frame detector — YOLO, RT-DETR, Mask R-CNN — producing boxes with scores.
A motion model — a Kalman filter predicting where each track should be in this frame, so association compares detections to expectations, not to stale boxes.
An association step — usually the Hungarian algorithm on a cost of 1 − IoU, sometimes fused with appearance similarity.
A track lifecycle — birth from unmatched detections, update on matches, coasting on the motion model, death after max_age missed frames.
The source’s loop, redrawn: the only step a detector does not already have is the assignment in the middle. Everything else is the bookkeeping around it.
There is a second family of designs worth naming before the classical stack takes over the lesson. Query-based trackers (TrackFormer, MOTR, MOTRv3) carry a set of learned track queries through time inside a transformer: a query that found a person in frame t is fed back into frame t+1 and predicts again, so association is implicit in the attention between queries and image features rather than in an IoU matrix. The trade is familiar — more accurate on benchmarks, harder to debug, and its identities live in a learned latent space instead of a table of boxes and IDs. SAM 2’s memory tracker is the prompt-driven cousin of this idea: the prompt (a click, a box) plays the role of the query, and the memory bank plays the role of the carried state.
question
tracking-by-detection
query-based / memory
Where does identity live?
in an explicit table: track ID → box, velocity, age
in the model state: a query embedding, or the instance’s memory features
What is matched each frame?
predicted boxes vs detections, cost 1 − IoU (plus appearance)
queries cross-attend to image features; the model outputs the association implicitly
What happens when the object disappears?
the Kalman prediction coasts for a few frames; the track dies at max_age
the query or the memory survives the gap and can re-find the object much later
Typical cost
a few milliseconds per frame for 100 objects
a full network forward pass; memory per instance
Debug story
print the cost matrix and the IDs
inspect attention maps and memory reads
This lesson builds the classical stack by hand — because it is small enough to implement in one sitting and because the ideas transfer. Then it adds the memory-based version as the modern alternative for the case classical trackers are worst at: long occlusions.
The tracker playground: same detections, different ideas
A car, a jogger and a cyclist move across 38 frames. The jogger is hidden behind the car for three frames (11–13) and the cyclist is reported at 0.42 for six frames (20–25). Run the tracker with motion prediction and ByteTrack’s second stage on and off, and watch the ID-switch counter.
Two failures, one fix each. Without prediction the jogger’s stale 40 px box sits 24 px behind its next detection — IoU 0.250, below the gate — so the track dies and is reborn with a new ID at frame 14. Without stage 2 the cyclist’s 0.42 stretch is dropped, its track dies at frame 25 and the 0.81 box at frame 26 starts a new ID. Run both on and both problems disappear. The toy detector here is exact except for the missing and weak frames: this lab isolates association, not detection.
the whole interface, in the source's own words (code/main.py)python
class SimpleTracker:
def __init__(self, iou_threshold=0.3, max_age=5):
self.tracks, self.next_id = [], 1
self.iou_threshold, self.max_age = iou_threshold, max_age
def step(self, detections, frame):
# 1. cost = 1 - IoU(track_boxes, det_boxes)# 2. cost[iou < 0.3] = 1e6 <- the gate# 3. linear_sum_assignment(cost) <- the Hungarian step# 4. unmatched detections -> new tracks (birth)# 5. frame - last_frame > max_age -> delete (death)return [(t.id, t.bbox) for t in self.tracks]
Sixty lines in the source. Everything else in this lesson — appearance features, second stages, camera compensation, memory — is an upgrade to one of the five numbered lines.
Quick check
Your detector already runs at 30 fps with excellent recall. A colleague says 'we have detection, so we have tracking — the boxes are right there.' What is missing?
02
THE FAMILY TREE
Six trackers, one loop. Each adds a single idea.
SORT matched boxes. DeepSORT matched looks. ByteTrack stopped throwing away weak boxes. BoT-SORT compensated for the camera. SAM 2 replaced association with memory. Read the list as a call stack of fixes to one fragile assumption at a time.
Six years of small, stackable ideas; the 2024 shift replaces the assignment step altogether rather than improving it.
SORT (Bewley et al., 2016) is the minimal statement of the pattern: a constant-velocity Kalman filter predicts each track, the Hungarian algorithm matches predictions to detections on 1 − IoU, anything below an IoU of 0.3 is treated as infeasible, and a track that misses more than max_age frames is deleted. No learning, no appearance, no second pass — and for years it was competitive, because a good detector plus a decent motion model already solves most frames. Read the original paper and notice how much of it is about the filter, not the matching.
DeepSORT (Wojke et al., 2017) attacks SORT’s worst failure: two objects crossing, where box overlap alone is ambiguous. Every detection also gets an appearance descriptor — a small CNN trained to make the same person’s crops similar and different people’s crops far apart — and the association cost fuses IoU with cosine distance, with a Mahalanobis gate on the motion side to reject impossible jumps. On MOT16 this cut ID switches sharply at the price of one extra embedding network per detection. The idea survives in every “ReID” tracker: identity is not just where the box is, it is what the box looks like.
ByteTrack (Zhang et al., 2022) attacks a quieter assumption: that detections below the confidence threshold are garbage. They are often not — an occluded pedestrian, a cyclist in a shadow, a car behind a bus produce weak boxes whose location is roughly right. ByteTrack keeps every box above 0.1, splits the stream at 0.5, runs the usual Hungarian pass on the high half, and then gives the tracks that failed a second pass against the low half. No appearance model, no extra network, and 1–10 IDF1 points on MOT17. It is the default tracker in Ultralytics and Roboflow Supervision for exactly that ratio of gain to cost.
BoT-SORT (Aharon et al., 2022) adds the fix for cameras that move: a global motion estimate (sparse optical flow or ECC alignment) that shifts every track prediction by the camera’s frame-to-frame transform before association, plus optional ReID. On a panning sports camera every box moves several pixels per frame even when nothing moves in the world; without compensation, every track’s IoU with its own next detection drops at once and the assignment starts pairing neighbours. If the camera is on a mast and never moves, BoT-SORT’s extra machinery earns nothing.
Two descendants are worth being able to name when you read a leaderboard.OC-SORT (observation-centric SORT, 2022) keeps a short history of observations per track and uses it to recover a track after an occlusion — its velocity estimate comes from two past observations rather than the last prediction, so a bad prediction does not poison the recovery. StrongSORT is DeepSORT with a better embedding network, better matching and a link model that stitches fragments after the fact. Both are the same loop with a stronger memory of what happened, which is the theme of the next chapter.
What ByteTrack's two stages actually gate on (a correction to the folklore)
The friendly summary is “ByteTrack gives low-confidence detections a second chance with a looser IoU threshold”. The reference implementation is more interesting: the looseness is in confidence, and the overlap gate gets stricter. Detections at or above track_thresh = 0.5 are the stage-1 pool; detections between 0.1 and 0.5 are the stage-2 pool; a new track is only born from a detection at or above det_thresh = track_thresh + 0.1 = 0.6. The association calls use linear_assignment(dists, thresh=...) with dists = 1 − IoU and a cost limit of 0.8 in stage 1 and 0.5 in stage 2, which forbids pairs with IoU below 0.2 in the first pass and below 0.5 in the second. That is the opposite direction from “looser”, and it is deliberate: a weak box is only trustworthy when it lands almost exactly on the track’s predicted position. Worked example — a track predicted at box (270, 180, 310, 280) with a weak detection at (276, 180, 316, 280): IoU 0.739 clears 0.5 comfortably, so the track keeps its ID; the same track against a weak box 24 px further along scores (40−24)/(40+24) = 0.250 and is correctly refused. The lab’s playground runs exactly this code path: with stage 2 on, the cyclist’s six weak frames are six recoveries; with it off, the track dies at frame 25 and a new ID is born at 26.
ByteTrack's second stage, in the shape of the reference codepython
# detections are split by score, then two Hungarian passes are run
high = [d for d in detections if d.score >= 0.5] # track_thresh
low = [d for d in detections if0.1 <= d.score < 0.5]
# stage 1: every live + lost track against the high-confidence boxes
matches, u_track, u_det = linear_assignment(
iou_distance(track_pool, high), thresh=0.8) # forbids IoU < 0.2# stage 2: the tracks stage 1 could NOT match, against the weak boxes
weak_matches, still_unmatched, _ = linear_assignment(
iou_distance([track_pool[i] for i in u_track], low), thresh=0.5) # IoU < 0.5 is refused# births are stricter than matches: a new ID needs score >= 0.6for det in [high[i] for i in u_det]:
if det.score >= 0.6:
new_track(det)
The asymmetry is the point: matching an existing ID tolerates a weaker box than creating a new one. Tracks are cheap to keep and expensive to be wrong about.
SAM 2 (Ravi et al., 2024) changes the question. Given a prompt on one frame — a click, a box — it encodes that instance into a memory bank, and on every following frame the image features cross-attend to the bank; the mask decoder then outputs the mask of the same instance. There is no Kalman filter, no cost matrix, no Hungarian assignment: association is implicit in the attention. The released configuration keeps seven memory slots per instance (the prompted frame plus the six most recent), and the bank is a FIFO — old memories are dropped, which is why the mechanism is robust to long occlusions but not to infinitely long ones. It is slower than ByteTrack for many objects and its memory grows with the number of instances, but for mask-quality tracking of a handful of objects with real occlusions, nothing classical comes close.
SAM 3.1 Object Multiplex (reported March 2026) is the answer to that scaling complaint: instead of one memory bank per instance, one shared memory with per-instance query tokens that fetch the features belonging to each object. The growth is still linear in the number of instances — but the constant is 16× smaller, because one forward pass covers up to 16 objects, so the tracker needs ceil(O/16) passes per frame instead of O. That is what makes memory-based tracking plausible for concert crowds, warehouse floors and traffic intersections — the places where per-instance banks would need gigabytes. Chapter 04 does the arithmetic on those banks; it is the reason Multiplex exists.
The tracker chooser: pick a family, then a number to watch
Six deployments, six answers. The scaffolding is the same everywhere — detector, motion model, association, lifecycle — but the scene decides which part deserves the engineering.
USE CASE · Traffic intersection
A fixed pole camera, 40–80 cars, bikes and pedestrians, counting who crosses the line and how long they waited.
· fixed camera
· 30 fps
· many objects
· no appearance needed
RECOMMENDATION
tool ByteTrack
pipeline
YOLO detector at ~0.25 confidence → high/low split at 0.50 → Hungarian on 1 − IoU, gate 0.2 → second stage at 0.50 → count line crossings by ID.
watch IDF1 first (each vehicle keeps one ID across the intersection), MOTA for the detector's misses.
gotcha
Occlusion by buses makes weak boxes; if you drop everything below 0.5 you fragment exactly the long, slow vehicles you are counting.
family
key idea
what it needs
what it costs
when it wins
SORT · 2016
Kalman filter + Hungarian on 1 − IoU, gate 0.3, no appearance model
a detector good enough that box overlap is informative; max_age brings tracks back after a few missed frames
the cheapest tracker here — one small matrix solve per frame, no extra network
one class, a clean camera, latency matters more than identity
DeepSORT · 2017
SORT + a ReID appearance embedding per detection, fused with Mahalanobis motion gating
an embedding network per detection and a gallery of features per track
one extra CNN forward pass per detection; the strongest identity signal of the classical family
objects look different enough to tell apart (people in different clothes, animals)
ByteTrack · 2022
two-stage association: high-confidence boxes first, then leftover tracks against 0.1–0.5 boxes with a 0.5 overlap gate
nothing beyond a detector that emits scores — no embeddings, no second network
essentially SORT plus a second small assignment; the reference implementation runs at 30 fps
the default answer for pedestrians, vehicles and boxes; weak boxes at crossings are information, not noise
a frame-to-frame camera motion estimate (sparse optical flow or ECC) before association
one global motion step per frame; more accuracy when the camera moves, more knobs
the camera pans, zooms or shakes: sports, drones, handheld footage
SAM 2 memory · 2024
one prompt encodes an instance into a memory bank; each frame cross-attends to the bank and the decoder outputs the same instance
a prompt per object (click or box) and the memory bank per instance
a full segmentation forward pass per frame, 7 memory slots per instance — slower than ByteTrack for many objects
masks, video annotation, rotoscoping, one-to-few objects with long occlusions
SAM 3.1 Object Multiplex · 2026
one shared memory with per-instance query tokens instead of N separate banks
the shared-memory model; prompts still identify each instance
ceil(O/16) tracker passes per frame instead of O — a 16× smaller constant, the first memory tracker that scales to crowds
many instances of one concept: concert crowds, warehouse floors, traffic intersections
The rule that survives every one of these: the tracker cannot be better than the detections it is handed. If boxes disappear for 20 frames, no Kalman filter invents them — that is the gap memory trackers were built for, and also why they cost more.
None of this makes the older families obsolete. A fixed pole camera with a solid detector is still best served by ByteTrack, and a microcontroller with a 10 ms budget still wants SORT. The family tree is not a ladder; it is a menu with prices.
03
THE COST MATRIX AND THE MATCH
One table of overlaps. One optimal matching.
Take M predicted tracks and N detections, compute the IoU of every pair, turn overlap into cost, cross out everything below the gate, and solve for the cheapest one-to-one assignment. That is the whole classical association step — and the only place in the loop where a local decision can ruin a global identity.
The cost matrix is the interface between the detector and the tracker. Its rows are tracks, its columns are detections, and each cell holds cost = 1 − IoU: 0.000 means the boxes coincide, 1.000 means they do not overlap at all. For 100 tracks and 100 detections that is a 100 × 100 = 10,000-cell table — 40 KB of float32, computed, solved and discarded thirty times a second. Everything the tracker believes about this frame is in there.
Two adjustments turn the table into something a solver can use. First, the gate: every cell whose IoU is below a threshold becomes infeasible rather than merely expensive. A track that would have to move half its own width to reach a detection is not matched to it; it is left out of the assignment altogether. SORT’s original code uses 0.3, ByteTrack 0.2 for its first pass and 0.5 for its second, and production systems live between 0.2 and 0.5 — the lab lets you slide it and watch the ID switches change. Second, unmatched is an option: the assignment must be allowed to leave rows and columns out, which is implemented by padding the matrix with zero-cost dummy rows and columns.
cost[i][j] = 1 - IoU(track_i, detection_j)
gate cost[i][j] = BLOCKED if IoU < threshold (0.2 ... 0.5)
padding R x C -> (R + C) x (R + C), dummy rows/cols cost 0
Hungarian O((R + C)^3) for the padded problem
scipy.optimize.linear_sum_assignment, or lapjv in C++ trackers
Why not just let every track take its best free box? Because a greedy pass makes decisions in an arbitrary order and can spend a detection that another track needed more. The smallest version of the failure is a 2 × 2 matrix. Suppose track A overlaps detection 0 at IoU 0.80 (cost 0.20) and detection 1 at IoU 0.50 (cost 0.50), while track B overlaps detection 0 at IoU 0.70 (cost 0.30) and detection 1 at IoU 0.10 — below the 0.30 gate, so infeasible. Greedy, walking tracks in order, gives A its favourite: cost 0.20, then B has only a gated cell and dies. The optimal assignment pays 0.50 + 0.30 = 0.80 to give detection 0 to B, whose account of the frame is the only one that keeps an identity alive. A dead track is an ID switch waiting to happen; a slightly worse IoU is a rounding error.
cell
IoU
cost
greedy (A first)
optimal
A → d0
0.800
0.200
taken by A (its best)
given to B
A → d1
0.500
0.500
—
taken by A (fallback)
B → d0
0.700
0.300
blocked: d0 already used
taken by B
B → d1
0.100
infeasible
—
—
total
—
—
0.200, and B is left with nothing — it dies
0.800, both tracks continue
The same situation, with real boxes and a real scene, is in the stepper lab below: a car and a jogger at the moment of a pass, where greedy hands the jogger’s strong box to the car because the car was created first. The optimal assignment instead gives the jogger the box it overlaps at 0.739 and lets the car recover its own weak box in stage 2 at 0.786. Two identities survive; under greedy, one dies.
The association stepper: from overlaps to identities
One frame, three tracks and four detections. Step through the cost matrix, the gate, the assignment and the lifecycle — then flip the solver to greedy and watch which identity pays for it.
Scene
Solver
Step
Move
Frame 33 — the crossing
gate 0.30 · second-stage gate 0.50 · max_age 5
solver: Hungarian (global optimum)
· T1 · car -> d0 · car at IoU 0.786 (stage 1)
· T2 · jogger -> d1 · jogger at IoU 0.739 (stage 2)
· d2 · new person -> new track (score 0.77)
· d3 · dog? discarded (score 0.28, no match)
· T5 · dog unmatched -> coasts, ages toward max_age
Two trap cells sit at IoU 0.316: the car's box overlaps the jogger's detection and the jogger's box overlaps the car's weak box. Both are above the 0.30 gate, so the assignment has to choose — and the one-to-one constraint plus the stage split gives each object its own box.
Two lines of infrastructure — a vectorised IoU and a mask — decide which pairs even enter the assignment. In the lesson's crossing scene the two 0.316 cells survive the 0.30 gate only barely; at 0.35 the picture changes completely.
The Hungarian algorithm is the solver of choice: for a square n × n problem it runs in O(n³), which at n = 100 is a million primitive operations — sub-millisecond in optimised C, a few milliseconds in pure Python, and the reason every serious implementation calls out to scipy.sparse.csgraph orlapjv. Rectangular problems are padded to square, gated cells are removed from the feasible set, and the result is a permutation that minimises the total cost. It is also worth knowing what the solver is not doing: it does not know which box belongs to which person, it only minimises a sum. Every semantic assumption lives in the cost function.
The Kalman predict, with the arithmetic
The motion model is what the detector is compared against, so it has to be cheap and honest. The source’s tracker uses a constant-velocity state of eight numbers per track: x = [x, y, w, h, vx, vy, vw, vh], with an 8 × 8 covariance alongside it — 64 more numbers per track, trivial next to a single appearance embedding. One frame of prediction is one addition per dimension:
predict: x' = x + vx y' = y + vy w' = w + vw h' = h + vh
update: blend the matched detection with the prediction,
weights set by the relative uncertainty of each
worked: centre (100, 200), size (50, 120), velocity (6, 2, 0, 0) px/frame
next frame predicted at centre (106, 202), size (50, 120) — no change
after 3 missed frames: (118, 206)
The update step is where the covariance earns its keep: when the prediction is uncertain (a long gap, an accelerating object) the measurement is trusted more; when the prediction is confident, a jittery detection barely moves the track. That is the difference between a Kalman filter and the naive last + last_delta predictor this lesson’s labs use — the labs take the same constant-velocity assumption but skip the uncertainty bookkeeping, and they say so. One more piece of honesty about parameterisations: the original SORT code runs a seven-dimensional state (u, v, s, r, u̇, v̇, ṡ) — centre, scale (area), aspect ratio and their velocities — while DeepSORT and most modern implementations use the eight-dimensional (x, y, a, h) + velocities form the source describes. The two are the same idea in different coordinates.
The lifecycle is the last piece, and the one most often left to defaults. A detection that matches nothing becomes a new track — but only if its score clears a birth threshold, because tracking every weak box would flood the scene with phantoms. A track that matches nothing keeps its predicted box and increments a miss counter; when frame − last_seen > max_age it is deleted. Those two numbers, birth score and max_age, decide the character of a tracker more than any cost function: a low birth score and a short max_age produce many short-lived IDs (the fragmentation the source’s synthetic experiment shows at 30 objects), while a generous max_age keeps identities through occlusions but can resurrect a track that is long gone.
the lifecycle in ten lines (adapted from the source's SimpleTracker.step)python
matched_track, matched_det = set(), set()
if cost.size > 0:
row, col = linear_sum_assignment(cost)
for r, c in zip(row, col):
if cost[r, c] < 1.0: # accept only feasible, affordable pairs
self.tracks[r].update(det_boxes[c], frame)
matched_track.add(r); matched_det.add(c)
for i, d in enumerate(det_boxes): # unmatched detection -> birthif i notin matched_det:
self.tracks.append(Track(self.next_id, d, frame)); self.next_id += 1
self.tracks = [t for t in self.tracks if frame - t.last_frame <= self.max_age]
# ^ unmatched tracks survive max_age frames, then die
Birth from unmatched detections, death from the age counter, update on the rest. Nothing here is learned — and nothing here is optional: an ID switch is exactly this code taking the wrong branch.
Quick check
A track's last box has IoU 0.25 with every detection this frame, and the gate is 0.30. The track is on frame 3 of max_age 5. What happens?
04
WHERE THE MEMORY LIVES
A box that is not there cannot be matched.
Every classical tracker is secretly an optimist about the detector. It assumes that if an object is real, some box will show up for it. Occlusion is where that assumption dies — and where the argument for memory starts.
Run the numbers on a stale box. An object of width w moving v pixels per frame is not detected for g frames; if the tracker holds the last box, the overlap with the next detection is
IoU(gap) = (w - gap · v) / (w + gap · v)
w = 50 px, v = 6 px/frame:
gap 1 0.786 gap 4 0.351
gap 2 0.613 gap 5 0.250
gap 3 0.471 gap 6 0.163
the 0.30 gate dies between gap 4 and gap 5 — about 30 px of travel
Two escape routes exist inside the classical loop. The first is the Kalman prediction: instead of holding the last box, keep moving it at the estimated velocity, so the compared box tracks the object for the 1–5 frames where constant velocity is still roughly true. This is why the playground in the previous chapter has a Kalman toggle at all — the jogger’s 3-frame occlusion is exactly a gap-3 case at 6 px/frame, whose stale IoU is 0.379, marginal, while the predicted box scores 1.000. The second fix is appearance: a ReID embedding of the object’s pixels changes more slowly than its position, so DeepSORT can afford to distrust the box and trust the look. Both are patches on the same assumption. The assumption itself — that association happens between boxes — is what memory trackers replace.
How SAM 2 tracks an instance, step by step
1. Prompt. On one frame, the user clicks a point or draws a box on the object. The prompt encoder turns that into tokens and the mask decoder produces the instance’s mask. Nothing about this step is temporal; it is SAM 1.
2. Encode into memory. The frame’s image features and the predicted mask are fused by the memory encoder into a spatial feature map — in the released configs 64 channels at stride 16 — and pushed into the memory bank. The decoder’s output token for that instance is also stored as a lightweight object pointer, a high-level summary of what the object is. The bank is a FIFO of seven slots: the prompted frame plus the six most recent unprompted frames.
3. Condition the next frame. For frame t+1, the image encoder produces features as usual; memory attention then lets those features cross-attend to the memories of the instance — the current prediction queries the object’s own past. Self-attention mixes information inside the frame, cross-attention to memory brings the identity, and the mask decoder reads the result and outputs the mask of the same instance.
4. Close the loop. The new mask feeds the memory encoder again, the oldest slot is dropped, and the cycle repeats. When the object is occluded, there is simply no visible object to segment — but the memory is still there, and when the object reappears the cross-attention finds it. That is the entire trick: no Kalman filter, no cost matrix, no Hungarian assignment, association implicit in the attention operation.
SAM 2 video tracking — the reference APIpython
from sam2.build_sam import build_sam2_video_predictor
import numpy as np
predictor = build_sam2_video_predictor("sam2.1_hiera_s.yaml", "sam2.1_hiera_small.pt")
# one prompt on one frame: this box is instance 1
state = predictor.init_state(video_path="clip.mp4")
predictor.add_new_points_or_box(
inference_state=state, frame_idx=0, obj_id=1,
box=np.array([x1, y1, x2, y2]),
)
# propagate: every frame is keyed by the same obj_id, no matching requiredfor frame_idx, obj_ids, masks in predictor.propagate_in_video(state):
for mask in masks: # (1, H, W) logits per object
save_mask(frame_idx, obj_ids, mask)
# a corrective click at frame 300 is appended to the same object's memory
predictor.add_new_points_or_box(
inference_state=state, frame_idx=300, obj_id=1,
points=np.array([[cx, cy]]), labels=np.array([1]),
)
One obj_id survives the whole clip: identity is not recomputed per frame, it is carried. The corrective click is worth noticing — a human edit becomes new memory, and the tracker propagates it forward.
The price is memory and compute. One memory slot for a 1024 × 1024 frame is a 64 × 64 spatial map of 64 channels: 64 × 64 × 64 = 262,144 values, about 1.05 MB in float32. A full bank of seven slots is therefore ≈ 7.3 MB per tracked instance, before masks, features and activations. Five objects is a rounding error; fifty is ≈ 367 MB — which is exactly the arithmetic that motivated SAM 3.1 Object Multiplex: one shared memory with per-instance query tokens instead of fifty private banks. The growth is still linear in the number of instances; what changes is the tracker cost — one forward pass covers up to 16 objects, so it runs ceil(O/16) passes per frame instead of O, a 16× smaller constant on the same trend. If you have ever wondered why crowd tracking went from “one memory per object” to “one memory, many queries”, this is the reason.
It also explains when not to reach for memory. A fixed camera with 80 vehicles and a 30 fps budget is a ByteTrack problem: cheap, box-based, and its failure modes are measurable with MOTA and IDF1. A memory tracker is the right tool when masks matter (annotation, editing, medical video), when objects genuinely vanish for seconds (wildlife, sports scrums, warehouse shelves) or when the object count is small enough that 7.3 MB and a forward pass per object are affordable. And a useful middle option remains DeepSORT-style appearance tracking: it needs one embedding per detection instead of a segmentation network, and it brings the same “identity is more than position” insight at a fraction of the cost.
The occlusion simulator: a gap IoU cannot cross
One object walks behind a pillar. Tracker A matches boxes frame to frame and holds its last box while the detections are gone; tracker B carries the instance in a memory bank. Slide the gap, max_age and the memory horizon to see who keeps ID #7.
Playback
gap (frames with no detection) 6
tracker A (IoU only, max_age 5) dies: gap 6 > max_age 5
tracker B (memory, horizon 7) survives: gap ≤ horizon, the instance stays in the bank
ID switches A 1 B 0
stale-box arithmetic without memory
IoU(gap) = (50 − gap × 6) / (50 + gap × 6)
gap 2 → 0.613
gap 3 → 0.471
gap 4 → 0.351
gap 5 → 0.250
gate 0.30 dies at 5 frames (IoU 0.250)
memory bank, the SAM 2 numbers
slots 7 = 1 prompted frame + 6 recent
one slot 262,144 values = 1.05 MB fp32
per instance 7.34 MB
50 instances 367 MB before masks and features
IoU-only association cannot bridge a gap it cannot see across: the stale box falls behind by gap × 6 px, and a 50 px box is below the 0.30 gate by the fifth frame. Memory does not predict the object — it remembers it, which is why SAM 2 keeps an identity for as long as the instance stays in the bank.
Quick check
A tracker loses a cyclist behind a bus for 12 frames. The motion-model prediction would need the cyclist to keep constant velocity that whole time, and a 50 px box at 6 px/frame is below the 0.30 gate after five frames (IoU 0.250) and has no overlap left after nine. Which mechanism is built for this case?
05
THREE NUMBERS, THREE QUESTIONS
MOTA counts errors. IDF1 counts identities. HOTA splits the difference.
A tracker can fail in three ways — miss an object, invent one, or give one object two names — and no single number weights all three the way your product does. Read the three the way an engineer reads them: as three different questions, not three scores of the same thing.
MOTA (Multi-Object Tracking Accuracy) is the oldest and the bluntest:
MOTA = 1 − (FN + FP + IDSW) / GT
GT total ground-truth detections over the clip (objects × frames present)
FN detections the tracker missed
FP detections the tracker invented
IDSW identity switches: frames where an object's matched track ID changed
worked: GT 500, FN 40, FP 25, IDSW 6
1 − (40 + 25 + 6) / 500 = 1 − 71/500 = 0.858
and the awkward case: GT 500, FN 100, FP 450, IDSW 10
1 − 560/500 = −0.120 MOTA can go negative
Two things to internalise about MOTA. It is a per-detection error rate, not a percentage of tracked objects, and it is allowed to go negative because a tracker that sprays boxes can out-error the ground truth. Second, all three error types enter with the same weight, so ten false positives hide one ID switch perfectly. That is fine for a product that counts boxes and terrible for a product that names people.
IDF1 asks the identity question directly. Instead of counting per-frame errors, it matches each ground-truth trajectory to at most one predicted trajectory over the whole clip — maximise the number of frames they agree on — and then takes the harmonic mean of ID precision and ID recall:
The same run that scored MOTA 0.500 scores IDF1 0.533, and the interpretation is different: half the predicted identity-frames belong to the right object. IDF1 is the number to report when the product is “who is who” — surveillance review, re-identification, per-player analytics.
HOTA (Luiten et al., 2020) is the community standard because it refuses to conflate the two failure modes. It evaluates at 19 localisation thresholds from 0.05 to 0.95, computes at each one a detection accuracy and an association accuracy, and combines them geometrically:
DetA = TP / (TP + FN + FP) detection quality
AssA = Σ over matched trajectory pairs of m / (gt + pred − m)
weighted by matched frames, divided by TP association quality
HOTA = √(DetA · AssA) averaged over 19 IoU thresholds
the messy run again, at one threshold (0.5):
DetA = 11 / (11 + 7 + 1) = 0.579
AssA = (3·0.50 + 3·0.50 + 5·0.833) / 11 = 0.652
HOTA = √(0.579 · 0.652) = 0.614
Read those three numbers on the same run — MOTA 0.500, IDF1 0.533, HOTA 0.614 — and the trap is obvious: they are not three measurements of one quantity, and 0.614 does not mean “better than 0.500”. MOTA is a per-detection error rate, IDF1 is a trajectory-overlap F1, and HOTA is the geometric mean of a detection score and an association score: a tracker that detects everything but mixes up identities and a tracker that misses half the objects but never confuses an ID can both land near 0.6. The instrument panel tells you where to look; the product tells you which dial matters.
The metric calculator: one run, three verdicts
Pick a tracker run over six frames and three objects (18 ground-truth instances). Every frame is matched at IoU ≥ 0.5 first, then the three numbers are assembled from the errors — MOTA from per-detection counts, IDF1 from trajectory matching, HOTA from its two factors.
Run
Object A fragments mid-video, object B is never tracked at all, object C is missed once, and one phantom box appears. The number row a real tracker produces.
6 frames · 3 objects · GT instances = 18
Real HOTA averages over 19 localisation thresholds from 0.05 to 0.95 and is reported by TrackEval; this lab fixes the threshold at 0.5 so you can do the arithmetic by hand. py-motmetrics computes MOTA and IDF1 from a MOTChallenge-format file — the numbers here should match it exactly if you export the same boxes.
frame
ground truth (3 objects)
predictions (track IDs)
matched at IoU ≥ 0.5
FN
FP
0
A B C
#1
A→#1 1.00
2
0
1
A B C
#1 #2
A→#1 1.00 C→#2 1.00
1
0
2
A B C
#1 #2
A→#1 1.00 C→#2 1.00
1
0
3
A B C
#4 #2
A→#4 1.00 C→#2 1.00
1
0
4
A B C
#4 #2 #9
A→#4 1.00 C→#2 1.00
1
1
5
A B C
#4 #2
A→#4 1.00 C→#2 1.00
1
0
trajectory pair (matched by IDF1)
matched frames
GT frames
pred frames
J = m/(gt + pred − m)
A ↔ #1
3
6
3
0.500
C ↔ #2
5
6
5
0.833
A ↔ #4
3
6
3
0.500
the question you are actually asking
report
why
“Who is who over time?”
IDF1
it matches whole trajectories, so it is exactly the identity-preservation score. Surveillance, crowd analytics, any product that shows one ID per person.
“Is this tracker better overall?”
HOTA
it separates detection quality (DetA) from association quality (AssA) and averages over localisation thresholds, so it does not let a good detector hide bad IDs.
“Are we missing objects or inventing them?”
MOTA and DetA
per-detection errors: misses, false positives, switches. Useful for counting products and for spotting a detector regression at a glance.
“Does the box follow the object tightly?”
LocA (or HOTA's localisation term)
localisation accuracy: the IoU of accepted matches, averaged. It is the part of tracking quality that neither MOTA nor IDF1 reports.
the three formulas, and where the real tooling livespython
GT = tp + fn # ground-truth detections
MOTA = 1 - (fn + fp + idsw) / GT # can be negative
IDF1 = 2 * idtp / (2 * idtp + idfp + idfn) # trajectory matching first
HOTA = (detA * assA) ** 0.5# then sqrt, averaged over 19 IoUs# production tooling, do not hand-roll:# py-motmetrics -> MOTA, IDF1, IDP/IDR from MOTChallenge-format files# TrackEval -> HOTA, DetA, AssA, LocA across 19 thresholds# the HOTA paper's argument in one line: MOTA conflates detection and# association errors; report DetA and AssA when you need to know which broke
The lesson's metric lab recomputes all three from the same six-frame run so you can see the arithmetic; for a real benchmark, use TrackEval — hand-rolled HOTA quietly disagrees with the published numbers.
Quick check
Two trackers on the same fixed detector. A: FN 100, FP 80, IDSW 20, GT 1000. B: FN 60, FP 60, IDSW 80, GT 1000. Which is better for a product that keeps one ID per person, and why is MOTA the wrong tool to answer that?
06
WHAT YOU SHIP
Four lines of API, two habits that keep it honest.
In 2026 nobody implements a tracker to ship one. The stack is a detector plus a tracker config plus an evaluation harness — and the harness is the part that separates a demo from a product.
The default stack is Ultralytics: a YOLO detector with model.track() and a tracker YAML. ByteTrack and BoT-SORT both ship inside it, and the tracker’s knobs sit in a file you can diff and tune — which matters, because the defaults are tuned for pedestrian benchmarks, not for your camera. Roboflow’s supervision wraps the same trackers for people who want annotation utilities next to them (box labels, traces, zone counters, line-crossing events) rather than a training framework. And the source’s two artifacts are worth producing before you write any product code: a tracker-picker prompt that turns a scene description (camera motion, occlusion pattern, latency budget, object count) into a family and a configuration, and a metric harness that computes MOTA / IDF1 / HOTA against ground-truth tracks on your own footage. The chooser lab in chapter 02 is the first; the metric calculator in chapter 05 is the second.
the production stack: detector + tracker configpython
from ultralytics import YOLO
model = YOLO("yolo11n.pt")
# ByteTrack: the default for fixed cameras. Persist IDs across frames.
results = model.track(
source="intersection.mp4",
tracker="bytetrack.yaml", # or "botsort.yaml" when the camera moves
persist=True, # keep track state across frames
conf=0.25, # detector floor; the tracker sees 0.1+
stream=True,
)
for frame in results:
if frame.boxes.id is None:
continue
ids = frame.boxes.id.int().tolist()
boxes = frame.boxes.xyxy.tolist()
for track_id, box in zip(ids, boxes):
on_crossing_line(track_id, box) # your product lives here
persist=True is the line people forget: without it every call re-initialises the tracker and every frame gets fresh IDs.
the knobs worth touching, and what they dopython
# bytetrack.yaml (reference values; Ultralytics ships slightly looser ones)
track_high_thresh: 0.5# stage-1 pool: below this a box is a stage-2 candidate
track_low_thresh: 0.1# stage-2 floor: below this a box is discarded entirely
new_track_thresh: 0.6# births are stricter than matches
match_thresh: 0.8# cost = 1 - IoU, so this is IoU >= 0.2 in stage 1
track_buffer: 30# frames a lost track is kept at 30 fps# botsort.yaml adds: gmc_method (camera motion compensation) and with_reid# the three numbers that change behaviour most: match_thresh, track_buffer,# and the detector's conf — test them on YOUR footage, not on MOT17
Every knob here trades one failure for another: looser matching recovers occlusions and risks swaps, longer buffers resurrect tracks and risk ghosts. There is no universal setting, only settings that match a scene.
the second-stage gate — weak boxes are recovered only when they land on the prediction
Moving or panning camera (sports, drones, handheld)
BoT-SORT
HOTA, with DetA and AssA read separately
global motion estimation; without it every track's IoU drops at once when the camera pans
Objects that look different and disappear for seconds (wildlife, people in queues)
DeepSORT / StrongSORT
IDF1 — the product is re-identification
the appearance gallery: bad crops poison the embeddings that do the matching
Masks needed, 1–10 objects, human in the loop
SAM 2 memory tracker
mask J&F plus corrections per 100 frames
memory per instance — 7 slots × 1.05 MB before masks
Many instances of one class with heavy occlusion (crowds, warehouse floors)
SAM 3.1 Object Multiplex
flow/density accuracy; be explicit that IDF1 degrades in crowds
cost growth in the number of instances; shared memory is the point
10 ms budget, one class, simple scene
SORT
latency first, then DetA
crossings — SORT is the first family to swap IDs when two boxes overlap
TWO HABITS THAT KEEP IT HONEST
1. Evaluate on your own footage, at your own frame rate. MOT17 numbers tell you which paper is interesting, not which tracker fits a loading dock. Label twenty minutes of your camera — even three objects across six frames, as in the metric lab — and compute IDF1 and ID switches on that. Two effects dominate real footage and neither is in any benchmark: the frame rate (a 15 fps camera doubles the per-frame displacement and pushes half your matches near the gate) and the detector’s weak-box rate (all those 0.3–0.5 boxes are exactly what ByteTrack’s second stage exists for, or exactly what will flood a naive tracker with ghosts).
2. Log the lifecycle, not just the accuracy. Every event the tracker emits is a sentence about the video: births, deaths, coastings, second-stage recoveries, ID switches. A product that counts vehicles wants deaths to be rare; a product that follows individuals wants switches to be rare. In the playground, the difference between “Kalman on” and “off” is invisible in the detections and obvious in the event log: 2 switches against 0 for the same boxes. Ship the event log next to the numbers; it is what turns “the tracker is wrong” into “the jogger died at frame 14 because the gate was 0.30.”
The last piece of engineering honesty is the one the source ends on:a tracker cannot be better than its detections. If the bicycle is invisible for 30 frames, no association step will recover it, and the only answers are a better detector, a wider gate with all the risks that brings, or a memory tracker that does not need a box at all. Choose deliberately, and write down which one you chose and why.
07
CHECK YOURSELF
Six questions. Then the terms worth keeping.
Answer before you look. The ByteTrack question and the MOTA arithmetic are the two that separate “I have used a tracker” from “I can debug one”.
0 / 6 answered · 0 correct
01What does the Hungarian algorithm actually do in a tracking-by-detection tracker?
02ByteTrack keeps detections below the usual confidence threshold and matches them in a second stage. What exactly changes between the two stages?
03How does SAM 2's memory-based tracker avoid an explicit Hungarian-style association step?
04For surveillance footage where each person must keep one ID across a long clip, which metric should you report first?
05SAM 3.1 Object Multiplex (March 2026) shares one memory across many tracked instances. What does that buy?
06A tracker reports GT = 500, FN = 40, FP = 25, IDSW = 6. What is MOTA, and what would it be for a tracker with GT = 500, FN = 100, FP = 450, IDSW = 10?
Key terms, demystified
Click a card to swap the lazy description for what it actually means — each one carries the arithmetic from the labs.
Exercises from the lesson
Four problems with exact numbers: reproduce the source’s synthetic experiment, add the constant-velocity predict and derive when it stops working, run a memory tracker beside it, and compute the metric row by hand. Try first; a worked answer is one click away.
(Easy) Run the synthetic tracker from the source with 3, 10 and 30 objects over 25 frames and report active tracks and ID switches. Where does IoU-only association start to fail?Show one worked answer
Running the vendored code/main.py at seed 0 gives: 3 objects → 3 live tracks, 0 ID switches; 10 objects → 10 live tracks, 2 switches; 30 objects → 70 live tracks, 18 switches. The three-object world is the easy regime — the boxes never leave the 320 × 240 frame and stay far apart, so IoU-only association holds every identity for all 25 frames. Failure tracks density: at 10 objects two trajectories cross closely enough for the assignment to trade boxes; at 30 objects boxes crowd each other and clip at the frame edges, tracks die and are reborn, and the last frame holds 70 live tracks for 30 objects with 18 switches. The numbers support the rule of thumb — IoU alone is fine while objects keep their distance and their boxes overlap frame to frame; it breaks when (a) a box leaves the frame or is clipped, (b) objects cross, or (c) the per-frame displacement approaches the box size. Then run the dropout experiment: 5 objects with 20% missing detections and max_age 3 gives 1 ID switch — a motion model or a memory bank is what closes that gap.
(Medium) Add a constant-velocity predict before association. Show that short 2–3 frame occlusions no longer cause ID switches, and derive exactly when they still do.Show one worked answer
Keep (vx, vy) for each track: on every update set v = Δcentre / gap-in-frames, and when a track is unmatched, shift its box by v for each frame that passes. Prediction helps because the stale-box IoU decays with the gap: without prediction the next detection after a g-frame gap overlaps the last box by (w − g·v)/(w + g·v). Plug numbers in: a 40 px-wide object at 6 px/frame survives a 3-frame gap at (40 − 18)/(40 + 18) = 0.379 ≥ 0.30 and dies at a 4-frame gap at (40 − 24)/(40 + 24) = 0.250 < 0.30 — which is exactly the jogger in the playground, whose track dies at frame 14 when prediction is off and keeps its ID when prediction is on (the predicted box matches the detection at IoU 1.000). Prediction is not a licence to wait forever: it is only as good as constant velocity is true, so an accelerating object drifts by ½at², and after enough frames the predicted box stops overlapping its own detection; SAM 2's memory bank is the answer for gaps longer than a motion model can bridge, which is why the cyclist in the playground needs the *second stage plus* prediction — 22 px/frame against a 50 px box gives a one-frame stale IoU of 0.389, below the second stage's 0.5 gate.
(Hard) Run a memory-based tracker (SAM 2 via transformers) beside the simple tracker on a 30-second crowd clip, manually label 5 salient people, and compare ID switches on those five.Show one worked answer
Setup that makes the comparison fair: fix the frame range and resolution, run both trackers on the same frames, and label the five people by hand in every frame (~450 annotations for 30 s at 30 fps) so that a switch is objective. Report per person: number of ID switches, longest correctly-tracked run, and the frames lost. Expected shape of the result: the memory tracker wins on the two people who are occluded for more than 5–10 frames (it is not asked to find a box at all, it is asked to keep segmenting the instance), while the detector-plus-association stack wins on latency — ByteTrack runs at 30 fps on a single GPU while a SAM 2 forward pass per frame is several times heavier, and its memory is per instance: with 7 slots per person and 64 × 64 × 64 features per slot, five people is ≈ 37 MB of memories, fifty people ≈ 367 MB before masks. The honest conclusion is not 'one wins': use memory when occlusion and masks dominate, ByteTrack when frame rate and object count dominate. Also log the failures you did not expect (people leaving frame, identical clothing) rather than only the switches, and state the frame range and the label protocol or the number means nothing.
(Medium) Write your own evaluation for the metric lab's six-frame, three-object sequence. Recompute MOTA, IDF1 and HOTA by hand, then check them against py-motmetrics (MOTA, IDF1) and TrackEval (HOTA), and explain why the three numbers disagree.Show one worked answer
The messy run has GT = 3 objects × 6 frames = 18 instances, TP = 11, FN = 7, FP = 1, IDSW = 1. MOTA = 1 − (7 + 1 + 1)/18 = 9/18 = 0.500. For IDF1, match trajectories globally: A pairs with one of its two predicted fragments for 3 frames (so IDTP loses the other 3), C pairs for 5 of its 6 frames, B pairs with nothing. IDTP = 3 + 5 = 8, total predicted frames = 12, total GT frames = 18, so IDFP = 4, IDFN = 10 and IDF1 = 2·8/(2·8 + 4 + 10) = 16/30 = 0.533. HOTA at the one IoU 0.5 threshold used in the lab: DetA = TP/(TP + FN + FP) = 11/19 = 0.579; the association term is TrackEval's per-trajectory Jaccard average, AssA = Σ matched·A(pair)/TP with A = m/(gt + pred − m) → (3·0.5 + 3·0.5 + 5·0.833)/11 = 0.652, so HOTA = sqrt(0.579 × 0.652) = 0.614. They disagree by construction: MOTA charges one identity change 1/18 and reports 0.500 while the identity actually held for 8 of 12 predicted frames, and HOTA's geometric mean is not comparable to a per-detection error rate — 0.614 does not mean 'better than MOTA 0.500'. Verify with py-motmetrics (it supports MOTA and IDF1 from a MOTChallenge-format file) and TrackEval for HOTA, and remember that TrackEval averages HOTA over 19 localisation thresholds while the lab computes one; the difference between the two numbers is the localisation term, and it is worth stating which protocol you report.
Terms this lesson borrows from later lessons (or outside)
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
IoU — Intersection over union of two boxes: the similarity every association cost here is built from, cost = 1 − IoU. (Phase 4, Lesson 06)
Confidence and NMS — The detector's score and the de-duplication step that decides which boxes even reach the tracker. ByteTrack's whole idea is to split this stream at 0.5 instead of throwing the low half away. (Phase 4, Lesson 06)
Instance masks — Per-object pixel masks instead of boxes — what a memory tracker outputs, and why annotation, not just counting, is its home turf. (Phase 4, Lesson 08)
Promptable segmentation — Click, box or text in, mask out. SAM 2 extends the promptable interface along time: prompt once, and the memory carries the instance forward. (Phase 4, Lesson 24)
Cross-attention — One sequence queries another: SAM 2's memory attention lets the current frame's features query the stored memories of the instance, which is exactly how 'which detection is this?' disappears as a separate step. (Phase 7, Lesson 02)
Embeddings and cosine similarity — Appearance-based tracking (DeepSORT, StrongSORT) matches detections by the cosine similarity of ReID feature vectors, fused with IoU and motion gating. The vector-similarity arithmetic is the same one used for retrieval. (Phase 11, Lesson 04)
KEEP GOING
A picture is a start. Practice is the rest.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 04, Lesson 27) and the Math Foundations Notebook reference build. The five labs — the tracker playground, the association stepper, the occlusion simulator, the metric calculator and the family chooser — are original to this page, as are the arithmetic they compute: the stale-box IoU ladder ((w − g·v)/(w + g·v), so a 40 px box at 6 px/frame holds 0.379 at a 3-frame gap and fails at 0.250 after four), the stage-gate correction for ByteTrack (confidence down to 0.1, overlap up to 0.5 by the reference implementation's cost limits of 0.8 and 0.5 on 1 − IoU), the two-scene stepper frame (the crossing with its two 0.316 trap cells, and the occlusion frame where greedy hands the jogger's box to the car and the jogger's ID dies), the metric row computed by hand (MOTA 1 − 9/18 = 0.500, IDF1 16/30 = 0.533, DetA 0.579, AssA 0.652, HOTA 0.614, with the negative-MOTA case 1 − 48/18 = −1.667), the Kalman arithmetic (predict (100, 200, 50, 120) with velocity (6, 2, 0, 0) → (106, 202, 50, 120), 64 covariance numbers per track, and the original SORT code's 7-dimensional parameterisation), the SAM 2 memory arithmetic (num_maskmem = 7 = one prompted frame plus six recent, 64 × 64 × 64 = 262,144 values ≈ 1.05 MB per slot, ≈ 7.3 MB per instance, ≈ 367 MB for 50 instances), and the source's synthetic experiment run from the vendored code/main.py at seed 0 (3/10/30 objects → 0/2/18 ID switches and 3/10/70 live tracks — three objects never fragment, 30 crowd each other and clip at the frame edges; 5 objects with 20 % dropouts at max_age 3 → 1 switch). Every number shown is computed live by the labs or verified by hand in the prose.