Seven stages — decode, detect, crop, classify, aggregate, validate, respond — stitched with a data contract at every seam. This capstone wires the phase together: Mask R-CNN and ConvNeXt-Tiny behind a Pydantic result, every failure path named, every stage timed, and a minimal FastAPI service with a health check and a worker count that survives the queue.
Decode the upload, detect at 800 × 800 with a 512-RoI cap, crop and resize to 224 × 224, classify the batch, join the answers by index, validate the contract, return JSON. Only two stages need a GPU — which is exactly why the bugs live in the other five.
7.8 + 68.0 + 3.5 + 10.0 + 2.5 = 91.7 ms · p95 133.0 msDetection carries a box, a score and a class id; Classification carries the detection index it belongs to; PipelineResult carries the response plus the version that produced it. Ranges and lengths are free; the box convention needs a validator with a sentence a human can read.
x1 < x2 ≤ W · 0 ≤ score ≤ 1 · class_id ≥ 0 · detection_index, never the crop indexEmpty detections and tiny crops are facts about the image and still return 200 — with a named code and a counted skip. Corrupt uploads and oversized files belong to the client and return 400/413. Model-load failures happen at startup, not inside somebody's first request.
guards on: 100 requests → 100 named responses · guards off: 50 errors + 50 silently wrongA vision product is a chain of contracts: two model stages joined to five ordinary ones, where the response schema, the crop-to-detection index and the failure codes are the load-bearing parts — and the median latency is the demo while the p95 is the product.
By the end you will be able to sketch the seven-stage pipeline and say what contract sits at each arrow; write the three Pydantic models with the guards that catch a (cx, cy, w, h) regression and a score of 1.41; wire a detector to a classifier, batched, with the crop-to-detection index map intact; time every stage and read p50 and p95 as two different objects; give every failure a name and a status; ship a FastAPI service that loads at startup, stays cheap on /healthz and degrades instead of dying; and size its workers from the p95 budget rather than from the capacity arithmetic.
A retail shelf audit is a detector plus a product classifier plus a price-OCR pipeline. A medical pre-screen is a segmenter plus a region classifier plus a clinician UI. This capstone builds the minimum viable version of that shape — and every failure you meet while wiring it is a failure you will meet in production.
Wiring models together is the part that separates a prototype from a product. Every interface between two stages is a new place for bugs: every coordinate transform, every normalisation, every mask resize is a silent-failure candidate. A pipeline is as strong as its weakest interface, and the interfaces are ordinary code — which is why they get less attention than the models and cause more incidents.
The plan is deliberately small, because a capstone you can finish beats an architecture you can only describe. Detect with a pretrained Mask R-CNN (or swap in YOLO) at an 800 × 800 input, capped at 512 regions of interest. Classify each crop with a pretrained ConvNeXt-Tiny at 224 × 224, batched. Emit structured JSON through a Pydantic contract with every failure path named. Then ship it behind a FastAPI upload endpoint with a health check, and benchmark it end to end until you can name the first bottleneck with a number.
Seven stages, and only two of them are models. The table is the whole design document: what each stage receives, what it must produce, and the specific bug that hides in it.
| # | stage | in → out | the bug that lives here |
|---|---|---|---|
| 1 | decode + preprocess | upload bytes (1.2 MB JPEG) → (3, 800, 800) float32 in [0, 1] | EXIF rotation, BGR/RGB swap, no draft-mode resize |
| 2 | detectorMODEL | (3, 800, 800) → boxes / scores / labels, ≤ 512 RoIs (our explicit box_detections_per_img cap; the default is 100) | coordinate convention, score scale, NMS skipped |
| 3 | crop + resize | boxes in absolute pixels → crops at 224 × 224 | inverted boxes, out-of-frame slices, tiny crops |
| 4 | classifierMODEL | (crops, 3, 224, 224) → crops × 1,000 logits | normalisation mismatch, missing .eval(), batch order |
| 5 | aggregate | detections + predictions → detections + classifications, index-mapped | crop index used as detection index |
| 6 | validate | PipelineResult fields → typed result or a field-path error | a guard you did not write |
| 7 | respond | validated result → JSON + trace id + timing | payload size, no way to trace a slow request |
Everything else in Phase 4 slots into this skeleton: swap Mask R-CNN for YOLOv8, add an OCR head for the price tags, add a segmentation branch, add a tracker for video. The architecture is stable; the pieces are pluggable. That is the whole reason to build the skeleton properly once.
import time
from pydantic import BaseModel
class Detection(BaseModel):
box: tuple[float, float, float, float] # (x1, y1, x2, y2), absolute pixels
score: float # [0, 1]
class_id: int # index into the detector's label map
class Classification(BaseModel):
detection_index: int # which detection this crop came from
class_name: str
score: float
class PipelineResult(BaseModel):
image_id: str
detections: list[Detection]
classifications: list[Classification]
inference_ms: float
def run(image, image_id: str) -> PipelineResult:
t0 = time.perf_counter()
tensor = preprocess(image) # decode, convert, resize, normalise
det = detect(tensor) # boxes, scores, labels
crops, index_map = crop(tensor, det) # crop, resize to 224x224
preds = classify(crops) # one batched forward pass
result = aggregate(image_id, det, preds, index_map)
result.inference_ms = (time.perf_counter() - t0) * 1000
return result # contracts checked on the way outA data contract is five seconds of code and an hour of debugging saved, every time. It is also the only place in the pipeline where you get to decide what “valid” means before an uncalibrated score or an inverted box decides for you.
The capstone’s contract has three models. Detection is one box with a score, a class id and an optional mask. Classification is one classifier answer, carrying the detection_index it belongs to. PipelineResult is the response: image id, detections, classifications, inference time. Nothing exotic — and every field is a decision you now only have to make once.
Be honest about what a type system can and cannot catch, because the difference decides which guard you write next. Pydantic validates types, ranges and lengths for free: Field(ge=0, le=1) rejects a 1.41 score, class_id: int = Field(ge=0) rejects the −1 that would otherwise index class_names[-1] — valid Python, silent nonsense, a fish labelled as a shelf. But four floats in a tuple validate in any order: a detector that switches to (cx, cy, w, h) passes every type check ever written. The convention needs a validator of its own, with a sentence a human can read.
from typing import List, Optional, Tuple
from pydantic import BaseModel, Field, model_validator
class Detection(BaseModel):
box: Tuple[float, float, float, float] # (x1, y1, x2, y2), absolute pixels
score: float = Field(ge=0, le=1) # a probability, not a logit
class_id: int = Field(ge=0)
mask_rle: Optional[str] = Field(default=None, max_length=65536)
@model_validator(mode="after")
def box_is_a_box(self) -> "Detection":
x1, y1, x2, y2 = self.box
if x2 <= x1 or y2 <= y1:
raise ValueError(
f"box is inverted: x2 ({x2}) <= x1 ({x1}) or y2 ({y2}) <= y1 ({y1})"
" - is the detector returning (cx, cy, w, h)?"
)
return self
@model_validator(mode="after")
def sanity_bounds(self) -> "Detection":
x1, y1, x2, y2 = self.box
if x1 < 0 or y1 < 0 or x2 > 100_000 or y2 > 100_000:
raise ValueError("box escapes a plausible pixel range; clamp to the frame before validating")
return self
class Classification(BaseModel):
detection_index: int = Field(ge=0) # the detection this crop came from
class_id: int = Field(ge=0)
class_name: str
score: float = Field(ge=0, le=1)
class PipelineResult(BaseModel):
image_id: str = Field(min_length=1, max_length=128)
contract_version: str # service + weights hash
detections: List[Detection]
classifications: List[Classification]
inference_ms: float = Field(ge=0)
# five seconds of code, and the boundary now names the field it refused
result = PipelineResult(
image_id="shelf_0042.jpg",
contract_version="maskrcnn-r50fpn-v2+convnext-tiny@2026-03",
detections=[Detection(box=(40, 120, 200, 420), score=0.93, class_id=0)],
classifications=[],
inference_ms=91.7,
)
print(result.model_dump_json(indent=2)[:120])Three more decisions belong in the contract rather than in a wiki. Version it: put the model name and the weights hash in every response, so a stored result can always be traced to the weights that produced it. Cap the payload: masks are the reason a 4 KiB response becomes a 14 MB one, so max_length=65536 per mask is a budget, not a formality. Name the failure: a contract violation returns 400 with the field path, and the client learns it sent something the service cannot accept — instead of receiving a plausible 200 built from an empty crop.
Six guards, six payloads. Toggle a guard off and the same bad payload stops being an error and becomes a silent bug somewhere downstream — which is exactly the trade every pipeline makes when it skips the schema.
class Detection(BaseModel): box: Tuple[float, float, float, float] # (x1, y1, x2, y2) absolute px x1 < x2, y1 < y2, sane bounds # model_validator score: float = Field(ge=0, le=1) class_id: int = Field(ge=0) mask_rle: str | None = Field(None, max_length=65536) class PipelineResult(BaseModel): image_id: str = Field(min_length=1, max_length=128) contract_version: str # maskrcnn-r50fpn-v2+convnext-tiny@2026-03 detections: list[Detection] classifications: list[Classification] inference_ms: float = Field(ge=0)
value_error.box_order loc=( detections.0.box ) box is inverted: x2 (120) < x1 (420) and y2 (240) < y1 (380) — the detector may be returning (cx, cy, w, h)
a one-line change in the detector wrapper: box = [420, 380, 120, 240] meaning centre (420, 380), size 120×240
A contract only catches what it names. Types, ranges and length caps are free; coordinate conventions and meanings need a validator with a human-readable message — which is why the bounds guard is the one that pays for itself.
A teammate changes the detector wrapper to return (cx, cy, w, h) instead of (x1, y1, x2, y2). Which guard in the contract catches it?
Between the two models sits the code nobody demos: clamp the boxes, skip the specks, crop, resize, batch, softmax, and — the step that breaks silently — remember which detection each crop came from.
A torchvision detector returns a list of dictionaries, one per image, with boxes (float32, (x1, y1, x2, y2) in absolute pixels), scores and labels. It has already run non-maximum suppression inside its forward pass, and this pipeline sets box_detections_per_img = 512 explicitly (torchvision’s default is 100). The RoI cap is part of the interface, not a detail: a dense shelf can produce thousands of candidates, and the cap is what keeps the response and the crop loop bounded — every extra box is another crop the classifier has to score. Clamp every box to the frame before slicing, in both axes: a box that ends at 1,024 in an 800-pixel image produces a slice that is silently truncated, and an inverted box produces an empty tensor that raises somewhere else entirely.
Then the small decisions that add up to quality. Crop with min_crop = 32: below that, upsampling 12 pixels to 224 × 224 invents texture and the classifier answers confidently about nothing. Resize with bilinear interpolation (matching how ConvNeXt was trained) and normalise with the weights’ own transform — weights.transforms() in torchvision — because a normalisation mismatch never throws; it just makes every label quietly worse. Then put all crops in one batch: ten crops in a single forward pass is 10.0 ms in the lesson’s model, against 46.0 ms for ten separate calls.
import torch
from torch.nn import functional as F
MIN_CROP = 32
CLASSIFIER_SIZE = 224
@torch.no_grad()
def crops_and_map(tensor, det, min_crop=MIN_CROP):
"""tensor: (3, H, W) float in [0, 1]; returns crops and their detection indices."""
_, height, width = tensor.shape
crops, index_map, detections = [], [], []
for i, (box, score, label) in enumerate(zip(det["boxes"], det["scores"], det["labels"])):
x1, y1, x2, y2 = [int(b) for b in box.tolist()]
x1, y1 = max(0, x1), max(0, y1) # clamp, do not trust the model
x2, y2 = min(width, x2), min(height, y2)
detections.append(Detection(box=(x1, y1, x2, y2), score=float(score), class_id=int(label)))
if x2 - x1 < min_crop or y2 - y1 < min_crop:
continue # named skip, not an exception
crop = tensor[:, y1:y2, x1:x2]
crop = F.interpolate(crop.unsqueeze(0), size=(CLASSIFIER_SIZE, CLASSIFIER_SIZE),
mode="bilinear", align_corners=False)[0]
crops.append(crop)
index_map.append(i) # <- the line that matters
return crops, index_map, detections
@torch.no_grad()
def classify_batch(crops, index_map, classifier, class_names):
"""The body of VisionPipeline.classify: one forward pass for the whole list."""
if not crops:
return []
batch = torch.stack(crops) # one forward for all crops
probs = classifier(batch).softmax(-1)
scores, ids = probs.max(-1)
return [
Classification(detection_index=index_map[k],
class_id=int(ids[k]), class_name=class_names[int(ids[k])],
score=float(scores[k]))
for k in range(len(crops))
]| det. | label | box | score ≥ 0.25 | min 32 px | crop → detection_index |
|---|---|---|---|---|---|
| 0 | shelf | 640 × 520 | 0.98 | ✓ | 0 → 0 |
| 1 | shelf | 240 × 540 | 0.95 | ✓ | 1 → 1 |
| 2 | product | 96 × 140 | 0.93 | ✓ | 2 → 2 |
| 3 | product | 104 × 132 | 0.91 | ✓ | 3 → 3 |
| 4 | product | 88 × 120 | 0.88 | ✓ | 4 → 4 |
| 5 | product | 112 × 148 | 0.84 | ✓ | 5 → 5 |
| 6 | product | 92 × 126 | 0.79 | ✓ | 6 → 6 |
| 7 | product | 100 × 138 | 0.72 | ✓ | 7 → 7 |
| 8 | product | 84 × 118 | 0.66 | ✓ | 8 → 8 |
| 9 | product | 96 × 128 | 0.58 | ✓ | 9 → 9 |
| 10 | product | 3 × 4 | 0.41 | ✗ | skipped |
| 11 | product | 90 × 130 | 0.19 | ✗ | gated out |
The classifier returns 10 rows, the detector produced 12 boxes, and 2 boxes were skipped as tiny crops. How do you attach each classification to the right detection?
A latency budget is only useful when it has line items. Time every stage, then look at the distribution rather than the average — the tail has a different owner than the median, and usually a different fix.
Three facts hold in nearly every vision pipeline. Preprocessing is the biggest surprise: decoding JPEGs, converting colour spaces and resizing are CPU-bound, scale with the source megapixels, and never look like something worth profiling. The detector dominates the GPU: 70–90% of accelerator time goes to the detection forward pass. Postprocessing is cheap on the GPU and expensive on the CPU: NMS and RLE encoding are fine until the masks arrive, and then the serialisation is the request. Knowing the distribution is what turns optimisation from a feeling into a prioritised list.
Read the p50 and the p95 as two different objects. The p50 is the request you ship in the demo; the p95 is the request a customer screenshots. In the spread above the mean (101.5 ms) sits above the median (93.6 ms) because a right tail pulls it there — and the tail is not noise, it has a cause: an 800 × 800 upload is 0.64 megapixels and a phone photo is 12, so the same pipeline spends 7.8 ms or 128.2 ms in preprocessing depending only on which file arrived. That single fact moves the p95 from 133.0 ms to 307.6 ms and turns a comfortable service into one that misses its budget by 23%.
So the first bottleneck is usually preprocessing, then the detector — and the only way to know which applies to you is to time every stage on the images you actually receive. The source’s benchmark loop is thirty lines and it is the highest-value code in the capstone.
import time
import numpy as np
import torch
def benchmark(pipe, images, num_runs=20):
stages = {name: [] for name in ("preprocess", "detect", "crop", "classify")}
for image in images[:num_runs]:
pipe.run(image) # warm up caches and kernels
sync = torch.cuda.synchronize if pipe.device == "cuda" else lambda: None
sync(); t0 = time.perf_counter(); tensor = pipe.preprocess(image)
sync(); t1 = time.perf_counter(); det = pipe.detect(tensor)
sync(); t2 = time.perf_counter(); crops, index_map, _ = pipe.crops_and_map(tensor, det)
sync(); t3 = time.perf_counter(); pipe.classify(crops, index_map)
sync(); t4 = time.perf_counter()
for name, delta in zip(stages, (t1 - t0, t2 - t1, t3 - t2, t4 - t3)):
stages[name].append(delta * 1000)
for name, times in stages.items():
times.sort()
p = lambda q: times[min(len(times) - 1, int(q * len(times)))]
print(f"{name:11s} p50={p(0.50):7.1f} ms p95={p(0.95):7.1f} ms")
# typical output on CPU, per the source: preprocess ~3 ms for a 400×600 image,
# detect 300-500 ms, classify 20-40 ms, total 350-550 ms.
# On a GPU with an 800² input the detector drops to tens of milliseconds and
# preprocessing becomes the stage worth attacking.Five stages, one budget. Move the source size, the detector and the classifier batch and watch the bottleneck change hands — the board names the stage that owns the request and the lever that moves it.
The p95 is the 19th of 20 requests in the lab’s fixed spread, not a measurement. Both tails below are real: a 12 MP source makes preprocessing the bottleneck, and dropping to batch 1 adds 36.0 ms without touching the model.
Your benchmark says p50 93.6 ms and p95 133.0 ms for the same pipeline. What is the most useful next move?
A production pipeline never returns “something went wrong”. Every failure class gets a code, a status and a log line — because a name is what lets a client decide between retrying, downscaling and asking the human to take a better photo.
The source lists five failure modes, and each one has a right answer that is not “catch everything”. Empty detections: return an empty list with a 200 and log the counts — an empty shelf is a fact about the image, not a server error. Out-of-bounds boxes: clamp to the frame before cropping. Tiny crops: skip classification below min_crop, keep the detection, and count the skip. Corrupt uploads: a 400 with image_decode_failed. Model load failure: fail at startup — a worker that cannot load its weights must never pass a readiness probe.
The scorecard is the argument. Run 100 requests through the failure simulator — 25 of each hazard — and with guards on every one of them is a handled, named response: 100 handled, 0 silently wrong, 0 errors. Turn the guards off and the same 100 requests become 50 errors and 50 silent wrong answers, because two of the hazards crash and two of them sail through as a confident 200. The silent half is the expensive half: a 500 gets noticed by an alert within minutes, while a fabricated label gets noticed by a customer weeks later.
MAX_UPLOAD_BYTES = 20 * 1024 * 1024 # 413 above this, before any decode
ALLOWED_TYPES = {"image/jpeg", "image/png", "image/webp"} # 415 otherwise
class PipelineError(Exception):
"""Every failure carries a name a client can act on."""
def __init__(self, code: str, status: int, detail: str):
super().__init__(detail)
self.code, self.status, self.detail = code, status, detail
def decode_or_400(data: bytes, content_type: str) -> np.ndarray:
if content_type not in ALLOWED_TYPES:
raise PipelineError("unsupported_content_type", 415, f"got {content_type}")
if len(data) > MAX_UPLOAD_BYTES:
raise PipelineError("image_too_large", 413, f"{len(data)} bytes")
try:
image = Image.open(BytesIO(data))
image.load() # force the decode here, not later
except Exception as exc: # noqa: BLE001 - this is the boundary
raise PipelineError("image_decode_failed", 400, str(exc)) from exc
return np.asarray(image.convert("RGB"))
# ...and inside the crop loop, the two soft failures:
# boxes are clamped to the frame before slicing (never raise)
# boxes smaller than min_crop are skipped, counted and logged (never raise)
# an empty detection list returns 200 no_detections with a trace id, not a crashFive requests cycle through the chain: one clean and four hazards. Switch the guards off and watch the same four hazards stop being named responses and turn into two 5xx crashes and two silently wrong 200s.
The two expensive rows are the silent ones. A 500 gets noticed in minutes by an alert; a 200 with a fabricated label gets noticed by a customer, weeks later.
| Hazard | Guarded response | Unguarded |
|---|---|---|
| no detections | 200 no_detections — detections: [] · classifications: [] · inference_ms: 78.2 · logged as a metric, not an error | 500 — classify([]) never gets called, but the response serializer was written assuming detections[0] exists |
| tiny crops | 200 tiny_crop_skipped — the detection stays in the response, the classification field for it stays absent, and the count is in the trace | 200 · silent — bilinear upsample to 224×224 manufactures a texture; ConvNeXt answers 'class 812, 0.51' and the response claims a confident label for 12 pixels |
| low-confidence boxes | 200 below_score_gate — the 0.25 gate runs before cropping: 23 candidates → 9 detections, and the crop count is 9, not 23 | 200 · silent — 23 crops include 14 background patches; the classifier labels them anyway and the JSON ships 14 confident-looking false products |
| malformed upload | 400 image_decode_failed — caught before the model: 3 ms, no GPU touched, and the client knows the file is theirs to fix | 500 — the decoder exception escapes the handler; the client retries the same broken file, and the retry storm lands on a service that already did not sleep |
FastAPI plus uvicorn is enough for a small product — if the models load at startup, blocking work stays off the event loop, and every error crosses the boundary as a status code a client can act on.
The service is deliberately boring. Models load in the lifespan hook, so a checkpoint that cannot load fails the deployment instead of a user’s first request. The endpoint checks content type and size before decoding anything, forces the decode at the boundary so a truncated JPEG becomes a 400 with a name, runs the pipeline, and returns result.model_dump() — the contract validating the response on the way out, for free. A /healthz endpoint answers in about 1.8 ms without touching the model, which is what makes it a readiness probe rather than a load test.
from concurrent.futures import ThreadPoolExecutor, TimeoutError as FutureTimeout
from contextlib import asynccontextmanager
from io import BytesIO
from fastapi import FastAPI, File, HTTPException, UploadFile
from PIL import Image
CLASSIFIER_BUDGET_S = 0.100 # after this, ship boxes without labels
_pool = ThreadPoolExecutor(max_workers=4)
@asynccontextmanager
async def lifespan(app: FastAPI):
global pipe
detector = maskrcnn_resnet50_fpn_v2(weights="DEFAULT").eval()
classifier = convnext_tiny(weights="DEFAULT").eval()
pipe = VisionPipeline(detector, classifier, class_names, device="cuda", min_crop=32)
yield
pipe = None
app = FastAPI(title="shelf-audit", version="2026-03", lifespan=lifespan)
@app.get("/healthz")
def healthz():
# cheap on purpose: no model call, so a busy worker still reports ready
return {"status": "ok" if pipe else "loading", "model_version": MODEL_VERSION}
@app.post("/detect")
def detect(file: UploadFile = File(...), image_id: str | None = None):
# def, not async def: FastAPI runs sync handlers in a threadpool, so a
# 91.7 ms of blocking inference does not stall the event loop.
if file.content_type not in {"image/jpeg", "image/png", "image/webp"}:
raise HTTPException(415, "unsupported_content_type")
data = file.file.read()
if len(data) > 20 * 1024 * 1024:
raise HTTPException(413, "image_too_large")
try:
image = Image.open(BytesIO(data))
image.load()
except Exception as exc:
raise HTTPException(400, f"image_decode_failed: {exc}") from exc
tensor = pipe.preprocess(image)
det = pipe.detect(tensor)
crops, index_map, detections = pipe.crops_and_map(tensor, det)
future = _pool.submit(pipe.classify, crops, index_map)
try:
classifications = future.result(timeout=CLASSIFIER_BUDGET_S)
except FutureTimeout:
classifications = [] # partial beats missing: boxes still ship
# note: the abandoned forward keeps running on the pool thread unless
# the backend supports cancellation, so size the pool accordingly
return PipelineResult(
image_id=image_id or file.filename or "upload",
contract_version=MODEL_VERSION,
detections=detections,
classifications=classifications,
inference_ms=0.0,
).model_dump()
# uvicorn main:app --host 0.0.0.0 --port 8000 --workers 4
# curl -F 'file=@shelf.jpg' -F 'image_id=shelf_0042' http://localhost:8000/detectSix real request shapes against the same service. Pick one and read the status line, the headers, the body and the per-stage trace — then switch the model load off startup and watch where 8.4 seconds goes. The 8,412 ms cold start and the 1.8 ms /healthz response are teaching numbers from the source’s reported ranges; measure your own.
ok91.7 ms end to end{
"image_id": "shelf_0042.jpg",
"contract_version": "maskrcnn-r50fpn-v2+convnext-tiny@2026-03",
"detections": 11,
"classifications": 10,
"skipped_tiny": [
10
],
"inference_ms": 91.7
}12 raw boxes → 11 above the 0.25 gate → 10 crops (1 tiny skip) → one classifier batch of 10
The body is the contract made visible: detections carry boxes, classifications carry detection indices, and the timing is inside the response so a slow client report can be tied to a stage without server access.
Worker sizing is where arithmetic beats instinct. One worker’s ceiling is 1 / 0.0917 s = 10.9 rps, so capacity arithmetic at 16 rps says two workers — and two workers sit at ρ = 0.73, which the queue turns into 253 ms of waiting and a p95 of 499.5 ms. Four workers put ρ at 0.37, the modelled wait at 53.2 ms and the p95 at 210.1 ms, inside the budget. The queue, not the model, is what the third and fourth workers buy.
Your endpoint is written as `async def detect(...)` but calls `pipe.run(image)`, which blocks for 91.7 ms of CPU and GPU work. What happens under load?
Every step exists because skipping it costs more later: the contract before the classifier, the timing before the optimisation, the failure paths before the launch. And “done” is a checklist with numbers on it, not a feeling.
The order matters more than the schedule. The contract first, because it is the interface every later step has to satisfy — and because the six payloads from the contract lab become the first tests. Then preprocessing and the detector alone, so real boxes reach real JSON before a second model is added. Then the crop map and the batched classifier, with the index test that fails if a skip is mis-mapped. Then the timers, because an optimisation without a per-stage breakdown is a guess. Then the failure paths, one at a time, each with a name. Only then the FastAPI wrapper and the health check — and finally the benchmark that signs off the latency budget.
“Done” is the same idea applied to the deliverable: a short list of checks that each carry a number or a name, so “looks good to me” is never the acceptance criterion.
| what “done” means | how you check it |
|---|---|
| The contract is versioned and enforced | every response carries contract_version + the weights hash; an invalid payload returns 400 with a field path |
| Every failure has a name | no_detections 200, tiny_crop_skipped 200, image_decode_failed 400, image_too_large 413, model_not_ready 503 |
| The latency budget is per-stage | preprocess ≤ 15, detector ≤ 100, classifier ≤ 10 per batch, validate ≤ 5, p95 ≤ 250 ms end-to-end |
| Golden-image tests pin the quality | 5 images with expected detection counts and class labels; a model swap that changes them fails CI |
| Load test at the target QPS | 100 requests at the production arrival rate: p50, p95, error rate and GPU utilisation recorded in the repo |
| The service degrades instead of dying | classifier timeout → detections-only 200; detector timeout → 504 with a trace id; health stays 200 |
Twenty requests at the arrival rate you choose. Push the QPS up and watch the tail separate from the median — then compare the worker count capacity arithmetic asks for with the count the p95 budget actually needs.
Set a micro-batch window and the service time grows by the window while the queue shrinks; batching trades latency for throughput, one request at a time.
Then the roadmap, which is not more architecture but more pieces in the same slots: swap Mask R-CNN for YOLOv8 and measure the p95 again (YOLO-Nano at 416² is 6 ms of detector, which makes preprocessing the bottleneck by default); add an OCR head after classification for the price tags; add a segmentation branch and put the masks behind the mask_rle budget from the contract; add a tracker for video, which is one more stage between detect and crop. The two measurements to keep repeating are the per-stage p50/p95 at your real arrival rate and the failure-code counts by name — those two tables catch almost everything a growing vision service can do wrong.
The contract question and the tiny-crop question are the two you will be asked in a code review. The preprocessing question and the batching question separate people who profile from people who guess; the 400-versus-500 question and the startup-load question are the two that show up in every incident postmortem.
0 / 6 answered · 0 correct
01Why does the capstone put a Pydantic model at every stage boundary?
02Which stage is most often the biggest *unexpected* latency block in a vision pipeline?
03A detection is 3×4 pixels and the classifier was trained on 224×224 crops. What should the pipeline do?
04You add a 10 ms micro-batch window in front of the classifier. What changes?
05A user uploads a truncated JPEG. What should the service return?
06Where do the detector and classifier get loaded?
Click a card to swap the lazy description for what it actually means.
Four problems with exact numbers — profile ten mixed-size images and choose the number for the status report, add masks and prove the payload stays under 1 MB, build the micro-batcher and report a gain and a cost, and write the acceptance suite that ends with a weighted p95 and a named bottleneck. Try first; a worked answer is one click away.
Instrument all five stages with time.perf_counter() around each block, store the arrays, and sort before computing percentiles. With the lesson's teaching model and its fixed 20-request spread, the 800×800 service time is 91.7 ms and the stage split is preprocess 7.8, detector 68.0, crop 3.5, classify 10.0, validate 2.5. Across the spread: p50 93.6 ms, p95 133.0 ms, mean 101.5 ms — and the two phone photos land at 212.1 ms each (preprocess 128.2) with a spread p95 of 307.6 ms, 23% over a 250 ms budget. The detection-count distribution is its own finding: the shelf photos return 12 raw boxes, 11 above the 0.25 gate, 10 crops after the min_crop filter — write the histogram down, because it decides the classifier batch size, not the model. The number for the status report is the p95 with the image-size mix stated beside it, because the mean hides exactly the two phone photos that your users are uploading. Reporting '96 ms average' while 1 request in 20 takes 308 ms is how a latency budget gets signed off and then fails in production.
A raw 800×800 binary mask is 640,000 bits = 80,000 B = 78.1 KiB. Ten of them are 800,000 B = 781.3 KiB, and JSON carries binary as base64, which inflates by 4/3: 781.3 × 4/3 = 1,041,667 B ≈ 1.04 MB — the raw option misses the 1 MB budget by about 4% before any other field is added, so RLE is not optional, it is the payload budget. COCO RLE stores a column-major run-length sequence as counts, and a typical product mask is a few thousand pixels in a few hundred runs: 200 runs × 2 B = 400 B per mask, ~4 KB for ten objects, ~5.3 KiB after base64 — a 195× reduction, and the reason the contract can cap mask_rle at 64 KiB per detection and still be generous. Verify with len(result.model_dump_json()) on a fixture image, assert < 1_000_000 in a test, and log the payload size per request: the 1.4 MB-per-mask row in the contract lab is what happens when the masks are never resized before encoding.
The measurable identity is expected requests per window = arrival rate × window. At 5 rps and a 10 ms window that is 5 × 0.01 = 0.05 requests, so 95% of windows fire with a single request: the same 10-crop forward (4 ms launch + 10 × 0.6 = 10 ms) runs 5 times per second, exactly as without the batcher, and every request pays a mean wait of window/2 = 5 ms plus up to 10 ms of p95. The honest report at 5 rps is a negative result: zero GPU-time saved, ~5 ms latency added — which is why the source asks you to measure instead of assuming. The same batcher at 200 rps collects 2 requests per window, halves the number of forward passes (100/s instead of 200/s) and saves roughly 400 ms of launch time per second, with per-request GPU work unchanged and the same 5 ms average wait. Build the in-request batch first — one forward for all 10 crops is 10 ms against 46 ms for 10 calls, a 36 ms saving with no waiting whatsoever — then add the cross-request window only when a load test shows the arrival rate makes it real. Report both numbers, gain and cost, or it is not a measurement.
Contract half: a clean payload is 200 with 3 detections; the (cx, cy, w, h) payload is 400 contract_violation at detections.0.box (the validator catches x2 < x1); score 1.41 is 400 at detections.0.score; class_id −1 is 400 at detections.0.class_id; an empty image_id is 400 because Field(min_length=1) exists; the 1.4 MB mask is 400 at detections.7.mask_rle. All six are one pytest param table against the TestClient, and the assertion is on the error code, not just the status. Load half: at 16 rps with 4 workers, ρ = 0.37, the modelled queue adds 53.2 ms, and end to end p50 is 147.8 ms with p95 210.1 ms — inside the 250 ms budget but only by 16%, which is the 'tight' band worth reporting. The trap the numbers expose: capacity arithmetic (16 rps × 0.0917 s) says 2 workers, and 2 workers give ρ = 0.73, a 252.8 ms queue wait and a 499.5 ms p95 — the queue, not the model, is what the third and fourth workers are for. Then the phone photos arrive: 12 MP preprocess is 128.2 ms against a 15 ms preprocessing band, service time 212.1 ms, and at 8 rps on 2 workers ρ = 0.85 gives a 1,188 ms queue wait and a 2,031 ms p95. The first stage to miss its budget is preprocessing, and the first fix is at the edge of the system: resize or decode at upload time, before the file reaches the pipeline.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 04, Lesson 16) and the Math Foundations Notebook reference build. The five labs — the latency board, the contract builder, the canvas failure simulator, the endpoint console and the benchmark histogram — are original to this page, as is the per-stage latency ledger (decode 6.4 + convert 0.4 + resize 0.6 + normalise 0.4 = 7.8 ms, detector 68.0 at 800², crop 3.5, classifier 10.0 in one batch of 10 against 46.0 in ten separate calls, validate 2.5, total 91.7 with p50 93.6, mean 101.5 and p95 133.0), the 12 MP phone-photo case (preprocess 128.2 ms, total 212.1 ms, p95 307.6 ms), the CPU comparison (Mask R-CNN 442 ms of a 494.9 ms request), the crop-to-detection index table (12 boxes → 11 gated → 10 crops with detection 10 skipped), the Pydantic honesty note with the six guard/payload pairs and their validation errors, the raw-mask payload arithmetic (800,000 B → 1.04 MB after base64), the micro-batcher arithmetic that shows why a 10 ms window buys nothing at 5 rps, the failure-path scorecard (100 requests: 100 named responses with guards, 50 errors and 50 silently wrong without them), and the worker-sizing gap (capacity arithmetic says 2 workers at 16 rps; the p95 budget needs 4, because the queue owns 253 ms at ρ = 0.73). Every number shown is computed live by the labs or verified by hand in the prose.