A training-time vision model is a floating-point monster: 25.6M parameters, 4.1 GFLOPs, 100 MB of weights. An edge device gives you 33.3 ms of frame budget, 2 GB of RAM and no tolerance for a slow tail. This lesson is the discipline of closing that gap — measured percentiles, honest FLOPs, INT8 post-training quantisation, an ONNX export that survives the trip, and the four backbones that fit.
1000 ÷ 30 = 33.3 ms per frame, split across preprocess, inference and postprocess. The device also has a memory ceiling (peak, not average) and a power ceiling (millijoules per inference). Every number is measured on the target device in percentiles — p50, p95, p99 — because a 12 ms mean with a 41 ms p99 still misses one frame in a hundred.
30 fps → 33.3 ms · 60 fps → 16.7 ms · report the tailShrink the architecture (MobileNetV3-Small is ≈68× fewer FLOPs than ResNet-50), quantise to INT8 (≈4× smaller, 2–4× faster, 0.1–1 point of accuracy), and pick the runtime that matches the silicon: ONNX Runtime for neutral, TensorRT for NVIDIA, Core ML or TFLite for phones. Pruning and distillation come last, when the first three still leave you short.
0.06 vs 4.1 GFLOPs · INT8 4× smaller · <1 point budgetPyTorch → ONNX → the target runtime. The three failures that eat a week: a shape baked into the graph (fix with dynamic axes), an operator the exporter cannot translate (fix by replacing it, or by scripting the control flow), and an opset version one side of the toolchain does not speak (fix by pinning 17 or upgrading the runtime). None of them is fixed by retraining.
opset 17+ · dynamic axes · verify numerics, not just the file sizeA real-time model is a budget you spend, not a benchmark you admire: 33.3 ms of frame, 2 GB of RAM and about one point of accuracy — and every optimisation you skip shows up later as a dropped frame in the p95.
By the end you will be able to measure latency the way production does (warmup, fixed input size, percentiles, synchronisation) and say why the mean is the last number you look at; compute a convolution’s FLOPs by hand and explain why depthwise models beat their own FLOPs estimate on memory bandwidth; run PyTorch post-training static quantisation — fusion, per-channel scales, 100–500 calibration images — and hold the accuracy loss under one point; export to ONNX at opset 17 and diagnose the three classic export failures; and pick between MobileNetV3, EfficientNet-Lite, ConvNeXt-Tiny and MobileViT-S from a device RAM, a frame-rate target and an accuracy floor.
A training-time vision model is a floating-point monster: 100M parameters, 10 GFLOPs, 2 GB of VRAM. A phone, a car’s infotainment unit, an industrial camera and a drone all offer the same thing instead — a hard frame deadline, a hard RAM ceiling, and no patience for the tail. Measurement comes before every optimisation, and the mean is the last number you look at.
The same predictions have to fit in a budget about 100× smaller than the training machine’s. Three knobs do most of the work: model choice (a smaller architecture with the same recipe), precision (INT8 instead of FP32) and the runtime (ONNX Runtime, TensorRT, Core ML, TFLite). Getting them wrong is the difference between a demo that runs on your workstation and a product that ships on a $30 camera module. But before any of the three, you need the discipline of the budget itself.
There are three budgets, and a shipping decision needs all three measured on the target device — never on the workstation:
| budget | what you measure | why it kills you |
|---|---|---|
| Latency | p50, p95, p99 milliseconds per frame | real-time systems fail on tails; the mean hides exactly the frames a human notices |
| Peak memory | the maximum resident set the device ever sees | OOM is fatal on embedded targets — there is no swap file to save you |
| Power / energy | millijoules per inference; usually proxied by utilisation × time | a battery device that thermally throttles delivers a different latency curve an hour in |
The frame arithmetic is one division, and it is worth writing down because everything else is measured against it:
Note what the last line implies: a pipeline that is 12% over budget loses about 10% of frames, and there is no scheduling trick that recovers them. Frame queues turn dropped frames into latency, and latency is the thing you were trying to bound. The only real fixes are a cheaper model, a smaller frame, or a faster numeric type.
The second discipline is percentiles. Latency distributions in vision are right-skewed — most frames are fast, a few are far slower, because of cache misses, another process on the CPU, thermal throttling or a bus conflict. A lognormal model captures the shape well enough to reason with: with median p50 and log-spread σ, the tail is p95 = p50 · e^(1.645σ) and p99 = p50 · e^(2.326σ). Two models, same median, different σ:
This is also where throughput and latency stop being the same thing. Batching amortises the fixed cost of a forward pass — kernel launches, framework dispatch, weight streaming — across several images, which raises throughput and raises latency at the same time:
import time
import torch
def measure_latency(model, input_shape, device="cpu", warmup=10, iters=50):
"""p50 / p95 / p99 in milliseconds, the way production reads them."""
model = model.to(device).eval()
x = torch.randn(input_shape, device=device)
with torch.no_grad():
for _ in range(warmup): # cold caches and JIT lie about speed
model(x)
if device == "cuda":
torch.cuda.synchronize() # otherwise you time dispatch, not execution
times = []
for _ in range(iters):
if device == "cuda":
torch.cuda.synchronize()
t0 = time.perf_counter()
model(x)
if device == "cuda":
torch.cuda.synchronize()
times.append((time.perf_counter() - t0) * 1000)
times.sort()
return {
"p50_ms": times[len(times) // 2],
"p95_ms": times[int(len(times) * 0.95)],
"p99_ms": times[-1],
"mean_ms": sum(times) / len(times),
}Split one frame at 30 fps across preprocess, inference and postprocess. The dashed line is the budget; the top strip samples the latency tail, the bottom strip shows which arriving frames find the device still busy. Add jitter to separate the tail from the median.
Inference is usually 80–95% of the budget; pre- and post-processing are the cheap parts and the ones people forget to count.
A model benchmarks at mean 12 ms per image, p99 41 ms, on a 33.3 ms budget. What does the p99 tell you that the mean does not?
Counting multiply-adds is cheap, device-independent and useful for sorting architectures. It is also wrong as a prediction, because hardware does not charge for arithmetic alone: it charges for weights streaming out of memory, for activations that have to be materialised, and for every kernel launch between them.
A convolution’s FLOP count has a closed form. For a dense layer with C_in input channels, C_out output channels, kernel k×k and output H×W:
That last line is the whole problem with FLOPs as a predictor. Params and FLOPs do not even agree with each other: ResNet-50 is 10.2× the parameters of MobileNetV3-Small but 68× its FLOPs. Params are a memory bill, FLOPs are an arithmetic bill, and on a real device the two are paid from different accounts. Four effects break the correlation, and every one of them has killed a shipping deadline:
The rule the source states is the one to keep: use FLOPs for architecture search, use on-device latency for deployment decisions. FLOPs will tell you that MobileNetV3-Small is worth trying before ConvNeXt-Tiny on a camera-class chip. It cannot tell you whether either one ships at 30 fps — only a stopwatch on the target can.
Arithmetic intensity is FLOPs divided by bytes moved. Take two 3×3 convolutions with 96 channels at 112×112 and count both sides:
This is the teaching version of the roofline argument, with round activation numbers and a labelled 2× safety factor; the ordering is what matters, because it explains why the FLOPs ranking and the latency ranking disagree in exactly the direction they do.
import torch
import torch.nn as nn
def flops_estimate(model, input_shape):
"""Rough conv + linear FLOP count. For production use fvcore or ptflops."""
total = [0]
def conv_hook(m, inp, out):
c_out, c_in_per_group, kh, kw = m.weight.shape
h, w = out.shape[-2:]
# Groups matter: a depthwise conv reads one input channel per output
# channel, so c_in_per_group = 1 rather than c_in.
total[0] += 2 * c_in_per_group * c_out * kh * kw * h * w
def linear_hook(m, inp, out):
total[0] += 2 * m.in_features * m.out_features
hooks = []
for m in model.modules():
if isinstance(m, nn.Conv2d):
hooks.append(m.register_forward_hook(conv_hook))
elif isinstance(m, nn.Linear):
hooks.append(m.register_forward_hook(linear_hook))
model.eval()
with torch.no_grad():
model(torch.randn(input_shape))
for h in hooks:
h.remove()
return total[0]
# for real projects: fvcore.nn.FlopCountAnalysis or ptflops handle every moduleSelect up to four backbones, pick the target device and the numeric type, and read the three bars against each other. The pale ghost behind each latency bar is what the FLOPs alone would predict; the gap is where memory bandwidth lives.
The efficiency factors are teaching numbers: depthwise-heavy mobile models land near 0.3 of the device’s dense-conv peak, dense convs near 0.85. Published params, FLOPs and top-1 are the papers’.
Model A has 0.5 GFLOPs and model B has 2.0 GFLOPs, both trained on the same data with the same top-1. On a phone, A is slower. How?
Replace FP32 weights and activations with 8-bit integers plus a scale. The file is 4× smaller, the memory traffic is 4× lower, and the arithmetic runs 2–4× faster on every modern SoC, GPU and DSP — for a loss that lands between a tenth of a point and one point when the calibration is done right.
The idea is a change of units, not a change of model. A tensor’s values are mapped onto the 256 integers a byte can hold, using one scale (and, for activations, one zero point) per block of values:
Two things fall out of that arithmetic. First, quantisation is not only about arithmetic: on bandwidth-bound models it is a memory win, which is exactly where depthwise architectures live. Second, the win scales with how much of the frame is inference. Put it on the budget:
There are three ways to get there, and the difference is who pays for the rounding error:
| flavour | what is quantised | effort | when |
|---|---|---|---|
| Dynamic | weights to INT8; activations computed in FP | one function call, no calibration | a quick size win; small speedup, mostly servers |
| Static (PTQ) | weights and activations, ranges from a calibration set | four steps, 100–500 unlabelled images, minutes | the default for vision — the source calls it 95% of the benefit for 5% of the effort |
| QAT | rounding simulated inside the training loop | labels plus a full training run | only when PTQ costs more than the 1-point budget |
Static PTQ is four steps, and the order matters: fuse (Conv + BN + ReLU becomes one module, so the quantiser observes one activation range and the pipeline needs one fewer requantisation), prepare (insert observers that record activation min/max while the model runs), calibrate (100–500 representative images, no labels), convert (turn observed ranges into scales and swap in INT8 kernels). Two details decide whether you land at 0.2 points or 4:
The last thing to know is where INT8 struggles: normalisation layers (LayerNorm has no cheap integer version in older runtimes), smooth activations (GELU, swish), un-fused BatchNorm, and attention blocks whose softmax saturates. Convolution stacks with fused batch norm — the MobileNet / EfficientNet / ResNet family — quantise almost for free. Hybrid and transformer-flavoured models (MobileViT, ConvNeXt) quantise well too, but they are where you verify the loss instead of assuming it.
import torch
def quantise_ptq(model, calibration_loader, backend="x86"):
"""Post-training static INT8: fuse, configure, calibrate, convert."""
import torch.ao.quantization as tq
model = model.eval().cpu()
# 1. fuse Conv + BN + ReLU into one module: fewer ops, cleaner ranges.
# (torch.ao.quantization.fuse_modules handles whole graphs; the point is
# that an un-fused BatchNorm leaves two quantised ops where one will do,
# doubling the per-op activation ranges and the requantisation steps
# between them, and each extra range and rescale adds rounding error.)
tq.fuse_modules(model, [["features.0.0", "features.0.1"]], inplace=True)
# 2. configure and prepare: observers are inserted after every conv.
model.qconfig = tq.get_default_qconfig(backend) # "x86" (FBGEMM) | "qnnpack" (ARM)
tq.prepare(model, inplace=True)
# 3. calibrate on 100-500 representative images — labels are NOT needed.
with torch.no_grad():
for x, _ in calibration_loader:
model(x)
# 4. convert: observers become scales, fused modules become INT8 kernels.
tq.convert(model, inplace=True)
return model
# dynamic: weights INT8, activations computed in FP — weights only, small speedup
# qat: simulate rounding inside training — best accuracy, needs labels
# modern: torch.ao.quantization.quantize_fx (graph mode) or the torchao toolkit
# quantify the same three steps with fewer manual fusionsTake one weight tensor to INT8: the accent staircase is what the hardware would actually store. Add wide output channels, drop the calibration set, and watch the predicted accuracy drop cross the one-point budget — then fix it with per-channel scales.
Your PTQ INT8 model loses 4 points of top-1. The calibration loader has 12 near-black images in it. What is the first thing to change?
PyTorch is a development runtime; the device will not have it. The production path is PyTorch → ONNX → the runtime that matches the silicon, and the trip has exactly three ways to fail: a shape baked into the graph, an operator the exporter cannot translate, and an opset version one side of the toolchain does not speak.
ONNX is the lingua franca: one file format that ONNX Runtime, TensorRT, Core ML, TFLite and OpenVINO all read. The runtime is chosen by the hardware, not by taste:
| runtime | hardware | notes |
|---|---|---|
| PyTorch eager | your workstation | development only; 200 dispatcher hops per frame is not a deployment story |
| TorchScript | anywhere the ONNX path runs | legacy, superseded by torch.compile and ONNX export — but still the tool that puts control flow into a graph when tracing cannot |
| ONNX Runtime | CPU, CUDA, Core ML, TensorRT, OpenVINO via providers | the neutral default; start here and profile before reaching for anything else |
| TensorRT | NVIDIA GPUs, Jetson | best latency, biggest build effort; engines are compiled per GPU and are not portable |
| Core ML / TFLite | Apple / Android and ARM | quantise before export; the mobile delegates expect INT8 |
| OpenVINO | Intel CPU, iGPU, VPU | takes ONNX and emits its own IR; strong on x86 edge boxes |
The opset is the versioned set of operators the file may use, and it is the single most common source of version pain. Opset 17 is the conservative 2026 default — it is where LayerNormalization became a native op, which matters for every transformer-flavoured backbone in the picker. Newer opsets exist and newer exporters default to them; the rule is to export at the oldest opset that contains every op your model needs, because the runtime on the device is usually older than the toolchain on your laptop. When an op is missing, the exporter silently decomposes it into primitives — “failing fallback strategy chosen” — and the decomposed numerics can cost you points without raising an error.
The three families of export failure, with the fix for each:
| family | what you see | the fix |
|---|---|---|
| Dynamic shapes | “Got invalid dimensions for input: Expected 1 Actual 8”, or the 320×320 production stream is rejected | declare the axes in dynamic_axes (batch, and height/width if the resolution varies); give TensorRT a min/opt/max profile |
| Unsupported ops | “Unsupported operator: aten::…”, or an exported model that silently takes only one branch | replace the op with supported primitives, script the module so control flow enters the graph, or register a custom op |
| Opset mismatch | “Failing fallback strategy chosen”, or the target runtime rejects a graph over its opset ceiling | pin the export to the opset the device speaks (17 is the safe floor), or upgrade the runtime — then verify the numerics |
Two rituals turn export from a gamble into a step. First, numerical parity: run 50 random inputs through eager and through the runtime and assert the maximum absolute difference is under 1e-3. A file that passes the ONNX checker can still be a different model. Second, profile on the target: the same ONNX file can be 3× faster under TensorRT than under a CPU provider, and a TensorRT engine built on your workstation GPU will not load on a Jetson at all. Both rituals are cheap; both catch exactly the failures that survive to production.
import torch
def export_onnx(model, sample_input, path="model.onnx", dynamic_batch=True):
"""PyTorch → ONNX at opset 17, with the batch axis declared dynamic."""
model = model.eval() # train mode changes the graph
dynamic_axes = (
{"input": {0: "batch"}, "output": {0: "batch"}} if dynamic_batch else None
)
torch.onnx.export(
model,
sample_input, # traced ONCE: this shape is baked in
path, # unless you declare it dynamic
input_names=["input"],
output_names=["output"],
dynamic_axes=dynamic_axes, # {2: "height", 3: "width"} if the
opset_version=17, # resolution varies at runtime too
)
import onnx
onnx.checker.check_model(path) # structural check — not numeric
return path
# then, in ONNX Runtime:
# import onnxruntime as ort
# session = ort.InferenceSession("model.onnx", providers=["CPUExecutionProvider"])
# ort_out = session.run(None, {"input": x.numpy()})
#
# and numerically, because a valid file is not a correct model:
# assert (torch_out - torch.from_numpy(ort_out[0])).abs().max() < 1e-3Nine symptoms that show up between a working PyTorch model and a working ONNX file. Pick the one you are staring at; the board names the cause and the one-line fix.
Dynamic shapes
The trace baked one input size into the graph, so any other batch or resolution is refused.
Unsupported op
An operator in the model has no ONNX mapping, or tracing cannot see the control flow it lives in.
Opset mismatch
The op needs a newer (or older) opset than the exporter or runtime supports.
torch.onnx.export(
model,
sample, # traced shape: (1, 3, 224, 224)
"model.onnx",
input_names=["input"],
output_names=["output"],
dynamic_axes={"input": {0: "batch"}, "output": {0: "batch"}},
opset_version=17,
)The three families cover almost everything: a shape baked into the graph, an operator the exporter cannot translate, or an opset that one side of the toolchain does not speak. The fix is a declaration, an op swap, or a version pin — never a retrain.
Four backbones do almost all the work on real devices — MobileNetV3, EfficientNet-Lite, MobileViT and ConvNeXt-Tiny — with ResNet-50 as the reference point they beat. The numbers are public; what most teams get wrong is the order. Pick a model by reputation and you discover the weight bill, then the frame rate, then the export. Start from the constraint instead, walk up the accuracy ladder, and stop at the first model that clears the floor.
These are the published ImageNet-1k numbers for the architectures an edge deployment actually uses. Params set the weight bill; GFLOPs hint at the arithmetic; top-1 is the accuracy ceiling before your own data:
| model | params | GFLOPs | top-1 | why it exists |
|---|---|---|---|---|
| MobileNetV3-Small | 2.5M | 0.06 | 67.4% | designed for phones: depthwise convs, h-swish, squeeze-excite, quantisation-aware from day one |
| MobileNetV3-Large | 5.4M | 0.22 | 75.2% | the same recipe with more capacity — the default mobile workhorse |
| EfficientNet-Lite-B0 | 4.7M | 0.39 | 75.1% | best accuracy per parameter in the 3–10M band; Lite variants swapped swish for ReLU6 to be TFLite-friendly |
| MobileViT-S | 5.6M | 2.0 | 78.4% | hybrid conv + transformer; the best accuracy per megabyte in the zoo |
| ResNet-50 | 25.6M | 4.1 | 76.1% | the 2015 default — 2× MobileViT-S’s arithmetic for 2.3 fewer points |
| ConvNeXt-Tiny | 28.6M | 4.5 | 82.1% | the accuracy ceiling of the zoo; modern conv design, CPU-friendly, 114 MB of FP32 weights |
| Swin-V2-Tiny | 28.3M | 4.5 | 81.8% | window attention — only if your runtime handles the ops and the export survives |
The source’s picker is organised as parameter bands: under 3M take MobileNetV3-Small, 3–10M EfficientNet-Lite-B0, 10–20M ConvNeXt-Tiny, 20–30M MobileViT-S or EfficientViT, 30–80M Swin-V2-Tiny if the stack supports window attention. Read the bands as neighbourhoods rather than exact counts — MobileViT-S itself is 5.6M parameters — and then do the arithmetic that actually decides:
That flattening is the real decision rule. Pick the smallest architecture that clears the accuracy floor, quantise it, and only then consider stepping up — because the step from 75% to 78% can cost ten times the step from 67% to 75%. When even the smallest model cannot clear the floor, there are two more levers the source names: pruning (remove unimportant weights, or whole channels, from an over-parameterised model) and distillation (train a small student to mimic a large teacher’s logits — the standard way production edge models recover the points a small architecture lost). Both are training-time investments; quantisation and model choice are configuration-time ones, which is why they come first.
The order of operations for a real deployment, then: measure the frame budget on the target; set the weight ceiling from device RAM (halve it for the OS and activations); set the accuracy floor from business requirements, not from ImageNet; start at MobileNetV3-Small, quantise, measure; step up one row at a time until the floor is met. The chooser lab below does the same walk with sliders.
Three constraints — how much RAM the device has, how many frames per second it must sustain, and the accuracy floor you cannot go under — plus the numeric type. The board filters the zoo and recommends the most accurate model that clears all three.
| model | weights | latency | fps | accuracy | verdict |
|---|---|---|---|---|---|
| MobileNetV3-Small fits, fast enough, accurate enough | 2.5 MB | 3.7 ms | 270 fps | 66.9% | CLEARS |
| MobileNetV3-Large fits, fast enough, accurate enough | 5.4 MB | 9.8 ms | 102 fps | 74.7% | CLEARS |
| EfficientNet-Lite-B0 fits, fast enough, accurate enough | 4.7 MB | 15.1 ms | 66.1 fps | 74.6% | CLEARS |
| MobileViT-S only 21.4 fps of 30 fps | 5.6 MB | 46.7 ms | 21.4 fps | 77.9% | MISSES |
| ResNet-50 only 16.3 fps of 30 fps | 25.6 MB | 61.5 ms | 16.3 fps | 75.6% | MISSES |
| ConvNeXt-Tiny only 14.0 fps of 30 fps | 28.6 MB | 71.5 ms | 14.0 fps | 81.6% | MISSES |
| Swin-V2-Tiny only 10.5 fps of 30 fps | 28.3 MB | 95.0 ms | 10.5 fps | 81.3% | MISSES |
The zoo is the source lesson’s picker with published ImageNet numbers, and the device class is inferred from RAM — weights get half the RAM, the operating system and activations take the rest.
A team needs 30 fps at 224×224 on a $30 camera-class SoC and a 75% top-1 floor. Which zoo entry do they reach for first?
The last decision is not architectural — it is a profile: which providers are actually engaged, which layer is the slow one, and whether the model you exported is the model you trained. The source converges on three shipping paths, and all three end with the same table: model, latency, memory, accuracy.
Production stacks converge on one of three paths, and each one is a different promise about where the engineering effort goes:
For measurement, the tooling is the same discipline as chapter 01 with better instruments: torch.profiler / torch-tb-profiler for per-layer time on the eager side, nsys and nvprof for GPU kernels, Instruments on macOS, trtexec for TensorRT, benchmark_app for OpenVINO, and ONNX Runtime’s profiling flag for a per-node JSON you can diff against PyTorch. The first thing to check in any profile is not the slowest layer — it is whether the accelerator engaged at all. A provider list that falls back to CPU is the most common cause of a mysteriously slow deployment, and it is silent by design.
The ship checklist that comes out of the source, in order. Each line is cheap; each one catches a failure that costs days:
Two artifacts from the source make good habits, because both force the checklist into a reusable form: a deployment planner that takes target device, frame budget and accuracy floor and returns backbone, quantisation strategy and runtime; and a latency profiler that writes the warmup-and-percentile benchmark script with peak-memory tracking included. Neither is complicated — the value is that they are written once and run on every model, instead of being reinvented at 2 a.m. before a demo.
# ONNX Runtime: pick the provider, then verify it actually engaged
import onnxruntime as ort
session = ort.InferenceSession(
"model.onnx",
providers=[
("TensorrtExecutionProvider", {"trt_int8_enable": True}),
"CUDAExecutionProvider",
"CPUExecutionProvider", # the fallback that hides mistakes
],
)
print(session.get_providers()) # if CPU shows first, nothing accelerated
# TensorRT standalone: build the engine once, on the target, with a shape profile
# trtexec --onnx=model.onnx --int8 --saveEngine=model.plan \
# --minShapes=input:1x3x224x224 \
# --optShapes=input:1x3x224x224 \
# --maxShapes=input:8x3x320x320
#
# then measure the way chapter 01 taught: warmup, percentiles, peak memory
# trtexec --loadEngine=model.plan --iterations=100 --avgRuns=10 --dumpProfile
# ONNX Runtime's own profiler writes a per-node JSON you can diff against eager
# session_options.enable_profiling = TrueYour ONNX model loads cleanly, runs fast, and its top-1 drops 6 points against eager. What do you check first?
Answer before you look. The quantisation question and the export question are the two that separate “I followed the tutorial” from “I can debug a deployment at 2 a.m.”
0 / 6 answered · 0 correct
01You benchmark a model and report “average 12 ms per image on a 4090”. What is missing?
02FLOPs and on-device latency do not correlate perfectly. Why?
03Post-training static INT8 quantisation typically loses how much accuracy on ImageNet-class vision models?
04Your mobile app needs a vision model under 10 MB with sub-10 ms latency. Which backbone do you pick?
05You export a PyTorch model to ONNX with opset 17 and it fails with “Unsupported operator”. What are the most likely causes?
06Your freshly quantised MobileNetV3-Small loses 4 points of top-1, and the calibration loader holds 12 images, several of them near-black. What is the first fix?
Click a card to swap the lazy description for what it actually means.
Four problems with exact numbers: benchmark four backbones and read the accuracy-per-millisecond table, quantise MobileNetV3-Small PTQ and hold the loss under a point, race ONNX Runtime against PyTorch eager on ConvNeXt-Tiny, and work out where a four-stage pipeline actually blows its 33.3 ms budget. Try first; a worked answer is one click away.
The protocol matters more than the machine: model.eval(), torch.no_grad(), 10 warmup passes, time.perf_counter() around each of 50 passes, then sort and take times[25] and times[47]. Expected shape of the result on a laptop CPU: MobileNetV3-Small lands roughly an order of magnitude below ResNet-18, EfficientNetV2-S sits in between, ConvNeXt-Tiny is slowest per image on CPU but is the most accurate. Accuracy-per-millisecond is just top-1 ÷ p50 ms computed per row — on the published ImageNet values (67.4 / 69.8 / 83.5 / 82.1 for MobileNetV3-Small, ResNet-18, EfficientNetV2-S and ConvNeXt-Tiny) a typical laptop ordering puts MobileNetV3-Small first and ConvNeXt-Tiny last, which is exactly the trade the lesson is about: the smaller model wins on efficiency, the bigger one wins on accuracy, and no FLOPs table tells you where your device lands between them. Report p95 next to p50 for every row; if p95 is more than ~1.5× p50 on an idle machine, another process is sharing the CPU and the numbers are not usable.
The recipe is four lines plus a calibration loop: model.eval(), fuse Conv+BN+ReLU, model.qconfig = torch.ao.quantization.get_default_qconfig("qnnpack" for ARM, "x86" for server CPUs), torch.ao.quantization.prepare, run 100–500 calibration images through it (labels unused), then torch.ao.quantization.convert. Expectations from the source's numbers: weights shrink 4× (2.5M × 4 B = 10 MB → 2.5 MB), CPU latency improves roughly 2–4×, and top-1 moves by 0.1–1.0 points. On a 10-class CIFAR subset the interesting reading is the delta, not the absolute: if the drop is 3–4 points, look at the calibration set before anything else (12 near-black images set ranges that represent nothing), then at per-channel scales for the conv weights, then at whether BatchNorm was actually folded. A drop under 1 point with a 4× smaller file is the normal, expected outcome — and it is the number you quote in the deployment decision, not the FP32 accuracy.
Export with a fixed 1×3×224×224 sample (ConvNeXt does not need dynamic axes for a first run), input_names/output_names set, opset_version=17, then confirm the graph with onnx.checker and run both sides on the same 20 inputs. Typical finding: ORT is a few percent slower on the first pass and clearly faster after warmup, because the export lets ORT fold constants, fuse Conv+BN and pre-plan memory reuse, while PyTorch eager re-dispatches every op through the Python dispatcher. To find the first faster layer, profile rather than guess: ORT's --enable_profiling gives a per-node JSON and torch.profiler gives the eager side. The layer where the gap opens is usually the first fused Conv-BN block — in ConvNeXt-Tiny that is the stem's 4×4 convolution followed by LayerNorm. The honest takeaway: eager is a development runtime, ORT is a deployment runtime, and the size of the gap depends on your CPU and your batch size — which is why the source's rule is “measure on the target”, never “trust the export”.
The medians sum to 2 + 6 + 18 + 4 = 30 ms, which looks like a comfortable 3.3 ms of headroom — and that is the trap, because the p95 of a sum is not the sum of the p95s. Model each stage as lognormal with σ = ln(p95/median) ÷ 1.645: capture 0.246, preprocess 0.246, inference 0.331, postprocess 0.246. The variance of a lognormal is mean² · (e^{σ²} − 1), so the four stages contribute 0.25 + 2.26 + 37.4 + 1.00 ≈ 40.9 ms²; the combined standard deviation (normal approximation to the sum) is 6.4 ms and the p95 of the total is ≈ 30 + 1.645 × 6.4 ≈ 40.5 ms — over the 33.3 ms budget, with the inference stage contributing 91% of the variance. The fixes, in order: trim inference (quantise — a 2.5× speedup takes its mean to 7.2 ms and its variance to 6.0 ms², so the total p95 lands near 24 ms), then measure the end-to-end distribution with a timestamp on every frame rather than trusting the arithmetic. The lesson inside the exercise: means compose by addition, tails compose by quadrature, and the stage with the biggest mean usually owns the tail.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 04, Lesson 15) and the Math Foundations Notebook reference build. The five labs — the 33.3 ms budget board, the FLOPs/params/latency comparator, the INT8 quantisation playground, the export-failure triage board and the constraint-driven model chooser — are original to this page, as are the stage arithmetic (30 fps = 33.33 ms; a 4 + 26 + 3 ms pipeline spends 99% of the frame; 37 ms of work drops 3 frames in 30), the tail arithmetic (p95 = p50·e^(1.645σ), p99 = p50·e^(2.326σ), with the σ = 0.10 and σ = 0.35 examples), the convolution FLOP examples (ResNet-50's 7×7 stem at 0.236 GFLOPs ≈ 5.8% of the model; depthwise convs with a 16× arithmetic discount), the weight-streaming numbers (102.4 MB FP32 versus 25.6 MB INT8 at 50 GB/s), the accuracy-per-GFLOP frontier (23.3 / 5.7 / 3.3 / 2.2 points per added GFLOP) and the memory hook “PTQ has no labels, QAT does; dynamic leaves activations floating, static does not.” The latency distributions, device throughputs and architecture efficiency factors are labelled teaching models, because a real benchmark suite cannot be replayed inside a canvas; every other number is computed live by the labs or is a published figure quoted from the papers and documentation in the source list.