A box says where.
A mask says exactly which pixels.

The difference is not a detail of implementation; it is a different question. Semantic segmentation answers “what kind of thing is this pixel?” and merges instances of the same class — which is exactly right for road, sky, tumour, water, and useless for count-the-apples. Instance segmentation answers “which object is this pixel part of?”, keeps a separate mask and id per object, and only covers countable things: the grass, the sky and the road have no instances, so an instance model leaves them unlabelled. Panoptic segmentation is the union: class labels for stuff, ids for things, nothing left unlabelled.

Counting is the operational test. Cell biology, microscope fields, produce sorting, traffic counts, inventory audits, construction take-offs, wildlife camera traps — every one of them is a question of the form “how many, and where exactly is each one?” A detector answers “how many” with boxes; the masks add the exact silhouette, which is what you need when two objects touch or when the thing you measure is a shape, not a rectangle (the bounding box of each brick in a wall, each cell in a smear).

Mask R-CNN’s founding move was to refuse to build a new architecture. Faster R-CNN already finds objects: a region proposal network guesses where things are, and heads classify and refine each guess. Instance segmentation is that, plus answering “which pixels inside this box belong to it?” — so Mask R-CNN keeps the detector intact and adds one small branch whose only job is to paint a mask inside the box. The paper’s own framing is that clean, and it is why for the next five years almost every instance segmentation paper was a Mask R-CNN variant.

The hard part is not the mask branch — it is how you hand a box to a branch. A proposal is four floats; a feature map is a grid of integers. The crop that bridges them was, before 2017, computed with rounding, and that rounding quietly destroyed a large part of the localisation accuracy a mask needs. The fix is RoIAlign, and it is worth more than a chapter — it is worth understanding at the level of one bilinear sample, which is where this lesson goes next.

Five pieces, in order:

1. Backbone — ResNet-50 with its classifier removed. 23.5M conv and batch-norm parameters, feature maps at strides 4, 8, 16 and 32. It is the same kind of trunk Lesson 03 built and Lesson 05 fine-tuned, now feeding a detector instead of a single label.
2. FPN — the Feature Pyramid Network turns that single hierarchy into five outputs, P2 to P6, each with 256 channels, at strides 4 to 64. 3.3M parameters. Every level is a good feature map for one range of object sizes.
3. RPN — a 1.18M-parameter head (two 3×3 convolutions, then 1×1 objectness and delta convolutions) that, at every location of every level, scores three anchor shapes and predicts a delta box. One forward pass covers 159,882 anchors on an 800×800 image; the top 1,000 at inference (2,000 in training) go through NMS at IoU 0.7 and about 1,000 class-agnostic proposals reach the heads. “Class-agnostic” is the important word: at this point the network knows something is at a location, not what.
4. RoIAlign — the bridge from boxes to grids. For any proposal, on the FPN level that matches its size, it samples a fixed 7×7 (box head) or 14×14 (mask head) grid of 256-channel features at exact float coordinates, bilinearly. Chapter 04 is entirely about this step.
5. Heads — a box head (15.2M parameters: four 3×3 convolutions with batch norm and one 1,024-wide fully connected layer) that produces 91 class logits and 364 box deltas per proposal, and a mask head that produces 28×28 per class. Two predictions from the same aligned crop, trained with different losses.

The two stages are not two networks: they are one network whose output depends on model.training. In training mode the forward pass returns a dict of five losses; in evaluation mode it returns a list of per-image dicts with boxes, labels, scores and masks. Same weights, different outputs — which is why the training loop is only five lines.

A single feature map forces a single scale. At stride 32 — the last ResNet stage — a 32×32 object is one cell: the classification signal survives, the spatial signal does not, and the box can only be refined to within 32 pixels. A stride-4 map has 8×8 cells for the same object but far weaker semantics. The FPN refuses to choose. It keeps every level and gives them all the same 256 channels:

Each level is built from its ResNet stage with a 1×1 lateral convolution that squeezes the channels to 256, added to the top-down pathway from the coarser level above, then smoothed with a 3×3 convolution. The 1×1 laterals cost 0.98M parameters and the 3×3 outputs 2.36M, so the entire pyramid is 3.3M against the backbone’s 23.5M. P6 is not a stage at all: it is a max-pool of P5, giving the RPN a stride-64 level for huge objects at zero parameters.

With five maps, a proposal has to be routed to one of them. The paper’s rule is a logarithm of size:

In words: a 224×224 object is the reference size and lives on P4; every doubling of object size moves one level coarser, every halving one level finer. torchvision implements this same routing rule inside MultiScaleRoIAlign: despite the name, each RoI is still assigned to one level — a LevelMapper applies the equation above (with the level index clamped to the levels the heads read) — and the RoI is pooled only on that level’s feature map. There is no all-levels sampling or channel-wise max merge. Either implementation gives the head a feature map whose receptive field fits the object.

The RPN then reduces 160k anchors to a thousand boxes in three moves: sort by objectness and keep the top 1,000 at inference (2,000 in training), run NMS at IoU 0.7 to delete duplicates, keep the survivors. Training samples 512 of those per image — 128 positive (IoU ≥ 0.5 with some ground-truth box) and 384 negative — because classifying every proposal with the 15.2M-parameter box head each step would be wasteful and badly imbalanced. That funnel is the quiet hero of the design: it is what makes stage two affordable.

A proposal is four floats: the RPN predicts a box like (34.7, 51.3, 98.2, 142.9) in input pixels. A feature map is a grid of integer cells, and the mapping from pixels to cells is a division by the level’s stride: at stride 16 the same box is (2.169, 3.206, 6.138, 8.931) in feature coordinates, with a half-pixel convention (subtract 0.5) because a feature cell represents the pixel region around its centre. Now crop that region into a 7×7 grid. The crop boundaries do not land on cell boundaries, and they never will.

The pre-2017 answer was to round. RoIPool rounded the box corners to integers, then rounded each bin boundary, then took the max feature value inside each integer cell. Rounding the playground’s default box — 4.8 cells on a side — moves its left edge 0.4 cells to the right and its right edge 0.4 cells to the left: 12.8 input pixels of shrink per side at stride 32, and the bin boundaries round again inside the box, so the error compounds in two places. On a classification feature map that is noise. On a mask, where the head is asked to output a silhouette aligned with the proposal, it is the difference between a mask that fits the object and one that is uniformly offset — and the original paper measured it: RoIAlign was worth several points of mask AP on COCO for the same network.

RoIAlign keeps every coordinate exact. Subdivide the float box into bins, place samples inside each bin, read the feature value at each exact position with bilinear interpolation — using the four neighbouring cells and the fractional distances to them — and average. torchvision’s detector uses 2×2 samples per bin (the constructor argument is sampling_ratio=2): 7×7 bins × 4 samples = 196 samples for the box head, 14×14 × 4 = 784 for the mask head, per proposal, all at exact coordinates and all differentiable.

The playground makes the difference concrete on an 8×8 feature map. The default proposal (1.6, 1.1, 6.4, 5.9) is split into a 2×2 grid of 2.4 × 2.4-cell bins; each bin holds four samples at 1/4 and 3/4 through it. The top-left bin’s samples read 1.380, 1.722, 1.800 and 2.264, so RoIAlign’s answer for that bin is their average, 1.792. RoIPool rounds the box to (2, 1, 6, 6) — moving the left edge 0.4 cells right and the right edge 0.4 cells left, which at stride 32 is 12.8 pixels of shrink on each side — and then takes the maximum over each integer cell, which for the same bin is 2.278. The two crops differ by up to 0.908 feature units in that configuration, and even when you hold the reduction fixed at “average” the rounding alone moves a bin by up to 0.416.

One more detail the playground reveals when you press integer corners: a box that already starts and ends on the grid still gets cropped wrong, because the bin boundaries round as well. With the box (2, 1, 6, 6) and a 2×2 output, the vertical split wants boundaries at 1, 3.5 and 6; RoIPool rounds 3.5 to 4, so the crop is 1 / 4 / 6 — bins of height 3 and 2 instead of 2.5 and 2.5 — and the same-reduction difference is still 0.348 feature units. Rounding twice is worse than rounding once, and there is no way to fix it from outside: the sampled crop is simply not the region the detector asked about.

The mask head is a fully convolutional network, which means it contains no fully connected layers and no global pooling — it preserves the spatial layout of whatever it is given. Four 3×3 convolutions with 256 channels operate on the 14×14 crop, a 2× transposed convolution doubles the resolution to 28×28, and a final 1×1 convolution projects 256 channels down to num_classes: 91 channels at COCO scale. One channel per class, one mask each, for the price of about 2.6M parameters (4 convolutions, a 2× deconvolution and a 1×1 convolution).

At training time, the mask loss is per-pixel binary cross-entropy applied only to the channel of the class the box head selected — the mask head never has to learn to suppress masks for classes the proposal is not. At inference time, the whole 91-channel tensor is produced and all but one channel is thrown away. That looks wasteful (91 × 28 × 28 = 71,344 numbers per RoI to keep 784) and it is the design’s best idea: because the mask head never sees the class decision, it cannot trade mask quality against classification confidence. Mask shape and class probability are separate targets with separate gradients, and a wrong class does not corrupt a right mask — which matters when you later review the masks by hand or reclassify them downstream.

That last detail is worth leaning on, because it is where most first attempts lose an hour: masks is not a 28×28 tensor. The model resizes each 28×28 probability map to the box’s size, pastes it into an all-zero (H, W) canvas, and stacks the N canvases. So N masks of the full image size come back, N is usually under 100, and at 800×800 in float32 that is 2.56 MB per mask before compression. The 28×28 head output is an internal detail — it exists because 784 logits per class is a tractable thing to train against a 28×28 target, not because that is the resolution you ship.

The load is as ordinary as any torchvision model: the weights enum handles the download, weights.transforms() gives you the exact preprocessing the weights were trained with, and weights.meta[“categories”] gives you the label names. What is not ordinary is the forward pass: a detection model expects a list of tensors (one per image, variable sizes are fine), returns a list of dicts, and — the part that trips people — behaves differently depending on model.training. In evaluation mode you get predictions; in training mode you get losses. There is no third mode and no separate API.

Read the table the way you would read a price list. The mask branch itself is the difference between the two ResNet-50-FPN v2 rows: 46.4M against 43.7M parameters (2.7M, about 6%) and 333.58 against 280.37 GFLOPs (+53.2, +19% compute) — and what it buys is the mask column, 41.8 mAP. That is the whole economics of instance segmentation: masks are not expensive to add to a detector; they are expensive to annotate.

The gap between box mAP and mask mAP is a diagnosis, not a defect. On the v1 weights it is 3.3 points (37.9 → 34.6); on v2 it is 5.6 (47.4 → 41.8). A mask must be tight enough to clear the same IoU thresholds as a box, pixel by pixel, on a boundary it has only 28×28 logits to describe; some slippage is unavoidable. Watch the gap and it tells you which head to work on: a shrinking box-mask gap with flat box mAP means the mask head has hit its data limit; a growing gap on a new dataset usually means the masks are noisier than the boxes (mask annotation is 5–10× more expensive, and cheap mask annotation is visible).

Choosing between the four rows is a compute question more than an accuracy question. v2 costs 2.5× the FLOPs of v1 for +9.5 box and +7.2 mask points; MobileNetV3-Large-FPN costs about a seventy-fourth of v2’s FLOPs (4.49 against 333.58) for −14.6 box points and no mask head at all, which is the right trade when you cannot fit the mask branch at all. And note the units: 333.58 GFLOPs is a forward pass per image; a training step is roughly 3× that, which is why the memory arithmetic in the next chapter matters more than the accuracy table.

The recipe reuses everything: COCO’s backbone, FPN and RPN weights stay; only the two predictors are rebuilt for your class count, and only the parts that must learn are unfrozen. On a 500-image dataset, freezing model.backbone.parameters() freezes 26.9M parameters — the 23.5M ResNet-50 trunk and the 3.3M FPN, because torchvision packs the FPN inside model.backbone — and leaves 19.0M trainable (41.5%): the RPN plus the box and mask heads. The box head is the largest learner at 15.2M, which surprises people who expect the mask branch to dominate; it is four 3×3 convolutions with batch norm on 12,544 features per RoI, a 1,024-wide fully connected layer, and then the predictor’s second 1,024-wide layer with the class and delta heads.

Then there is the memory bill, and this is where the mask branch stops being small. Training samples 512 RoIs per image, and every one of them passes through the mask head at 14×14 and then 28×28, 256 channels deep. The biggest single tensor in the model is the 28×28 deconvolution output: 512 × 256 × 28 × 28 × 4 bytes = 392 MB for one image. Add the mask logits (512 × 91 × 28 × 28 = 139 MB), the 14×14 crops and the deconvolution, and the mask branch accounts for 727.3 MB of the 927.8 MB activation floor — 78% of the memory for 2.6M parameters, about 6% of the model. The trunk’s six stage outputs together are 122.1 MB.

So the freeze question and the memory question are the same question. Freezing the trunk removes its gradients and Adam moments (≈310 MB of fixed memory) but not its activations; dropping the batch from 2 to 1 removes a whole copy of the 927.8 MB floor. Between those two levers, an 8 GB GPU can train this model on a few hundred images, and a 16 GB GPU can do it at batch 2 with the mask branch you actually want.

Two failure modes are worth naming before you start. The first is the plateau: train loss keeps falling, validation mAP stops improving. That is what overfitting on 19M trainable parameters looks like, and the first move is not more epochs — it is freezing more (add the RPN) or training fewer. The second is subtler: freeze everything and you buy stability at the cost of a ceiling. COCO features are excellent for dogs, cars and people; on microscope fields or semiconductor wafers the frozen trunk may simply not contain the features your masks need, and the run converges beautifully to a mediocre number. The triage board below makes the first decision; this paragraph is the second.