A classifier answers with one word. A detector draws a few boxes. Segmentation answers for all 65,536 pixels of a 256×256 image — and U-Net gets both the scene-level context and the pixel-level detail by pairing a downsampling encoder with an upsampling decoder and wiring skip connections between them.
Semantic segmentation classifies every pixel into C classes: 256 × 256 = 65,536 predictions from a single forward pass. Two cars touching each other collapse into one car blob — that is a feature of the task, not a bug. Instance segmentation keeps them apart with ids; panoptic does both.
semantic: class only · instance: class + id · panoptic: bothThe encoder max-pools four times: resolution 256 → 16, channels 32 → 512, and each cell learns about a bigger patch. The decoder upsamples back to 256 and halves the channels. Skip connections concatenate each encoder map into its matching decoder block, so the sharp edges compressed on the way down come back on the way up.
256 → 128 → 64 → 32 → 16 → 32 → 64 → 128 → 256Predict background everywhere on a frame that is 95% background and pixel accuracy reports 95% while the foreground Dice is 0.000. Cross-entropy lets those background pixels cast 95% of the votes; Dice normalizes by mask area, so the votes arrive 1:1. Train with CE + Dice and report IoU and Dice per class.
Dice = 2|A∩B| / (|A|+|B|) · IoU = |A∩B| / |A∪B|A U-Net carries two kinds of information at once — context from a deep, small feature map and detail from the high-resolution maps the encoder built on the way down — and the skip connections are the wires that let the decoder use both.
By the end you will be able to tell semantic, instance and panoptic segmentation apart and pick one for a task; trace a tensor through a U-Net (256 → 16 → 256, channels 32 → 512 → 32, concat = decoder ⊕ skip); write the double-conv, down and up blocks in PyTorch; explain why cross-entropy bottoms out at the majority class (H(0.047) = 0.189 with 95.3% background) and how Dice fixes the votes; read IoU and Dice per class to separate small-object recall from boundary error from imbalance; and choose between transposed convolution and bilinear upsample without shipping a checkerboard.
Classification outputs one label per image. Detection outputs a handful of boxes. Segmentation outputs one label per pixel — 65,536 answers for a single 256×256 image — because a box can say roughly where something is, and a mask can say exactly which pixels are tumour, road or water.
Here is the ladder, with its arithmetic. A classifier collapses an image into one label — a torchvision ResNet with 1,000 classes outputs 1,000 numbers and you keep the largest. A detector outputs a handful of boxes; four boxes with coordinates, confidence and class is about 24 numbers. Semantic segmentation outputs a class for every position in a H × W grid:
The structure is why segmentation powers almost every dense-prediction product: medical imaging (tumour masks), autonomous driving (road, lane, obstacle), satellite (building footprints, crop boundaries, flood water), document parsing (layout zones), robotics (which surface can be grasped). None of those can be solved by drawing a box around the object. A box that is 70% correct is a useful detection; a mask that is 70% correct may be unusable, because the missing 30% is the part that matters — the thin membrane, the lane line, the edge of the water.
The architectural problem is simple to state and not simple to solve. The network must see the global context — what kind of scene is this, roughly where is everything — and the local pixel detail — exactly which pixel is road versus pavement. A standard CNN buys context by compressing the spatial grid with pooling and stride, and compression throws the detail away. The rest of this lesson is the design that keeps both.
A semantic segmentation model makes predictions for a 256×256 image with 3 classes. How many pixel predictions is that, per image?
All three tasks produce masks, and they answer different questions. The deciding question is not technical: do the objects need to be counted separately, and does the background need labels at all?
Semantic segmentation says “this pixel is road, that pixel is car.” The output is one class id per position, so two cars parked bumper to bumper collapse into a single car-shaped region. That is not a flaw to fix — it is the definition of the task, and it is exactly what you want for road, sky, water, tumour or any region where individual identity is irrelevant.
Instance segmentation says “this pixel is car #3, that pixel is car #5.” It separates distinct objects of the same class, and it only covers foreground objects — the things you can count (cars, people, apples). Amorphous stuff (sky, road, grass, water) has no instances, so instance models leave it unlabelled. This is the Mask R-CNN family, and it is the next lesson.
Panoptic segmentation unifies both: every pixel gets a class label, and every thing instance also gets a unique id. Stuff is labelled, things are labelled and counted. Modern architectures — Mask2Former, OneFormer — handle all three task types with one model and a different head, which is why “which task do I actually need?” is worth answering before you pick an architecture.
| Task | Deciding question | Output | Typical use |
|---|---|---|---|
| Semantic | Do I need a class for every pixel, even if neighbours merge? | (N, H, W) class ids; (N, C, H, W) logits during training | Tumour masks, road/water cover, layout zones |
| Instance | Do I need to count or separate individual objects? | (H, W, N_instances) binary masks + class + score, things only | Cell counting, produce sorting, people tracking |
| Panoptic | Do I need both, with nothing left unlabelled? | Per-pixel class + per-pixel instance id | Full scene understanding for driving and robotics |
A photo shows three apples touching in a bowl. You must count each apple and produce its own exact mask. Which task type is this?
U-Net is not one clever layer; it is a contract about who knows what. The encoder trades resolution for context, the bottleneck holds the scene, the decoder trades context back for resolution, and the skip connections make sure the fine detail survives the round trip.
The encoder is a standard convolutional stack with four 2×2 max-pools. Each pool halves the grid, and each following double conv doubles the channels: 256 → 128 → 64 → 32 → 16 pixels on a side, 32 → 64 → 128 → 256 → 512 channels. Nothing here is unusual — this is the same compression a classifier performs. Its effect is that a cell at the bottleneck sees a large patch of the original image, which is what “context” means in practice.
The decoder reverses the process. Each up block bilinearly doubles the spatial grid, concatenates the matching encoder feature map, and runs a double conv that compresses the channels back down. After four up blocks the tensor is 256×256 again, and a final 1×1 convolution turns the 32 remaining channels into one logit per class per pixel.
Why the skips are necessary is worth saying twice. By the time the decoder produces its first 32×32 prediction, every tensor it has seen has been through a 16×16 bottleneck: the information that pixel (137, 84) is exactly on a car door has been averaged into a cell that covers 16 pixels of the original image. Upsampling cannot invent that detail. The skip connections hand the decoder the high-resolution maps the encoder computed on the way down — low in semantics, high in detail — so the concat gives every output stage both kinds of information at once.
Click any block (or use the buttons) to read its tensor shape, operation and role — and watch the matching skip connection light up.
The parameter counts are exact sums of the source’s layer definitions at base 32, three classes. The source rounds the total to “about 7.7M”; the arithmetic gives 7,849,667.
import segmentation_models_pytorch as smp
model = smp.Unet(
encoder_name="resnet34", # any torchvision or timm backbone
encoder_weights="imagenet", # pretrained encoder: the head start that matters
in_channels=3,
classes=3,
)The whole architecture is a double conv, a down block, an up block and a final 1×1 conv. What separates a working U-Net from a broken one is not the layers — it is the shape arithmetic in the concatenation.
Start with the workhorse. DoubleConv is two 3×3 convolutions with batch norm and ReLU, and it appears at every stage of the network. The first conv changes the channel count, the second keeps it. Two details worth knowing: padding=1 keeps the spatial size unchanged, and bias=False because batch norm’s shift parameter already provides a learnable bias per channel.
import torch
import torch.nn as nn
import torch.nn.functional as F
class DoubleConv(nn.Module):
def __init__(self, in_c, out_c):
super().__init__()
self.net = nn.Sequential(
nn.Conv2d(in_c, out_c, kernel_size=3, padding=1, bias=False),
nn.BatchNorm2d(out_c),
nn.ReLU(inplace=True),
nn.Conv2d(out_c, out_c, kernel_size=3, padding=1, bias=False),
nn.BatchNorm2d(out_c),
nn.ReLU(inplace=True),
)
def forward(self, x):
return self.net(x)
# parameters = 9*in_c*out_c + 9*out_c*out_c + 4*out_c
# conv1 weights conv2 weights 2 BN layers (w + b each)
# DoubleConv(3, 32) = 864 + 9216 + 128 = 10,208Down is a 2×2 max-pool followed by a DoubleConv: resolution halves, channels double. Up is the interesting one — it upsamples, concatenates the matching encoder map, and then runs a DoubleConv. The shape check before the concat is not decoration: it repairs inputs whose size is not divisible by 16.
class Down(nn.Module):
def __init__(self, in_c, out_c):
super().__init__()
self.net = nn.Sequential(
nn.MaxPool2d(2), # 256 -> 128 -> 64 -> 32 -> 16
DoubleConv(in_c, out_c), # channels double: 32 -> 64 -> 128 -> 256 -> 512
)
def forward(self, x):
return self.net(x)
class Up(nn.Module):
def __init__(self, in_c, out_c):
super().__init__()
self.up = nn.Upsample(scale_factor=2, mode="bilinear", align_corners=False)
self.conv = DoubleConv(in_c, out_c)
def forward(self, x, skip):
x = self.up(x) # 16 -> 32, channels unchanged
if x.shape[-2:] != skip.shape[-2:]:
x = F.interpolate(x, size=skip.shape[-2:], mode="bilinear",
align_corners=False)
x = torch.cat([skip, x], dim=1) # channels: skip C + decoder 2C
return self.conv(x) # DoubleConv compresses back to Cclass UNet(nn.Module):
def __init__(self, in_channels=3, num_classes=3, base=32):
super().__init__()
self.inc = DoubleConv(in_channels, base) # 3 -> 32
self.d1 = Down(base, base * 2) # 32 -> 64
self.d2 = Down(base * 2, base * 4) # 64 -> 128
self.d3 = Down(base * 4, base * 8) # 128 -> 256
self.d4 = Down(base * 8, base * 16) # 256 -> 512
self.u1 = Up(base * 16 + base * 8, base * 8) # 768 -> 256
self.u2 = Up(base * 8 + base * 4, base * 4) # 384 -> 128
self.u3 = Up(base * 4 + base * 2, base * 2) # 192 -> 64
self.u4 = Up(base * 2 + base, base) # 96 -> 32
self.outc = nn.Conv2d(base, num_classes, kernel_size=1)
def forward(self, x):
x1 = self.inc(x) # 32 @ 256
x2 = self.d1(x1) # 64 @ 128
x3 = self.d2(x2) # 128 @ 64
x4 = self.d3(x3) # 256 @ 32 skip -> u1
x5 = self.d4(x4) # 512 @ 16 bottleneck
x = self.u1(x5, x4) # up 512 + skip 256 = 768 -> 256 @ 32
x = self.u2(x, x3) # up 256 + skip 128 = 384 -> 128 @ 64
x = self.u3(x, x2) # up 128 + skip 64 = 192 -> 64 @ 128
x = self.u4(x, x1) # up 64 + skip 32 = 96 -> 32 @ 256
return self.outc(x) # (N, 3, 256, 256)
net = UNet(in_channels=3, num_classes=3, base=32)
x = torch.randn(1, 3, 256, 256)
print(net(x).shape) # torch.Size([1, 3, 256, 256])
print(sum(p.numel() for p in net.parameters())) # 7,849,667Where does 7,849,667 come from? Every parameter in the model is in a convolution or a batch norm, so the stage table is the parameter budget. The first double conv is 10,208 parameters; the bottleneck double conv is 3.5 million, because it is a 256 → 512 convolution followed by 512 → 512; and the first up block is another 2.4 million, because it ingests 768 channels.
Two resolution conventions are worth keeping straight, because both appear in real code. The original 2015 U-Net used valid convolutions — no padding — so each 3×3 conv eats one pixel per side and each double conv eats two. The paper’s architecture diagram is drawn for a 572×572 input tile and a 388×388 output tile: 92 pixels of border are consumed per side (572 − 2 × 92 = 388). To segment a large image you cut it into overlapping tiles, feed them, stitch the 388×388 outputs so the seams land inside the consumed border, and mirror the input at the real image edges to manufacture the missing context. The modern padded U-Net, including the code above, keeps 256×256 → 256×256 exactly, and only needs the input divisible by 16 (four halvings).
That resolution is a memory decision as much as a quality decision. A single float32 activation of 32 channels at 256×256 is 65,536 × 32 × 4 bytes = 8.4 MB; the same shape at 1,024 channels and 1024×1024 is 1024 × 1024 × 1024 × 4 bytes = 4.3 GB — one tensor, before gradients and before the batch dimension. Hence the two standard workarounds: train on 256×256 tiles (the source’s recommendation for a first model on 8 GB of VRAM), or replace the aggressive bottleneck with dilated convolutions (the DeepLab family) that keep resolution higher while widening the receptive field.
Watch resolution fall from 256 to 16 and climb back to 256, channels rise to 512 and return, and each skip connection sit in the queue until its decoder block concatenates it.
Step to the bottleneck (d4) and watch channels peak at 512 — then step through the decoder and watch the concat push the input to the double conv to 768. Turn skips off to see the tables change: fewer parameters, and nothing carrying the encoder’s edges back up.
Follow a single batch of shape (1, 3, 256, 256) through the exact code above, and check every number against the tables in the stepper lab.
import numpy as np
from torch.utils.data import Dataset, DataLoader
def synthetic_segmentation(num_samples=200, size=64, seed=0):
"""3 classes: background (0), circles (1), squares (2).
Background and shape colours are random, so the network must learn
shape, not pixel colour — an honest little segmentation task."""
rng = np.random.default_rng(seed)
images = np.zeros((num_samples, size, size, 3), dtype=np.float32)
masks = np.zeros((num_samples, size, size), dtype=np.int64)
yy, xx = np.meshgrid(np.arange(size), np.arange(size), indexing="ij")
for i in range(num_samples):
images[i] = rng.uniform(0, 1, (3,))
for _ in range(rng.integers(1, 4)):
cls = int(rng.integers(1, 3))
cx, cy = rng.integers(10, size - 10, size=2)
r = int(rng.integers(4, 12))
if cls == 1:
mask = (xx - cx) ** 2 + (yy - cy) ** 2 < r ** 2
else:
mask = (np.abs(xx - cx) < r) & (np.abs(yy - cy) < r)
images[i][mask] = rng.uniform(0, 1, (3,))
masks[i][mask] = cls
images[i] = np.clip(images[i] + rng.normal(0, 0.02, images[i].shape), 0, 1)
return images, masks
class SegDataset(Dataset):
def __init__(self, images, masks):
self.images, self.masks = images, masks
def __len__(self):
return len(self.images)
def __getitem__(self, i):
img = torch.from_numpy(self.images[i]).permute(2, 0, 1).float() # (3, H, W)
mask = torch.from_numpy(self.masks[i]).long() # (H, W)
return img, mask
images, masks = synthetic_segmentation(num_samples=60, size=64)
split = int(0.85 * len(images))
train_loader = DataLoader(SegDataset(images[:split], masks[:split]), batch_size=8, shuffle=True)
val_loader = DataLoader(SegDataset(images[split:], masks[split:]), batch_size=8)
model = UNet(in_channels=3, num_classes=3, base=16)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
for epoch in range(8):
model.train()
loss_sum, total = 0.0, 0
for x, y in train_loader:
loss, parts = combined_loss(model(x), y, num_classes=3)
optimizer.zero_grad()
loss.backward()
optimizer.step()
loss_sum += loss.item() * x.size(0)
total += x.size(0)
print(f"epoch {epoch} train_loss {loss_sum / total:.3f}")
# watch mIoU climb past 0.9 on the shape classes — the per-class
# evaluation pass is in the metrics chapterOn a balanced dataset the two losses mostly agree. On a frame that is 95% background they disagree about everything — and the disagreement is the difference between a model that segments and a model that reports good numbers while predicting nothing.
Cross-entropy on a pixel grid is exactly the classification loss, just applied at every spatial position. The model outputs (N, C, H, W) logits and the target is (N, H, W) integer ids; PyTorch’s F.cross_entropy consumes that pair natively, with no reshape:
“Equally” is the problem. Take the lesson’s 16×16 grid: 12 foreground cells out of 256 is 4.7% foreground, 95.3% background. A model that assigns the same probability p to every pixel has cross-entropy
That is the whole failure mode in four lines. Every pixel casts one vote, so the background outvotes the foreground 20.3 to 1; the cheapest way to lower the loss is to side with the majority. The number on the dashboard (95.3% accuracy, CE 0.189) describes a model whose mask is empty.
Dice loss optimizes the overlap directly. For a class with probability map p and binary target y:
Compute it per class and average (macro Dice) so a rare class is not washed out by a common one, and add eps so a class absent from a batch does not divide by zero. In practice you use both losses at once:
def dice_loss(logits, targets, num_classes, eps=1e-6):
probs = F.softmax(logits, dim=1) # (N, C, H, W)
one_hot = F.one_hot(targets, num_classes) # (N, H, W, C)
one_hot = one_hot.permute(0, 3, 1, 2).float() # (N, C, H, W)
dims = (0, 2, 3) # sum over N, H, W
inter = (probs * one_hot).sum(dim=dims) # (C,)
denom = probs.sum(dim=dims) + one_hot.sum(dim=dims) # (C,)
dice = (2 * inter + eps) / (denom + eps) # (C,)
return 1 - dice.mean() # macro Dice
def combined_loss(logits, targets, num_classes, lam=1.0):
ce = F.cross_entropy(logits, targets) # (N, C, H, W) vs (N, H, W)
dc = dice_loss(logits, targets, num_classes)
return ce + lam * dc, {"ce": ce.item(), "dice": dc.item()}A predicted mask and a target mask, counted by hand: 50 pixels in both (TP), 10 predicted but not target (FP), 15 target but not predicted (FN).
The bridge is why the medical community can prefer Dice and the driving community IoU without the two ever disagreeing about which model is better: Dice = 2·IoU/(1+IoU) is strictly increasing in IoU, so the rankings are identical. The disc example is the part that stings: a mask that is visually perfect except for a one-cell shift scores 0.667, because for a small object most of what changed is the boundary.
One slider moves the model’s constant foreground probability p. Watch where cross-entropy’s curve bottoms out — and what mask it is describing when it gets there.
A constant predictor is a simplified teaching model — a real U-Net predicts per pixel. The mechanism it exposes is real: CE lets the 95% background cast 95% of the votes, so the cheapest way to lower it is to side with the majority class. Dice normalizes by mask area, so the votes arrive 1 : 1.
On the 16×16 grid, cross-entropy for the constant model is minimized at p = f = 0.047, where CE = 0.189. What is the model doing at that point?
Every segmentation metric is a variant of one of two ideas: how many pixels did you get right (accuracy), or how well do your masks overlap the truth (IoU and Dice). On skewed data the first idea lies, and the second one only tells the truth if you read it per class.
Pixel accuracy is the percent of pixels classified correctly. It is cheap, and on segmentation data it is almost always the wrong headline: the lesson’s 16×16 grid gives it 95.3% for predicting nothing, and the medical scenario below gives it 97.7% for a model that catches less than a third of the tumour core (recall 0.31). Accuracy is worth logging as a sanity check, never as the decision metric.
IoU per class is the intersection over union of the predicted mask and the true mask for that class alone. Average the per-class values over the classes present and you get mIoU, the community-standard summary. Dice (F1 on pixels) is the medical community’s favourite, and the two are monotonically related: Dice = 2·IoU/(1+IoU), so IoU 0.255 ↔ Dice 0.406 and IoU 0.522 ↔ Dice 0.686. Use whichever your field reads; never compare a Dice from one model with an IoU threshold from another.
Two reporting rules follow directly. First, publish the per-class vector, not just the mean: a mean of 0.78 on ten classes is equally consistent with eight classes at 0.9 and two at 0.3 as with everything at 0.78 — and those two worlds call for different decisions. Second, classes with zero support are not zero: if a class never appears in the batch, its IoU is undefined, not 0. The source returns nan and averages with torch.nanmean at evaluation time; scoring absent classes as 0 quietly deflates mIoU by 1/C per missing class and makes model comparison meaningless.
One more metric earns its keep in high-precision work: boundary F1, an F1 computed only on boundary pixels (usually with a small tolerance band). A mask can have high IoU and still shift every outline by a pixel; for a blood vessel, a lane marking or a semiconductor defect, that shift is the whole error.
@torch.no_grad()
def iou_per_class(logits, targets, num_classes):
preds = logits.argmax(dim=1) # (N, H, W) class ids
ious = torch.full((num_classes,), float("nan"))
for c in range(num_classes):
pred_c = preds == c
true_c = targets == c
inter = (pred_c & true_c).sum().float() # |A ∩ B|
union = (pred_c | true_c).sum().float() # |A ∪ B|
if union > 0:
ious[c] = inter / union # absent classes stay nan
return ious
# the honest evaluation pass: nanmean over batches, absent classes left as nan
model.eval()
rows = []
with torch.no_grad():
for x, y in val_loader:
rows.append(iou_per_class(model(x), y, num_classes=3))
per_class = torch.nanmean(torch.stack(rows), dim=0)
print([f"{v:.3f}" if v == v else "n/a" for v in per_class.tolist()])Paint a prediction on the grid, pick a target, and watch Dice and IoU judge the overlap. Then try “one cell off” — a near-perfect mask is not a perfect score.
This 16×16 grid is small on purpose: 12 foreground cells out of 256 is 4.7%, so the always-background score really is ~95%. Overlap metrics are the fix — accuracy is not.
mIoU is one number; the table is the diagnosis. Switch the loss, switch the scenario, and read which classes moved.
| Class | Pixels | TP | FP | FN | Precision | Recall | IoU | Dice | Diagnosis |
|---|---|---|---|---|---|---|---|---|---|
| background | 63,400 (96.7%) | 63,000 | 1,000 | 400 | 0.98 | 0.99 | 0.978 | 0.989 | healthy: precision and recall are both high |
| tumour core ★ | 800 (1.2%) | 250 | 180 | 550 | 0.58 | 0.31 | 0.255 | 0.407 | rare + under-predicted: small-object recall, add Dice or oversample |
| oedema | 1,120 (1.7%) | 620 | 300 | 500 | 0.67 | 0.55 | 0.437 | 0.608 | rare + under-predicted: small-object recall, add Dice or oversample |
| necrosis | 216 (0.3%) | 130 | 56 | 86 | 0.70 | 0.60 | 0.478 | 0.647 | rare + under-predicted: small-object recall, add Dice or oversample |
▲ / ▼ compares this row with the same scenario trained on cross-entropy alone. Dice loss buys the most where the class is tiny — and the diagnosis column is a rule of thumb, not a verdict: read it beside the counts.
Your 10-class model reports mIoU 0.78 and a reviewer asks for per-class numbers. Which situation is the mean hiding?
The decoder has to double the grid four times. You can learn that upsampling with a transposed convolution, or do it smoothly with bilinear interpolation and let a 3×3 convolution do the thinking. Both are everywhere in the wild; one has a periodic failure mode you can predict with arithmetic.
Transposed convolution (nn.ConvTranspose2d) is a learnable upsampling: it takes each input pixel, writes the kernel’s weights into the output around the corresponding position, and sums the overlaps. It was the original U-Net’s choice (as a “up-conv”) and it can learn a sharper upsampling kernel than any fixed interpolation.
Bilinear upsample + 3×3 conv splits the job: smooth interpolation doubles the grid with no parameters, then a normal convolution mixes the channels and sharpens what matters. It has fewer parameters, no periodic artifacts, and it is what the source’s Up block uses:
The artifact to understand is the checkerboard. Every output position of a transposed convolution receives a contribution from each input pixel whose kernel tap lands on it: position o gets a vote from input i through tap j when o = stride · i + j. Count the votes and the failure mode falls out:
The practical ordering, straight from the source: for a first U-Net, use bilinear upsample + conv. It is fewer parameters, it cannot checkerboard, and it is the modern default. If you want to try transposed convolutions, make the kernel a multiple of the stride (4 with stride 2, or 2 with stride 2 for a blockier result), and check the artifact in the upsampled feature maps, not only in the final mask — by the time it reaches the output it has been blended through several more convolutions and can look like vague texture instead of a grid.
One more decoder detail worth carrying into real code: after the upsample and concat, the double conv is doing two jobs at once — it mixes the skip’s high-resolution channels with the decoder’s context channels, then compresses the result back to the level’s channel count. That is why the concat is not optional bookkeeping: it is the only place where the two views of the image meet before the next prediction stage.
Answer before you look. The skip-connection question and the 95%-background question are the two that separate having read the chapter from being able to defend a segmentation run in review.
0 / 5 answered · 0 correct
01Why does U-Net need skip connections between the encoder and the decoder?
02A segmentation task has 95% background pixels and 5% object pixels. You train with plain cross-entropy and reach 95% pixel accuracy. What happened?
03Which task type separates individual cars of the same class from each other?
04You replace the decoder's bilinear upsample + 3×3 conv with a ConvTranspose2d with kernel_size=3, stride=2 and see checkerboard artifacts. Why?
05You report mIoU = 0.78 on a 10-class segmentation task. Why should you also publish per-class IoU?
Click a card to swap the lazy description for what it actually means.
Three problems with exact numbers — build the BCE + Dice loss for a 5% foreground dataset, swap the up block and explain where the checkerboard comes from, and train a real dataset to within 2 IoU of the smp.Unet reference. Try first; a worked answer is one click away.
Use BCEWithLogitsLoss on the (N, 1, H, W) logits and a soft Dice on the sigmoid probabilities of the single foreground channel: dice = (2Σ(p·y) + eps) / (Σp + Σy + eps), loss = bce + (1 − dice). The 5% foreground is the whole point: with f = 0.05 the constant-predictor BCE curve bottoms out at p = f = 0.05 with value H(0.05) = −0.05 ln 0.05 − 0.95 ln 0.95 = 0.1985, and at that point the model's thresholded mask is empty — BCE has nowhere to push it, because the gradient is exactly zero there. Dice's gradient at the same point is −2f²/(p+f)² = −0.5, so the combined loss keeps pushing confidence up. Run both for the same number of steps, log per-class Dice and IoU every few steps, and expect the combined run to reach foreground Dice > 0.8 in the time BCE alone spends parking near 0.99 pixel accuracy with Dice under 0.1. Verify with a per-class report, not accuracy: on 5% foreground, 'predict background' already scores 95%.
Keep everything else identical (same seed, same split, same epochs) and vary only the up block: nn.ConvTranspose2d(in_c, out_c, kernel_size=k, stride=2, padding=k//2 − 1 if you want the same output size). The arithmetic behind the artifacts: every output position counts how many kernel taps land on it. k = 3, s = 2 gives 2, 1, 2, 1 in the interior (the first output positions are boundary cases) — a period-2 magnitude pattern visible as a grid over the whole mask; k = 4, s = 2 gives 2, 2, 2, 2 and the pattern disappears; k = 2, s = 2 gives 1, 1, 1, 1 — no checkerboard but a hard nearest-neighbour look. So the rule is kernel = stride × n, and the first thing to try is k = 4 with s = 2. Expect mIoU within about a point between bilinear + conv and a well-chosen transposed conv, with the difference concentrated in boundary-heavy classes; if the transposed-conv run is several points worse, look for the checkerboard in the upsampled feature maps before blaming training.
The recipe: freeze the split once and for all (image-level, and by patient/scene when the data has groups), fit normalization statistics on the train split only, use the same augmentation and seed for both runs, and compare smp.Unet(encoder_name='resnet34', encoder_weights='imagenet', classes=C) against your from-scratch model on the same loader. The per-class table is the deliverable, and the lesson's driving scenario shows the expected shape of the answer: moving from cross-entropy only to CE + Dice raised mIoU 0.592 → 0.689, but the gains were not uniform — sign 0.079 → 0.353, pedestrian 0.253 → 0.367, car 0.386 → 0.579, while road, sky and building moved by less than 0.01. Dice buys recall on classes whose pixels are rare and whose loss contribution is otherwise a rounding error; the large classes were already fine. Two cautions when you claim 'within 2 IoU': report the mean over the same class set for both models (exclude absent classes, never score them 0), and check boundary F1 if the dataset has thin structures, because IoU's tolerance hides a one-pixel shift that a boundary metric will not.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 04, Lesson 07) and the Math Foundations Notebook reference build. The five labs (the canvas architecture explorer, the canvas mask playground, the CE-vs-Dice loss comparison, the encoder→bottleneck→decoder stepper, and the per-class metric reader) are original to this page, as are the task-ladder arithmetic (65,536 labels vs ~24 box numbers, output logits the size of the input), the constant-predictor loss arithmetic (CE's minimum exactly at p = f, value H(0.047) = 0.189, the 20.3 : 1 vote split, Dice's gradient −0.5 at the same point), the exact U-Net stage and parameter tables (7,849,667 with skips, 7,066,307 without), the 572→388 valid-convolution arithmetic with 92 pixels eaten per side, the activation-memory check (8.4 MB vs 4.3 GB), the transposed-convolution contribution table and kernel = stride × n rule, the Dice↔IoU bridge with worked counts (TP 50 / FP 10 / FN 15 → 0.800 / 0.667; the 12-cell disc one cell off → 0.667 / 0.500), the three per-class scenarios with exact TP/FP/FN counts, the mIoU 'hide the worst class' dashboard trap, and the left-arm-squints memory hook. Every number shown is computed live by the labs or verified by hand in the prose.