A GAN has no labels and no target image — it has an opponent. One network draws from noise, one critiques real against fake, and the critique becomes the loss. At equilibrium the critic can do no better than a coin flip: D = 0.5 everywhere, L_D = 2 ln 2 = 1.3863, L_G = ln 2 = 0.6931.
There is no target image to diff against, so a second network learns the loss for you: D is a classifier trained to tell real from generated, and G is trained to make D wrong. At equilibrium the critic can do no better than a coin flip on every image, the two distributions coincide, and the losses sit on exact constants.
p_G = p_data · D = 0.5 · L_G = ln 2 = 0.6931 · L_D = 2 ln 2 = 1.3863z of 64 numbers reshapes to 1×1×64 and four transposed convolutions double it to 4 → 8 → 16 → 32: 296,896 parameters in the generator and 167,841 in the discriminator, about 1.77 MiB in float32. The generator and the training step are 59 lines together — the rest of the file is data, loading and sampling, and that is where the bugs live.
(in − 1)·stride − 2·pad + kernel · k4 s2 p1 doubles · tanh outputNon-saturating loss repairs the generator's signal: at D(G(z)) = 0.01 the old loss has slope 1.01 and the flipped one has 100. Spectral norm repairs the critic's steepness: without a bound, layer singular values multiply to 2.2 · 2.6 · 3.1 · 3.9 = 69.2 and the sigmoid goes flat. TTUR repairs the tempo: D 4e-4, G 1e-4, ratio 4.
signal 100 vs 1.01 · critic σ ≤ 1 · tempo 4e-4 / 1e-4A GAN is a fixed game between two networks — min_G max_D E_x[log D(x)] + E_z[log(1 − D(G(z)))] — whose only equilibrium is p_G = p_data, where D outputs 0.5 everywhere and the losses read L_D = 1.3863 and L_G = 0.6931. The art is keeping the game alive long enough to get there: signal (non-saturating loss), critic (spectral norm), tempo (TTUR).
By the end you will be able to state the minimax objective and its equilibrium readings from memory; write the 59-line DCGAN and explain each of the five DCGAN rules; apply non-saturating loss, spectral norm and TTUR and name the failure each one repairs; read a GAN curve as healthy, mode collapse, oscillation or D-wins-completely and say what to change; interpolate between two latent vectors and say what a jump would tell you; and pick a family for a task with a number — 3–8 ms for a GAN’s single pass versus 0.6–6 s for a 30-step diffusion sampler.
Classification maps an image to a label and measures the mistake with cross-entropy. Generation has to produce something that never existed — and the breakthrough was to train a second network whose only job is to say “real” or “fake”, then use its judgement as the loss.
Why not just compare the generated image to the nearest training image and minimise the pixel difference? Because the average of two valid answers is usually not a valid answer. If a dataset contains the same person facing left and facing right, the per-pixel-optimal output is a blurry ghost with two half-faces — a perfectly low MSE and a completely useless sample. The loss has to measure does this look like it came from the data? and no fixed formula does that. GANs learn it instead.
The generator G takes a vector of noise z ~ N(0, I) and outputs an image. The discriminator D takes an image and outputs a single number: the probability that it is real. G is trained to make D say “real” about fakes; D is trained to tell them apart. Both get better, and the only way G improves is by producing samples that survive an increasingly sharp critic. This is the minimax game:
That equilibrium is a theorem, not a hope. Goodfellow proved that the game’s value, played optimally, is 2·JSD(p_data ‖ p_G) − 2 ln 2 — a Jensen–Shannon divergence minus a constant — so the only way to reduce it is to make the generated distribution identical to the real one. The hard part is never the algebra; it is that gradient descent has to find that point by playing both sides at once, and the rest of this lesson is about keeping the game from ending prematurely.
Take a fixed generator and ask what the perfect critic looks like. For an image x, one of two things produced it: the dataset, with probability p_data(x), or the generator, with probability p_G(x). The odds that it is real are p_data / (p_data + p_G) — Bayes’ rule, nothing more. So the best discriminator reports exactly those odds.
Now feed that critic back into the game. The max over D collapses to 2·JSD(p_data ‖ p_G) − 2 ln 2. Jensen–Shannon divergence is a distance between distributions: it is 0 when they agree and positive when they do not. So the generator’s best play is to make the distance zero, i.e. p_G = p_data. Numerically: at p_G = p_data the critic outputs 0.5 everywhere, and the losses read 1.3863 (D) and 0.6931 (G). Any deviation hands the critic a signal it can exploit, which is exactly why GAN training is a balancing act rather than a descent.
One generator, one discriminator, two optimisers — and four ways the game can end. Drag the two rates and watch which regime you land in. The equilibrium lines are exact; the trajectories are stylised teaching dynamics.
The ratio is the whole trick: D has to stay a step ahead so its critique still carries information, but not so far ahead that its output saturates and the gradient to G disappears. Both sliders move in units of 1e−4.
Early in training, D(G(z)) = 0.01 for every fake. G is using the original loss log(1 − D(G(z))). What does G experience, and what changes with the non-saturating form?
The source claims a working DCGAN in under 60 lines, and the claim survives an audit: 19 lines for the generator, 22 for the discriminator, 18 for the training step that makes them one game. Everything else in the file is data, loading and sampling.
The architecture is a shape ladder. G starts from z of length 64, reshapes it to a 1×1×64 tensor, and runs four transposed convolutions that each multiply the spatial size by 4, 8, 16, 32. Every spatial step obeys one formula, and checking it by hand is the fastest way to understand transposed convolutions:
The discriminator is the mirror image: four strided convolutions that halve 32 → 16 → 8 → 4 and finally collapse the 4×4 map to a single logit with k4 s1 p0. It ends in no activation — the sigmoid lives inside binary_cross_entropy_with_logits— which is worth remembering when spectral norm enters the picture in the next chapter.
| Stage | Layer | Kernel | Parameters |
|---|---|---|---|
| z → 4×4 | ConvTranspose2d 64 → 128 | k4 s1 p0 | 131,072 weights + BN 256 |
| 4×4 → 8×8 | ConvTranspose2d 128 → 64 | k4 s2 p1 | 131,072 weights + BN 128 |
| 8×8 → 16×16 | ConvTranspose2d 64 → 32 | k4 s2 p1 | 32,768 weights + BN 64 |
| 16×16 → 32×32 | ConvTranspose2d 32 → 3 | k4 s2 p1 | 1,536 weights + tanh |
| 32×32 → 16×16 | Conv2d 3 → 32 · LeakyReLU 0.2 | k4 s2 p1 | 1,568 weights |
| 16×16 → 8×8 | Conv2d 32 → 64 · BN · LeakyReLU | k4 s2 p1 | 32,768 weights + BN 128 |
| 8×8 → 4×4 | Conv2d 64 → 128 · BN · LeakyReLU | k4 s2 p1 | 131,072 weights + BN 256 |
| 4×4 → 1 logit | Conv2d 128 → 1 | k4 s1 p0 | 2,048 weights + bias |
Those layer choices are not taste. Radford, Metz and Chintala distilled years of failed experiments into five rules, and every modern conv GAN still starts from them:
class Generator(nn.Module):
def __init__(self, z_dim=64, img_channels=3, feat=32):
super().__init__()
self.net = nn.Sequential(
nn.ConvTranspose2d(z_dim, feat * 4, 4, 1, 0, bias=False),
nn.BatchNorm2d(feat * 4),
nn.ReLU(inplace=True),
nn.ConvTranspose2d(feat * 4, feat * 2, 4, 2, 1, bias=False),
nn.BatchNorm2d(feat * 2),
nn.ReLU(inplace=True),
nn.ConvTranspose2d(feat * 2, feat, 4, 2, 1, bias=False),
nn.BatchNorm2d(feat),
nn.ReLU(inplace=True),
nn.ConvTranspose2d(feat, img_channels, 4, 2, 1, bias=False),
nn.Tanh(),
)
def forward(self, z):
return self.net(z.view(z.size(0), -1, 1, 1))class Discriminator(nn.Module):
def __init__(self, img_channels=3, feat=32, use_sn=False):
super().__init__()
layers = []
def conv(in_c, out_c, bn):
c = nn.Conv2d(in_c, out_c, 4, 2, 1, bias=not bn)
if use_sn:
c = spectral_norm(c)
layers.append(c)
if bn and not use_sn:
layers.append(nn.BatchNorm2d(out_c))
layers.append(nn.LeakyReLU(0.2, inplace=True))
conv(img_channels, feat, bn=False)
conv(feat, feat * 2, bn=True)
conv(feat * 2, feat * 4, bn=True)
last = nn.Conv2d(feat * 4, 1, 4, 1, 0)
layers.append(spectral_norm(last) if use_sn else last)
self.net = nn.Sequential(*layers)
def forward(self, x):
return self.net(x).view(-1)def train_step(G, D, real, z, opt_g, opt_d, device):
real = real.to(device)
bs = real.size(0)
# D step: does the critic separate real from fake?
opt_d.zero_grad()
d_real = D(real)
d_fake = D(G(z).detach())
loss_d = (F.binary_cross_entropy_with_logits(d_real, torch.ones_like(d_real))
+ F.binary_cross_entropy_with_logits(d_fake, torch.zeros_like(d_fake)))
loss_d.backward()
opt_d.step()
# G step: can the generator fool the critic?
opt_g.zero_grad()
d_fake = D(G(z))
loss_g = F.binary_cross_entropy_with_logits(d_fake, torch.ones_like(d_fake))
loss_g.backward()
opt_g.step()
return loss_d.item(), loss_g.item()The whole file is 128 lines, and the other 69 are the parts every project rewrites anyway: the synthetic-circles dataset, the DataLoader at batch 32, the optimizer construction, the epoch loop, the sampling function and the main() wiring. That is the real lesson of the budget: the model is small, the scaffolding is where the bugs live.
G = Generator(z_dim=64, img_channels=3, feat=32).to(device)
D = Discriminator(img_channels=3, feat=32, use_sn=True).to(device)
# TTUR: the discriminator learns 4× faster than the generator
opt_g = torch.optim.Adam(G.parameters(), lr=1e-4, betas=(0.5, 0.999))
opt_d = torch.optim.Adam(D.parameters(), lr=4e-4, betas=(0.5, 0.999))
for epoch in range(10):
for (batch,) in loader: # 400 circles, batch 32
z = torch.randn(batch.size(0), 64, device=device)
ld, lg = train_step(G, D, batch, z, opt_g, opt_d, device)
print(f"epoch {epoch} D {ld:.3f} G {lg:.3f}")DCGAN replaces pooling with strided convolutions in both nets. What does that buy, and what does the transposed-conv mirror do with it in G?
GAN training fails in three specific ways, and each of the three standard tricks repairs exactly one of them: the generator’s gradient (non-saturating loss), the discriminator’s steepness (spectral norm), and the relative speed of the two players (TTUR). Know which is which and you can debug a run from its curves.
1 · Non-saturating loss — repairs G’s signal. The original objective asks G to minimise log(1 − D(G(z))). Early on, when the critic can tell everything apart, D(G(z)) is near 0 for every fake and the loss is almost flat: at p = 0.01 its slope is 1/(1 − p) = 1.01. The flip to −log D(G(z)) has slope 1/p = 100 at the same point — the same direction, 99× the size — so G keeps getting a usable push exactly when it is worst. The two forms read very differently on a chart: the saturating loss falls toward 0 while nothing improves; the non-saturating loss starts at 0.6931 and rises to −log 0.01 = 4.6052 when the discriminator wins.
2 · Spectral norm — repairs the critic’s steepness. A discriminator with unbounded weights can make its logits arbitrarily large: each convolution multiplies the signal by its largest singular value, and the network-wide slope is the product of those values. For an unconstrained 4-conv D the layer singular values behave like 2.2 · 2.6 · 3.1 · 3.9 = 69.2 in this teaching illustration — so a unit change in the input can move the final logit by 69, landing the sigmoid on its saturated tail where the slope is e^−69.2 ≈ 9e-31 (even at a tamer logit of 20 it is 2.06e−9). G’s gradient passes through that sigmoid, so a strong-enough D deletes it. Spectral norm rescales every layer so its largest singular value is 1; the whole network becomes 1-Lipschitz, a unit input change moves the logit by at most 1, and the sigmoid always has slope (at least 0.197 at logit 1).
3 · TTUR — repairs the tempo. The two-timescale update rule (Heusel et al., 2017) gives D a faster learning rate than G, typically a factor of 2–4: the standard GAN pairing is D 4e-4, G 1e-4, ratio 4. If D lags, G exploits the stale critic (mode collapse); if D sprints far ahead, its output saturates and G’s gradient disappears. Note that this is the same reason DCGAN sets Adam’s beta1 = 0.5: in a game whose landscape shifts every update, both players must react quickly, and the relative rate is a stabiliser, not a detail.
from torch.nn.utils import spectral_norm
def build_sn_discriminator(img_channels=3, feat=32):
return nn.Sequential(
spectral_norm(nn.Conv2d(img_channels, feat, 4, 2, 1)),
nn.LeakyReLU(0.2, inplace=True),
spectral_norm(nn.Conv2d(feat, feat * 2, 4, 2, 1)),
nn.LeakyReLU(0.2, inplace=True),
spectral_norm(nn.Conv2d(feat * 2, feat * 4, 4, 2, 1)),
nn.LeakyReLU(0.2, inplace=True),
spectral_norm(nn.Conv2d(feat * 4, 1, 4, 1, 0)),
)The three tricks are complementary, and each has a cost. Spectral norm adds a power iteration per forward pass; TTUR means tuning two rates instead of one; the non-saturating flip changes what the printed number means. Their order of application is also the order of diagnosis: fix the signal first (it is one line), then bound the critic (the biggest single robustness upgrade), then separate the timescales if the pair still trades blows. If even that fails, the next step is a different game — WGAN-GP replaces the BCE criterion with an Earth-Mover distance and a gradient penalty, which is a rewrite rather than a trick.
Start from the healthy run and switch each trick off one at a time. Each switch owns exactly one failure — and each failure has one signature in the curves and one entry in the readout.
The order to reach for them: flip G’s loss first (it costs one line), then bound D with spectral norm (the single biggest robustness upgrade), then separate the timescales if the curves still trade blows.
A run shows D loss → 0.01, D accuracy 100%, and G loss climbing to 4.6. Which single change is most likely to rescue it, and why?
There is no ground-truth curve to compare against, so the four numbers worth memorising are the equilibrium readings — and then four shapes tell you which of the three failures you are looking at.
Start with the constants. At the game’s equilibrium the critic outputs 0.5 for everything, so the two losses sit on exact values that never change with scale or architecture:
Healthy looks like two noisy lines with no slope: D’s loss drifting from 1.3863 toward roughly 1.1, G’s from 0.6931 toward 0.8, D accuracy hovering just above chance and the sample sheet getting more varied. Mode collapse shows the generator winning the wrong game: one output fools the critic often enough, D’s loss falls to 0, G’s loss stays low and oscillates, and sample variety drains toward 0.1. The give-away is not either loss — it is the sheet of z draws all coming out the same. Oscillation is the two nets trading wins forever: both losses swing by ±0.5 with no downward trend, D accuracy bouncing around 0.5, because neither player can build on the other’s last move. Discriminator-wins-completely is visible in one number: L_G climbing to 4.6052 (that is −log 0.01) and staying there while D’s loss touches 0.01 and D’s accuracy reaches 100%. At that point the sigmoid is flat and G’s gradient is gone, so no further training helps.
| Run | Signature | First fix |
|---|---|---|
| Healthy | L_D drifts 1.4 → ~1.1, L_G 0.69 → ~0.8; both noisy with no trend; D accuracy settles around 0.55–0.65; variety climbs | change one variable per run and keep the shape as the reference |
| Mode collapse | L_D → 0, L_G low and oscillating, D accuracy → 0.95+, variety drains to ≈ 0.1; the sample sheet shows one image | slow G relative to D (4e-4 / 1e-4), add spectral norm or minibatch discrimination, raise the batch |
| Oscillation | both losses swing ±0.5 with no downward trend over tens of epochs; D accuracy bounces around 0.5 | TTUR (D 4e-4, G 1e-4); then WGAN-GP if the swing persists |
| D wins completely | L_D → 0.01, D accuracy → 100%, L_G climbs to −log 0.01 = 4.6052 and stays; samples never improve | spectral norm (σ = 1), lower D's rate, one-sided label smoothing at 0.9, WGAN-GP |
Two curves per run, four diagnoses. The curves are stylised teaching caricatures built from each failure’s mechanism; the dashed equilibrium lines and the numbers in the readout are exact.
Two habits make these curves usable. First: judge the trend, not the level — GAN losses are noisy by construction. Second: when the losses look great and samples do not, look at the variety band, not the numbers.
Losses are one instrument; samples are the other, and on this lesson they outrank the losses. Look at a sheet of 64 samples at the end of every epoch — that is the source’s one non-negotiable habit, because mode collapse is invisible in every scalar. For a number that compares runs, use FID (Fréchet Inception Distance): the distance between the Inception-v3 feature distributions of a real set and a generated set, lower is better. It measures quality and coverage together, which is why it displaced the older, more brittle Inception Score. The full toolkit adds precision/recall for generative models — precision for “are the samples realistic”, recall for “do they cover the data” — but for a small synthetic run, the sample sheet is enough and FID is overkill.
D loss has fallen to 0.02, G loss is low and oscillating, and D's accuracy on fakes is 0.96. You open the sample sheet and every frame is the same image. What happened, and what is the one-line reason the loss curves did not warn you?
Every sample a GAN produces is a function of one vector of noise. The structure of that vector — how it is sampled, what happens when you move inside it, what it means when it stops mattering — is the part of a GAN you keep using after the training run is over.
The latent vector is sampled from a standard normal, one independent number per dimension: z ~ N(0, I). The choice is not arbitrary — a normal is centred on zero, has a fixed spread, and treats every direction the same, so the generator’s learned function of z starts life on a well-conditioned domain instead of one where some coordinates need to be 1,000× larger than others. The paper that made DCGANs work uses 100 dimensions for 64×64 images; this lesson’s 32×32 run uses 64, because fewer pixels need fewer latent numbers. The expectation of the norm is the useful mental number: E‖z‖ ≈ √d — about 10 for the paper’s model, about 8 for ours.
# one draw per image, shape [batch, 64]
z = torch.randn(batch.size(0), 64, device=device)
# after training: walk a straight line between two latents
z1 = torch.randn(1, 64) # e.g. "a face turned left"
z2 = torch.randn(1, 64) # e.g. "the same idea, turned right"
for t in [0.0, 0.25, 0.5, 0.75, 1.0]:
z_t = (1 - t) * z1 + t * z2
frame = G(z_t) # a smooth morph, if training went well
# always sample in eval mode: batch norm must use its running stats,
# not the statistics of your 8-sample batch
G.eval()
with torch.no_grad():
samples = G(torch.randn(16, 64))
G.train()Interpolation works because the generator is a continuous function of z, so a straight line in latent space traces a continuous path in image space. The path is not itself straight — the map is deeply nonlinear, which is exactly what makes it useful: moving one coordinate can change a colour without moving the object, while another direction rotates it. When interpolation produces a jump, a sudden blur, or an image that morphs through an unrelated one, that is real evidence the latent space has holes. When every z produces the same image, the latent code has stopped mattering at all: mode collapse, seen from the input side.
One refinement is worth knowing because it is the knob behind most polished GAN demos. StyleGAN’s truncation trick samples z, then mixes it toward the mean latent: z' = mean + ψ·(z − mean). At ψ = 1 you get the full distribution — varied and occasionally broken; at ψ = 0.5 you get samples closer to the average, which are almost always clean and noticeably more similar. Quality and diversity are traded against each other with one number, and the number is a property of the latent space, not the decoder.
The generator never sees images while it works — it sees z. Slide t to walk a straight line between two latents and watch the output morph; the Δ numbers between frames are the generator’s smoothness test.
Interpolation is how you debug a generator: if the frames jump, the latent space has holes. And if every z gives the same image, the variety meter is the only thing that matters — that is mode collapse, and no loss curve tells you about it.
Every trick that makes diffusion practical — normalization choices, latent spaces, feature losses, adversarial refinements — was first understood on GANs. Knowing which family wins which task is now part of the job; knowing why is how you choose.
The comparison that settles most arguments is the sampling budget. A GAN generator produces an image in one forward pass: 3–8 ms on a modern GPU, which is why it can sit inside a live camera preview or a game renderer. A diffusion sampler produces an image by iterating — 20–50 denoising steps, so 0.6–6 seconds for a 30-step SDXL run on a consumer GPU. Distilled samplers cut that to 4 steps, still 4× a GAN and still a loop. When latency is a product requirement, the architecture question is answered for you.
When quality and language control are the requirement, the answer goes the other way. Text conditioning plus classifier-free guidance is what moved generation from a research demo to a product between 2021 and 2022, and the frontier has been diffusion-shaped since. GANs did not disappear; they specialised. Super-resolution is the cleanest example: MSE-trained upscalers produce blurry averages because that is what MSE asks for, and the adversarial loss is what asks for plausible texture — the ESRGAN family is still the production default on phones and in game remasters. Style transfer and precise image-to-image are the others: Pix2Pix and CycleGAN learn a deterministic translation from an input image, which is exactly what a pipeline wants when the input is a map, a satellite frame or a sketch. And the two families are converging: the fastest diffusion samplers of 2026 are trained with an adversarial loss — the technique is adversarial distillation, where the discriminator scores a 4-step student against a 30-step teacher. The critic from 2014 came back as the teacher.
| Dimension | GANs | Diffusion |
|---|---|---|
| Sampling cost | one forward pass, 3–8 ms | 20–50 denoising passes, 0.6–6 s (SDXL, 30 steps, consumer GPU) |
| Quality ceiling | StyleGAN reached 1024×1024 faces in 2018; excellent on narrow domains | current photorealism and diversity champion; keeps improving with scale |
| Control | deterministic image-to-image (Pix2Pix, CycleGAN), disentangled latent directions, exact style targets | text prompts, classifier-free guidance, inpainting and structural conditioning |
| Best-in-class tasks in 2026 | real-time generation, super-resolution, style transfer, precise image-to-image, few-step distillation | photorealistic text-to-image, editing, video, anything where a second of latency is acceptable |
In 2026 diffusion holds the quality throne — and GANs still run every millisecond-sensitive job. Choose a task and read the verdict with the reason and the number attached.
A GAN generator is one forward pass. A diffusion sampler needs 20–50 denoising passes, so even a distilled 4-step model costs 4× the compute and the latency shows up on every frame.
The honest answer on the source’s evidence: diffusion wins the quality race, GANs win the latency and control races, and the two are converging — the fastest diffusion samplers of 2026 are trained with an adversarial loss. Whether you pick one or distil the other, the lesson’s curve-reading and stability habits transfer unchanged.
The lesson’s “Ship It” deliverables are both small and reusable: a triage prompt that turns a description of a training curve into a failure mode plus one fix, and a scaffolding skill that writes the DCGAN from z_dim, image size and channel count. The prompt is the one that pays off in week one of a real run:
You are a GAN training triage assistant. The user describes a run in
terms of four signals: D loss, G loss, D accuracy on fakes, and sample
variety (or a description of the sample sheet).
Reply with exactly:
1. one verdict: HEALTHY | MODE COLLAPSE | OSCILLATION | D-WINS | SATURATED-LOSS
2. one sentence of evidence, quoting the numbers
3. one fix, and the number that should change if it worked
Rules:
- if D accuracy or sample variety is missing, ask for it; never guess
- a falling G loss is not progress: state which loss form is in use
- reference readings: ln 2 = 0.6931 (G equilibrium), 2 ln 2 = 1.3863
(D equilibrium), -log 0.01 = 4.6052 (G's gradient is gone)Answer before you look. The non-saturating-loss question and the D-loss-near-zero question are the two that separate having read about GANs from being able to debug a run at 2 a.m. — and both are answerable with the exact numbers in this lesson.
0 / 5 answered · 0 correct
01Why does GAN training use the non-saturating generator loss −log(D(G(z))) instead of the original log(1 − D(G(z)))?
02DCGAN's rules say to use strided convolutions instead of pooling. Why?
03You train a GAN and notice G produces almost identical samples regardless of the input noise. Which failure is this?
04Why do DCGAN training scripts use Adam with betas=(0.5, 0.999) instead of the default (0.9, 0.999)?
05A colleague reports a GAN with D loss near zero and G loss increasing over training. What is happening, and how do you fix it?
Click a card to swap the lazy description for what it actually means — every definition carries the number that makes it checkable.
Four problems with exact numbers: train the DCGAN and watch the sample grid, swap batch norm for spectral norm and compare three seeds, build the conditional version so a label controls the output, and write the curve-triage prompt — then test it on a run whose losses look perfect. Try first; a worked answer is one click away.
Run the source's main.py as written (400 images, batch 32 → 13 steps per epoch, z_dim 64, feat 32) but wrap the sampling step in the epoch loop. The dataset is a six-parameter family — centre (cx, cy), radius r, and three colour channels — so G is learning to map six smooth directions of a 64-dimensional noise vector onto a 32×32 image; the bottleneck is the game, not the data. Expect recognisably circular blobs by epoch 5–10 and circle/background separation from the loss alone by around the same point. The numbers to read: at equilibrium L_D = 1.3863 and L_G = 0.6931; a healthy toy run wanders around L_D ≈ 0.9–1.4 and L_G ≈ 0.6–1.0 with no trend. Three signatures that the samples will confirm: a D loss pinned near 0 with a variety drop means mode collapse; L_G climbing to ≈ 4.6 with D accuracy → 100% means D won; both losses swinging ±0.5 with no trend means oscillation. The lesson's five-epoch default run is a smoke test — the grid, not the loss, is the real output, and it should be opened at every epoch.
Keep everything else identical — same data order (torch.manual_seed before each run), same z draws, same optimizer settings — and change only the D constructor. With BN, D's accuracy on the 400-image toy set saturates near 1.0 within the first epoch, because batch norm lets each layer rescale its activations and the logits grow until the sigmoid is flat; L_G then climbs toward −log 0.01 = 4.6052. With spectral norm, every conv's largest singular value is rescaled to ≤ 1, so a unit change in the input can move the final logit by at most 1 and the sigmoid always has slope to give; D's accuracy typically settles around 0.65–0.75 and G's loss keeps falling. Convergence: SN usually reaches a given roundness score in fewer epochs on the toy set, and it is clearly ahead on the harder version. Variance: score each seed by roundness at a fixed epoch (mask area within, say, 10% of the target and the blob centred), then compare the standard deviation across the three seeds — SN's is typically 2–3× smaller, because the failure mode it removes (D saturating) is the main source of seed-to-seed blow-ups. Report both: a mean and a spread, not just the best-looking grid.
G: replace the input with z_cat = [z; one_hot(y)] of length 64 + 2 = 66, so the first transposed conv takes 66 input channels instead of 64; nothing else changes. D: turn the one-hot into a 32×32 constant plane (y broadcast) and concatenate it to the image, so D sees [3 + 2, 32, 32] = 5 channels and can test 'does this image match its claimed label'. Train on the synthetic shapes dataset where circles and squares are labelled; keep the label of each real image for D's extra channel. Sampling: fix y = [1, 0] and draw eight z's — all eight outputs should be circles of varied position, radius and colour; then y = [0, 1] and all eight should be squares. Two proofs worth logging: the loss for mismatched labels (real square shown with y = circle) should be high early and stay high, and the conditional samples' per-class statistics (fill ratio: π/4 ≈ 0.785 for a circle vs 1.0 for a square) should separate the classes. If both classes come out the same, the label is entering G but not D, or it is being concatenated to the wrong axis — print the tensor shapes, and expect the extra channel to change D's first-conv weight count from 3·32·16 + 32 = 1,568 to 5·32·16 + 32 = 2,592.
The prompt's job is to map a description onto one of four rows and name one fix, refusing to guess when the description is missing the discriminator's accuracy or the sample sheet. A working form: 'You are a GAN triage assistant. Given a description of D loss, G loss, D accuracy on fakes, and sample variety, reply with exactly one of HEALTHY, MODE COLLAPSE, OSCILLATION, D-WINS, SATURATED-LOSS, one sentence of evidence quoting the numbers, and one fix. If the description lacks D accuracy or sample variety, ask for it instead of guessing.' Four test cases: (1) 'D 1.3 drifting, G 0.8 drifting, accuracy 0.62, variety rising' → HEALTHY, fix: keep the shape, change one variable per run. (2) 'D loss ≈ 0.05, G loss ≈ 0.9 and oscillating, accuracy 0.95, variety 0.10, one image in the sheet' → MODE COLLAPSE, fix: TTUR plus spectral norm (and look at the sheet). (3) 'both losses swing ±0.5 with no trend, accuracy bouncing 0.5–0.7' → OSCILLATION, fix: TTUR — D 4e-4, G 1e-4. (4) 'D loss 0.01, G loss 4.61, accuracy 1.000, variety 0.12' → D-WINS, fix: spectral norm (σ = 1) and lower D's rate. A fifth case where the numbers look perfect — 'D loss ≈ 0.02, G loss ≈ 0.01, variety 0.12' — must come back SATURATED-LOSS: with a saturating G loss a curve heading to zero is the trap, because 1/(1 − p) ≈ 1.01 means G is not learning, and the fix is to flip the loss to −log D(G(z)).
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 04, Lesson 09) and the Math Foundations Notebook reference build. The five labs (the game board, the curve diagnostics board, the latent explorer, the stability switchboard and the family decision board) are original to this page, as are the non-saturating arithmetic (at D(G(z)) = 0.01 the slopes are 1.01 and 100), the equilibrium readings (L_D = 2 ln 2 = 1.3863, L_G = ln 2 = 0.6931, JS = 0), the itemised 59-line budget (19 + 22 + 18), the shape ladder and parameter counts (296,896 + 167,841 = 464,737 ≈ 1.77 MiB), the spectral-norm saturation numbers (σ ≤ 1 per layer vs the 2.2 · 2.6 · 3.1 · 3.9 = 69.2 product, slopes 0.197 and 9e−31), the TTUR reading (4e-4 / 1e-4 = ratio 4), the latent-space facts (100 vs 64 dimensions, ‖z‖ ≈ √d, the truncation trick), the 2026 latency comparison (3–8 ms per pass vs 0.6–6 s per 30-step run), and the four-exercise set including the curve-triage prompt. The training trajectories in the labs are stylised teaching curves built from each failure's mechanism and are labelled as such; every equilibrium line, parameter count and arithmetic check is exact.