A diffusion model learns exactly one skill: take a noisy image and report the noise that was added. Train it on a fixed schedule that pours noise in over 1000 steps, then run the same network backwards — and a picture emerges from pure noise. No adversary, one MSE, and a sampler you can swap for a 30× speed-up without retraining.
Add Gaussian noise in 1000 small steps (β from 1e-4 to 0.02) and the picture is gone: ᾱ_1000 = 4.0e-5 means 0.004% of the signal remains. The whole chain collapses into one line — x_t = √ᾱ_t·x_0 + √(1−ᾱ_t)·ε — so training samples any t without simulating a single intermediate step. Nothing here is learned.
ᾱ_0 = 1 · ᾱ_260 = 0.5 · ᾱ_500 = 0.0786 · ᾱ_1000 = 4.0e-5ε̂ = ε_θ(x_t, t), scored with plain MSE against the noise that was drawn. An untrained network scores E[ε²] = 1.0000; a trained one approaches 0. Predicting ε instead of the image keeps the loss O(1) at every timestep, where an x_0-target loss is reweighted by 1/SNR — 11.7× at t = 500 and 24,780× at t = 1000.
loss = ‖ε − ε̂‖² · starts at 1.0, no adversarial gameSampling starts at pure noise and applies the posterior mean step by step; the last step adds no noise, so a perfect model returns the image exactly. DDPM needs one call per timestep — 1000 at 30 ms is 30 s per image. DDIM takes the same trained weights, estimates x̂_0 and jumps: 20–50 calls, deterministic, no retraining.
1000 calls → 50 calls = 20× · 30 calls = 33× · 30 s → 0.9 sThe forward process is a fixed recipe for destroying an image in 1000 steps — and it folds into one line, x_t = √ᾱ_t·x_0 + √(1−ᾱ_t)·ε. The network learns one thing, the noise in x_t; the sampler walks the recipe backwards with that prediction; and because only the sampler changes, the same weights can be run in 1000 calls or in 30.
By the end you will be able to derive the forward closed form and read ᾱ out loud (0.897 at t = 100, half the signal at t = 260, 4.0e-5 at t = 1000); write the six-line training step and say why the loss starts at 1.0; explain where DDPM’s four sampling coefficients come from; read the posterior mean off a noise prediction by hand; say what the sinusoidal embedding buys (64 smooth numbers instead of 1000 one-hot slots, neighbour gap exactly 1.4718); and choose between DDPM and DDIM from a latency budget — 1000 calls is 30 s per image at 30 ms, 30 calls is 0.9 s.
A generator that maps noise to an image in one forward pass is fast and miserable to train. A generator that starts from noise and improves the picture in small steps is slow and easy to train. For five years the second trade has been the one that wins.
A GAN is two networks in a fixed game: one draws, one critiques, and they improve together. When it works, it is one forward pass — noise in, image out. When it fails, it fails in ways that have no error message: the generator collapses onto a handful of images, the critic becomes too strong, or the two oscillate forever. Training a good GAN is a craft learned over failed runs.
A diffusion model sidesteps the game. It learns exactly one skill: take a noisy image and predict the noise that was added. The loss is a plain mean squared error, the gradient is well behaved, and no second network is trying to outsmart the first. The price is paid at generation time: instead of one forward pass, sampling runs the network many times — 1000 calls per image for the original DDPM.
That price buys something the GAN cannot offer. Because generation is a loop, every step is a place where a new constraint can be injected. Text conditioning, inpainting, style transfer, super-resolution, image-to-image editing, control from a depth map or a pose skeleton — all of them are the same trained denoiser being nudged in the loop. Every controllable image model you have used is diffusion-based for this reason, not because the samples look better.
The forward process is fixed — no learning happens here at all. It is a recipe for destroying an image, and its only interesting property is that the whole thousand-step recipe collapses into one line of arithmetic.
Take an image x_0. Add a tiny amount of Gaussian noise to get x_1. Add a tiny amount more to get x_2. Keep going for 1000 steps and x_1000 is indistinguishable from pure noise. Each step shrinks the signal slightly and injects fresh noise:
The schedule β_t is small and linear: 1e-4 at the first step, 0.02 at the last. The 1000 β’s sum to 10.05, but because each step also rescales the signal the accumulated noise variance saturates at 1 − ᾱ_T ≈ 1 — not ten. Because each step adds so little, the reverse step (chapter 04) will be nearly Gaussian, which is what makes it learnable.
The collapse from a thousand steps to one line is the whole reason diffusion is practical. Adding independent Gaussians to Gaussians gives a Gaussian, so the composition of every step from 1 to t is itself a Gaussian: mean √ᾱ_t · x_0, variance 1 − ᾱ_t. During training you pick a random t, sample x_t in one line, and never simulate the chain. Training is O(1) per example instead of O(T).
def linear_beta_schedule(T=1000, beta_start=1e-4, beta_end=2e-2):
return torch.linspace(beta_start, beta_end, T)
def precompute_schedule(betas):
alphas = 1.0 - betas
alphas_cumprod = torch.cumprod(alphas, dim=0) # ᾱ_t
return {
"betas": betas,
"alphas": alphas,
"alphas_cumprod": alphas_cumprod,
"sqrt_alphas_cumprod": torch.sqrt(alphas_cumprod),
"sqrt_one_minus_alphas_cumprod": torch.sqrt(1.0 - alphas_cumprod),
"sqrt_recip_alphas": torch.sqrt(1.0 / alphas),
}
def q_sample(x0, t, noise, schedule):
sqrt_a = schedule["sqrt_alphas_cumprod"][t].view(-1, 1, 1, 1)
sqrt_one_minus_a = schedule["sqrt_one_minus_alphas_cumprod"][t].view(-1, 1, 1, 1)
return sqrt_a * x0 + sqrt_one_minus_a * noise # one closed-form lineScrub t and watch the same image dissolve. The right-hand curves are the two amplitudes that add up to x_t: √ᾱ_t on the signal, √(1−ᾱ_t) on the noise. The probe pixel’s arithmetic is worked beside them, with a new noise draw one button away.
Two things to internalise: ᾱ_t is the signal that is left, so √ᾱ_t multiplies the image; and at t = 1000 the signal amplitude is 0.0064 — the image is gone, but the noise that replaced it is still the number the network is asked to report.
At t = 500 with the linear schedule, ᾱ_500 = 0.0786. Which sentence is the correct reading?
For every batch, sample a timestep per image, destroy the image by exactly the right amount, ask the network what noise is in there, and compare its answer with the truth. There is no second network, no game, and nothing to balance.
The training loop is the shortest one in the phase:
Step 4 is the closed form from chapter 02, so each training example costs one line of arithmetic regardless of t. Step 6 is the only learned quantity in the whole lesson. Note what the network is asked for: the noise, not the clean image and not the next step. Given a good noise prediction, the sampler derives the rest analytically — which is the subject of chapter 04.
Why noise instead of the image? Because the two parameterisations are mathematically equivalent and numerically very different. Predicting x_0 makes the loss (1 − ᾱ_t)/ᾱ_t = 1/SNR times larger than the noise loss at the same error: 0.115× at t = 100, 11.7× at t = 500, and 24,780× at t = 1000. The noisiest timesteps would dominate the gradient, and those are exactly the examples where almost nothing about the image is recoverable. The ε-target keeps the loss O(1) at every t, so one scalar averages cleanly over all 1000 timesteps.
There is a second convenience. Since ε is unit-variance Gaussian, a network that predicts nothing at all scores E[ε²] = 1.0000. Your loss curve starts at ≈ 1.0, not at some unlabelled number — measured on this lesson’s six-pixel batch, 0.9907. Watch it fall toward 0 and you are watching the model learn to denoise.
def train_step(model, x0, schedule, optimizer, device, T=1000):
model.train()
x0 = x0.to(device)
bs = x0.size(0)
t = torch.randint(0, T, (bs,), device=device) # one t per image
noise = torch.randn_like(x0) # the target
x_t = q_sample(x0, t, noise, schedule) # closed form, step 4
pred = model(x_t, t) # ε̂ = ε_θ(x_t, t)
loss = F.mse_loss(pred, noise) # the whole objective
optimizer.zero_grad()
loss.backward()
optimizer.step()
return loss.item()A clean pixel value 0.90 and a noise draw 1.30 are fixed. Pick a timestep, move the model’s prediction ε̂, and watch the same MSE the source minimises: loss = (ε − ε̂)², gradient −2(ε − ε̂). No adversarial game anywhere.
| pixel | x_0 | ε | x_t = √ᾱ·x_0 + √(1−ᾱ)·ε | ε − ε̂ | (ε − ε̂)² |
|---|---|---|---|---|---|
| sun · red | 0.98 | 1.30 | 1.5226 | 1.300 | 1.6900 |
| sun · green | 0.82 | -0.42 | -0.1733 | -0.420 | 0.1764 |
| sky · blue | 0.64 | 0.91 | 1.0529 | 0.910 | 0.8281 |
| sky · red | 0.24 | -1.62 | -1.4878 | -1.620 | 2.6244 |
| ground · green | 0.42 | 0.18 | 0.2905 | 0.180 | 0.0324 |
| ground · blue | 0.19 | -0.77 | -0.6859 | -0.770 | 0.5929 |
| MSE | mean of the last column = 0.9907 | ||||
an untrained network outputs ≈ 0, so the loss is 1.300² = 1.690
The same (ε − ε̂)² is averaged over every pixel and every sampled timestep; that is the entire training objective. Predicting ε keeps the loss O(1) at every t: the equivalent x_0-target loss would be reweighted by (1−ᾱ_t)/ᾱ_t = 11.72× at this timestep, and by 24,780× at t = 1000 — which is exactly why every diffusion model predicts noise.
At initialization the training loss reads ≈ 1.0. Where does that number come from?
The forward process is fixed and known. The reverse process is the one thing we learn — and because the forward chain is Gaussian, its posterior has a closed form that costs one network call per step.
We want p(x{t-1} | x_t): given a noisy image, what was one step less noisy? In general that question has no closed-form answer, which is why diffusion looks hopeless on first contact. But the forward chain is not a general chain — it is a chain of Gaussian additions, and for those the posterior q(x{t-1} | x_t, x_0) is Gaussian with a mean and variance you can write down. The network’s job is to supply the one thing the closed form still needs: a good guess at the noise, which is equivalent to a good guess at x_0.
So the sampling step becomes arithmetic. Compute the noise prediction, form the posterior mean, add a small amount of fresh noise if you are not at the end, and step down. In code that is the four lines below; the coefficients 1/√α_t and β_t/√(1−ᾱ_t) look arbitrary until you expand them (the derivation is in the box).
@torch.no_grad()
def sample_ddpm(model, schedule, shape, T=1000, device="cpu"):
model.eval()
x = torch.randn(shape, device=device) # x_T: pure noise
betas = schedule["betas"].to(device)
sqrt_one_minus_a = schedule["sqrt_one_minus_alphas_cumprod"].to(device)
sqrt_recip_alphas = schedule["sqrt_recip_alphas"].to(device)
for t in reversed(range(T)): # t = 999 … 0
t_batch = torch.full((shape[0],), t, dtype=torch.long, device=device)
eps = model(x, t_batch) # ε̂ = ε_θ(x_t, t)
coef = betas[t] / sqrt_one_minus_a[t]
mean = sqrt_recip_alphas[t] * (x - coef * eps) # posterior mean
if t > 0:
x = mean + torch.sqrt(betas[t]) * torch.randn_like(x) # + √β_t·z
else:
x = mean # last step: no fresh noise
return xTwo details carry real meaning. The z term: at every step except the last, DDPM adds fresh Gaussian noise with variance β_t — 0.0100 at t = 500, about 1% of the noise currently in the sample. That injection is what makes DDPM stochastic: two runs from the same x_T end at different images. And the last step: the final jump from t = 1 to t = 0 adds nothing, so x_0 is simply the posterior mean — which, with a perfect noise prediction, is the original image exactly. The lab measures that: over all 3,072 values of the toy scene, the oracle walk’s mean absolute error at the end is 0.0000. Every visible flaw in a real sample is network error, not sampler error.
Every call asks the model one question: what noise is in this image? The answer gives x̂_0 and a step towards it. The right panel is what a perfect model would hand back at the same timestep — a real U-Net deviates, and that deviation is the whole training problem.
Watch the probe value: at t = 500 the pixel is further from its clean value than at t = 1000, because the noise amplitude √(1−ᾱ_t) is already 0.96 while the image amplitude √ᾱ_t still contributes. Detail only returns in the last stretch.
A denoiser that does not know t cannot know how much to remove. The fix is one small vector of numbers computed from t, added into the network at its bottleneck — the same trick transformers use for token positions.
Consider two noisy images that look identical: one at t = 100, one at t = 900. The first needs a tiny correction — it is 90% picture. The second needs a large one — it is 99.4% noise. A network that is not told which is which has to infer the noise level from the image itself. That can be done, and early diffusion models did it, but the network spends capacity on a job that one number would solve. Worse, the same image at different t leads to different optimal corrections, so the function it must learn is needlessly jagged.
The standard fix is a sinusoidal time embedding: 32 frequencies logarithmically spaced from 1 down to 1/10000, each contributing a sine and a cosine. The result is 64 numbers that change smoothly with t, where nearby timesteps are nearby vectors and any t is a unique point in the space. Then a small MLP maps those 64 numbers to a 32-number vector that is broadcast and added to the feature map at the network’s bottleneck.
The network that receives this vector is a U-Net — the encoder, decoder and skip connections from Phase 4 Lesson 07 — with the time signal injected at the deepest block. The source’s TinyUNet is deliberately tiny, at base = 16: enc1 (3→16) turns a 32×32×3 image into 32×32×16; enc2 (16→32, stride 2) halves it to 16×16×32; the mid convolution keeps that shape; dec1 (32→16, transposed) doubles it back; and dec2 concatenates the h1 skip with the upsampled features to output 32×32×3 — the predicted noise field. Add up the layers and the whole thing is 37,395 parameters, which is small enough to train on a laptop CPU. Stable Diffusion’s U-Net is ~860 million: the same idea, 23,000× larger, and in that one the projected time vector is added inside every residual block rather than only at the bottleneck — the same 64-number embedding, injected in more places.
def timestep_embedding(t, dim=64):
half = dim // 2
freqs = torch.exp(-math.log(10000) * torch.arange(half, device=t.device) / half)
args = t[:, None].float() * freqs[None]
return torch.cat([args.sin(), args.cos()], dim=-1) # [batch, 64]
class TinyUNet(nn.Module):
def __init__(self, img_channels=3, base=16, t_dim=64):
super().__init__()
self.t_mlp = nn.Sequential(
nn.Linear(t_dim, base * 4), # 64 → 64
nn.SiLU(),
nn.Linear(base * 4, base * 4), # 64 → 64
)
self.enc1 = nn.Conv2d(img_channels, base, 3, padding=1) # 448 params
self.enc2 = nn.Conv2d(base, base * 2, 4, stride=2, padding=1) # 8,224
self.mid = nn.Conv2d(base * 2, base * 2, 3, padding=1) # 9,248
self.dec1 = nn.ConvTranspose2d(base * 2, base, 4, stride=2, padding=1) # 8,208
self.dec2 = nn.Conv2d(base * 2, img_channels, 3, padding=1) # 867
self.time_proj = nn.Linear(base * 4, base * 2) # 64 → 32
def forward(self, x, t):
t_emb = self.t_mlp(timestep_embedding(t, 64))
t_proj = self.time_proj(t_emb)[:, :, None, None] # ready to broadcast
h1 = F.silu(self.enc1(x)) # 32×32×16, kept for the skip
h2 = F.silu(self.enc2(h1)) + t_proj # 16×16×32 — time enters HERE
h3 = F.silu(self.mid(h2))
d1 = F.silu(self.dec1(h3))
d2 = torch.cat([d1, h1], dim=1)
return self.dec2(d2) # the predicted noise ε̂32 frequencies, each contributing a sine and a cosine. Low channels oscillate fast and pin down the exact timestep; high channels drift slowly and say roughly where in the schedule you are. The vector is projected and added into the U-Net’s second block — the only place t enters.
At t = 0 every sine channel is 0 and every cosine channel is 1. Two neighbouring timesteps are never closer than 1.47 in this 64-number space, so the network can always tell them apart — the argument a one-hot vector cannot make in 64 dimensions.
Why does the denoiser need t at all — why not just look at the noisy image and remove the noise it can see?
DDPM walks one timestep at a time and rolls dice at every step. DDIM asks the network once, writes down its guess at the clean image, and jumps. The checkpoint does not change — only the loop does.
The key observation is that the DDPM step never really needed x{t-1}. From the noise prediction, the posterior-mean formula already gives x̂_0 — the model’s best guess at the clean image — and rewriting the same Gaussian in terms of x̂_0 turns sampling into a deterministic update:
Nothing is retrained. The DDIM paper’s point is that the reverse process, written this way, is an ODE, and an ODE trajectory can be integrated in 20–50 well-chosen steps while landing at almost the same endpoint as a 1000-step ancestral walk. In production every system uses DDIM or a faster descendant (DPM-Solver, Euler ancestral) with the same trained U-Net.
Two properties beyond speed matter in practice. Determinism: with η = 0 the sampler is a function, so the same x_T always produces the same image — reproducible generations, and the ability to interpolate between two noise seeds and get a smooth morph. And budget: a 30-call sampler means a prompt-to-image request costs 30 U-Net calls, which is the number that decides whether a product can run on a laptop, a phone, or only a cluster.
@torch.no_grad()
def sample_ddim(model, schedule, shape, steps=50, T=1000, device="cpu", eta=0.0):
model.eval()
x = torch.randn(shape, device=device) # same x_T
alphas_cumprod = schedule["alphas_cumprod"].to(device)
ts = torch.linspace(T - 1, 0, steps + 1).long() # 51 ladders for 50 calls
for i in range(steps):
t, t_prev = int(ts[i]), int(ts[i + 1])
t_batch = torch.full((shape[0],), t, dtype=torch.long, device=device)
eps = model(x, t_batch) # ONE network call
a_t, a_prev = alphas_cumprod[t], alphas_cumprod[t_prev]
x0_pred = (x - torch.sqrt(1 - a_t) * eps) / torch.sqrt(a_t) # x̂_0
sigma = eta * torch.sqrt(((1 - a_prev) / (1 - a_t) * (1 - a_t / a_prev)).clamp_min(0))
dir_xt = torch.sqrt((1 - a_prev - sigma ** 2).clamp_min(0)) * eps
noise = sigma * torch.randn_like(x) if eta > 0 else 0
x = torch.sqrt(a_prev) * x0_pred + dir_xt + noise
return xThe grey walk is DDPM: one network call per timestep from 1000 down, with fresh noise injected at every step. The accent walk is DDIM on the same trained model: it predicts x̂_0 and jumps, so thirty calls land on the manifold — and always on the same point for a given x_T.
On this toy both samplers are on the manifold by ~30 calls — a 2D spiral is an easy target. The numbers that transfer to pictures are the call counts, the determinism, and the published result: DDIM at 50 steps matches DDPM at 1000, with no retraining.
You have a DDPM checkpoint from a 1000-step run. You want to sample with DDIM at 40 steps. What has to be retrained or changed?
T = 1000 is not magic, and a linear β is not the only choice. The signal-to-noise ratio over time is the diagnostic that tells you whether a schedule spends its budget where the image is still there — and it is the plot to draw before any long training run.
Each forward step must be small — otherwise the reverse conditional is not close to Gaussian and the network cannot model it. That pushes T up. Sampling cost pushes T down: the last chunk of a linear schedule injects almost nothing. Look at where the signal actually dies with β linear from 1e-4 to 0.02:
The practical consequence is not just wasted compute. When the denoiser is trained on timesteps where ᾱ_t is already 4e-5, the best it can learn is “nearly everything here is noise” — a prediction that costs a network call and changes the sample by almost nothing. Cosine and sigmoid schedules spread the noise injection more evenly across t, which is why modern models use them and why they matter most when you intend to sample with 20–50 steps instead of 1000. The ᾱ_t curve is the diagnostic plot: if it is flat on the floor for a third of the axis, that third is not earning its network calls.
import math
def cosine_alpha_bar(t, T=1000, s=0.008):
f = math.cos((t / T + s) / (1 + s) * math.pi / 2) ** 2
return f / math.cos(s / (1 + s) * math.pi / 2) ** 2 # normalised: ᾱ_0 = 1
def snr_curve(alphas_cumprod):
return alphas_cumprod / (1 - alphas_cumprod) # the plot to draw first
# two numbers worth reading off that plot
# linear: ᾱ < 0.01 by t = 674, < 0.001 by t = 826
# cosine: signal survives into the late timesteps, so a 50-step sampler
# has something to denoise at every rung of its ladderWhere this goes next, in three directions. Down the stack: latent diffusion (Lesson 11) runs the same process inside a pretrained VAE, where a 512×512×3 image (786,432 values) becomes a 64×64×4 latent (16,384 values) — 48× fewer dimensions to destroy and rebuild, which is most of why Stable Diffusion runs at all. Straighter paths: flow matching and rectified flow (Lesson 23) train the model to follow a straight line from noise to data instead of a curved walk, so the sampler can take far fewer steps; Stable Diffusion 3 and FLUX switched to it, and the 2026 stack — SD4, Z-Image, Qwen-Image — trains the same way. And out of pixels entirely: the same denoising loop now generates video frames, audio spectrograms, 3D scenes, robot actions, and the coordinates of protein structures.
Answer before you look. The closed-form question and the DDIM question are the two that separate “I have seen diffusion code” from “I can explain why it works” — and the schedule question is the one that decides whether a run is worth its GPU hours.
0 / 5 answered · 0 correct
01Why can diffusion training pick any timestep t directly, instead of simulating the forward Markov chain step by step?
02What does a DDPM's neural network actually predict?
03Why does DDIM achieve similar sample quality to DDPM with ~20× fewer steps?
04You plot ᾱ_t for a linear β schedule with T = 1000 and see it drops below 0.01 by t = 674. Why does this matter?
05Why does time conditioning enter the U-Net as a sinusoidal embedding rather than a one-hot timestep?
Click a card to swap the lazy description for what it actually means — every definition carries the number that makes it checkable.
Three problems with exact numbers: plot the forward process and confirm that x_1000 is pure noise, train the tiny U-Net and compare DDPM with DDIM from the same noise seed, and implement a cosine schedule to show where the linear schedule wastes its calls. Try first; a worked answer is one click away.
Run q_sample with the same ε for every t so the frames are comparable, then look at three things. (1) The pixel statistics: the expected value of a pixel is √ᾱ_t·x_0, so the image information shrinks as ᾱ_500 = 0.0786 → ᾱ_1000 = 4.0e-5; the variance of the noise around it is 1 − ᾱ_t, which goes 0.000 → 0.92 → 1.00. On this lesson's toy scene, x_1000 has range [−3.51, 3.87] and mean 0.012 — the signature of N(0, 1) — while x_0 has range [0.06, 1.00]. (2) The equality test: because x_t = √ᾱ_t·x_0 + √(1−ᾱ_t)·ε, at t = 1000 the first term is 0.0064·x_0 (at most 0.0064 for an image whose pixels live in 0–1), so x_1000 should match the ε you drew to within 0.0064. Print the max absolute difference; if it is larger, your schedule or your indexing is off by one. (3) The visual test at 750 and 1000: both look like television static, but ᾱ_750 = 0.0034 still leaves a faint low-frequency ghost of the scene, which is why the sampler's late steps can still recover structure. The probe pixel makes it concrete: sun · green has x_0 = 0.863 and ε = −0.276, giving x_500 = 0.2803·0.863 + 0.9599·(−0.276) = −0.023 — the bright pixel is gone, and what replaced it is essentially the noise value.
Use the source's main.py with T = 200 and β from 1e-4 to 0.04 for a CPU-friendly run; 100 images of 16×16, batch 16, Adam 1e-3, three epochs is enough to see circles emerge, and the loss prints each epoch (it starts near 1.0 — E[ε²] — and falls to roughly 0.1–0.3). Fix the seed, draw one x_T, and feed it to both samplers. What you should see: DDIM-50 and DDPM-200 land on similar circles because both are using the same noise prediction function; the difference is that DDIM is deterministic (running it twice gives bit-identical output) while DDPM injects √β_t·z at every step, so the same x_T produces a different circle each run. Count the work: 200 calls versus 50 here, 1000 versus 50 in the paper's setting — a 4× to 20× saving with no retraining. Then measure the residual error honestly: the oracle walk in this lesson's lab reconstructs the toy scene with mean absolute error 0.0000, while a net trained for a few epochs lands around 0.05–0.15. Every visible flaw in the sample is that gap, not the sampler.
Use ᾱ_t = cos²((t/T + s)/(1 + s) · π/2) / cos²(s/(1 + s) · π/2) and derive the betas as β_t = 1 − ᾱ_t/ᾱ_{t-1} (clamped to [0, 0.999] for numerical safety). The two schedules put their noise in very different places: with cosine, ᾱ_500 = 0.494, ᾱ_674 = 0.237, ᾱ_826 = 0.0718 and ᾱ_955 = 0.0049, where the linear schedule has 0.0786, 0.0099, 0.00099 and 0.000099. Translation: at t = 674 the linear schedule has thrown away 99% of the signal while cosine still has 24% — so a 20-to-50-step ladder has real work to do at every rung. Train the identical TinyUNet twice, changing only the schedule, then sample both with 20, 50 and 200 DDIM steps from the same seed and score them the same way (a perceptual or distance metric on your synthetic circles; FID if you are on real data). Expect the cosine run to be clearly ahead at 20 steps and roughly tied at 200 — the paper's result. Two caveats: cosine's ᾱ_T is essentially 0 (3.7e-33 in exact arithmetic), which is the point — it finishes the process — and because the betas are derived rather than linear you must clamp them before taking square roots, or the last few steps produce NaN.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 04, Lesson 10) and the Math Foundations Notebook reference build. The five labs — the forward-noising canvas, the 250-call reverse stepper with its oracle pixel panel, the time-embedding explorer, the DDPM-versus-DDIM race and the pixel-loss board — are original to this page, as are the ᾱ_t ladder with its SNR column, the 1/SNR loss-reweighting numbers (0.115× / 11.7× / 24,780×), the probe-pixel hand-checks, the measured oracle reconstruction (error 0.0000), the 37,395-parameter count of the TinyUNet, the 1.4718 embedding neighbour gap, the sampler latency arithmetic, the schedule diagnostic (ᾱ < 0.01 at t = 674; cosine at 24% there) and the latent-dimension reduction (786,432 → 16,384). The 32×32 sunrise scene and the 2D spiral are labelled teaching stand-ins; the DDIM and DDPM quality-vs-calls curves are measured on that 2D toy, not on images.