A DDPM that never sees a pixel.
Text in, image out, 48× smaller.

A DDPM learns to predict the noise that was added to a sample. In pixel space the sample is the image: 3 channels of 512×512 values, and the denoiser has to spend capacity on raw pixels — gradients, noise textures, compression artefacts — before it gets to anything that looks like content. The Rombach et al. paper asked a simple question: what if a pretrained autoencoder handled perception, and the diffusion model only handled structure?

So Stable Diffusion trains a small variational autoencoder (VAE) first, and then freezes it. The VAE maps a 3×512×512 image down to a 4×64×64 grid of continuous values — one factor of 8 per side, three channels becoming four — and back. The diffusion model is trained entirely inside that grid. At sampling time, noise in the latent space becomes an image only at the very last step, when the VAE decodes.

Two multipliers stack on top of the 48×. The first is the step count: a DDPM-era pixel sampler wanted ~50 steps, while a second-order latent sampler matches it in ~20–25 (the sampler chapter). The second is attention: transformer blocks pair every token with every other token, so cost grows with the square of the token count. A 512×512 image is 262,144 pixel tokens; a 64×64 latent is 4,096. Square those and the attention pair count falls from 68.7B to 16.8M — 4,096× fewer. That is the honest reason nobody trains high-resolution pixel-space transformers.

The cost moves to the VAE, which runs twice per image (encode once, decode once) instead of 25 times. That is the trade in one sentence: pay a fixed compression bill, and stop paying the pixel bill at every single step.

Follow one generation through the line and the shapes tell the story. The prompt goes through a tokenizer and a text encoder into a 77 × 768 embedding — a sequence, not one vector, because the denoiser is going to attend to individual words. A random 4×64×64 latent — all 16,384 values of it — enters the U-Net together with the current timestep and that embedding. The U-Net predicts the noise; the scheduler uses the prediction to produce the next latent; the loop runs 20–50 times. Finally the VAE decodes the clean latent into a 512×512 image, and the optional safety checker decides whether you get to see it.

The order in that table is not the order of the data flow — the scheduler and the U-Net pass work back and forth, which is why the explorer draws them as a loop. And two of the five hold no trained weights of their own: the scheduler is a pure algorithm, and the safety checker is a separate classifier that runs on the output. When people talk about “the model”, they almost always mean the U-Net plus whatever encoder it was trained with.

The VAE is a variational autoencoder: its encoder does not output one point, it outputs a small Gaussian distribution, and a sample from it becomes the latent. Then a fixed constant rescales the raw latent to roughly unit variance — 0.18215 in SD 1.5/2.x, 0.13025 in SDXL. That constant is hardcoded in every pipeline, and getting it wrong is one of the classic silent bugs of DIY latent diffusion.

Two consequences follow from latent geometry, and both are products you have used.

Img2img works because encoding is near-invertible: encode the input image, add a controlled amount of noise, run the normal denoising loop, decode. The structure survives in the latent and the prompt reshapes the content. The dial is strength — the fraction of the schedule you start at. With 30 steps, strength 0.6 means 18 real denoising steps; strength 1.0 means the latent is replaced by pure noise before the loop begins, so the input image contributes nothing at all.

Inpainting is img2img with a second mask-shaped input: the model predicts the whole latent every step, but after each step the untouched region is pasted back from the encoded original. White pixels in the mask get regenerated; black pixels are preserved exactly. Everything else in the pipeline is unchanged, which is why an inpainting checkpoint is just a U-Net that was additionally trained with masked latents.

The text encoder turns the prompt into a 77 × 768 sequence of embeddings. Inside the U-Net, a cross-attention layer builds its queries from the latent patches and its keys and values from that text sequence: each of the 4,096 latent tokens asks which words matter for it. Because those layers exist at every resolution level, the prompt can influence coarse composition at 8×8 and fine detail at 64×64. Without cross-attention, text could only enter as one pooled sentence vector — and prompts would bias the image the way a colour tint biases a photo.

Conditioning is trained by dropping it. During training, the prompt is replaced by an empty embedding about 10% of the time, and the same weights learn to predict noise both with and without the condition. At inference you get two predictions from one model and combine them:

Why the amplification works: the conditional prediction points from “unconditional” toward “matches the prompt”, and w scales that direction. Too small and the prompt barely moves the sample; the source calls guidance the reason text-to-image works at production quality at all. Too large and the sampler walks past the region the VAE can decode, which is what the burned colours and posterised edges at w = 15 are. The production band is 7–8.5, with 7.5 the number every tutorial quotes.

One implementation detail explains the cost curve: the classic pipeline does not run the U-Net twice. It batches an empty prompt and your prompt into one call of size two, so a single forward pass returns eps_uncond and eps_cond together — and every guidance step still costs two forward passes’ worth of compute. Classifier-free guidance roughly doubles sampling time, which is exactly why the sampler chapter’s step-count arithmetic matters.

The DDPM training objective cares about every timestep, but the generation process is just an ODE (or SDE) to integrate backwards from noise to data. A first-order solver takes small, uniform steps; DDIM does exactly that and wants ~50 of them. A higher-order solver can reuse the previous prediction to take a smarter step, which is how DPM-Solver++ reaches comparable quality in 20–30. Distilled variants train a model or adapter to jump several steps at once, trading some quality and prompt fidelity for 1–4 step generation.

Swapping one is a one-line change, because scheduler state is decoupled from the weights: you can train on DDPM and sample with anything. The source puts it exactly that way — sometimes a swap fixes sample issues with no retraining at all.

The step count is where the wall clock lives. Take 130 ms as a stand-in for one 512² fp16 U-Net pass on a 2022-era consumer GPU, and remember that CFG doubles every step:

Treat that 130 ms as a labelled stand-in, not a current benchmark: the real number depends on your card, precision, resolution and attention implementation (PyTorch SDPA, xformers, FlashAttention). The ratio is the durable part — four-step generation is an order of magnitude cheaper than fifty, and that is what makes interactive and batch workloads possible.

The source runs the whole lesson through diffusers rather than rebuilding the pieces — the VAE, text encoder, U-Net and scheduler are each topics of their own. What you are learning here is fluency with the production API: which pipeline class, which inputs, and which two or three numbers actually change the picture.

Which one for which job: no input structure and no constraint ⟶ text-to-image; an image whose composition you want to keep while the style changes ⟶ img2img at 0.4–0.7; a region to replace ⟶ inpainting; and a structure you care about pixel-for-pixel but do not want to inherit the colours of ⟶ ControlNet. The last one is the professional favourite because it separates structure from appearance: draw a rough pose or a wireframe, and let the prompt own everything else.

LoRA (Hu et al., 2021) came out of language modelling and transferred to Stable Diffusion almost unchanged. For a weight matrix W of shape d_in × d_out, keep W frozen and learn two skinny matrices instead: A of shape d_in × r and B of shape r × d_out. The layer computes W + α·(A@B). A is initialised randomly, B at zero, so the adapter starts as an exact no-op and can only add to what the base already knows.

The training loop is the diffusion objective from Lesson 10 with three fixed pieces: the VAE is frozen and only encodes, the text encoder is frozen and only embeds, and the U-Net is frozen except for the LoRA matrices injected into its attention layers. Only those receive gradient.

At inference the adapter is a plugin, not a new model. load_lora_weights patches the attention layers; lora_scale dials its strength from 0 to 1; fuse_lora bakes it into the weights for speed, after which you must unfuse_lora before loading a different adapter. Several adapters can be loaded at once, each with its own weight — which is how a “style + character” stack is assembled.

One distinction worth keeping straight: LoRA changes how much of the model you train. DreamBooth is a different recipe about what you teach — a rare token bound to one subject, usually from 3–5 images, with class-prior preservation so the class (say, “dog”) does not get overwritten. In practice the two combine: DreamBooth-style data, LoRA-style parameters, which is how a subject adapter trains on a single consumer card.