A video model that predicts the next seconds of a scene is a world simulator. Condition that prediction on an action and it becomes a learned game engine — Genie plays, GWM Worlds explores, DreamerV3 imagines in latent space, GAIA drives. The bill is the token arithmetic: 16,384 tokens for a small latent, a quadratic attention pair count, and drift that compounds until the loop re-grounds.
A prompt is a destination. An action is a steering wheel.
Pure video generation conditions on text once, at t = 0 — Sora's report describes spacetime patches and a diffusion transformer, nothing more. An action-conditioned world model takes a control input at every step: Genie infers 8 discrete latent actions from unlabelled video; GWM Worlds 2 takes timestamped text actions plus a camera stream. That is what makes the model playable.
Sora looks · Genie plays · Dreamer dreams · GAIA drives02 / THE CUBE BECOMES TOKENS
Patchify (T, H, W), then pay for the pairs.
A 16 × 32 × 32 latent with 1 × 1 patches is 16,384 tokens — 256× the 1,024 tokens of a single image, because 16 frames multiply the pair count by 16². Joint attention is 268,435,456 pairs; the divided time-then-space pattern is 17,039,360, 15.75× less, and the gap widens to 73× on a 5 s 360p clip.
16,384 tokens · 268M joint pairs · 15.75× divided · 73× at 360p03 / THE LOOP RE-GROUNDS
Small errors compound — until the model sees reality again.
A VLM plans, a video model imagines H steps, inverse dynamics emits the action, the real world answers. Skip the reality check and 2% per-step error becomes 34.6% at DreamerV3's 15-step horizon and 569% over a 96-frame rollout. Re-ground every fourth step and the error stays under 8.2% — no retraining, just a shorter leash.
A video DiT is a diffusion transformer over spacetime patches — patchify the (T, H, W) latent, add a 3D position per patch, and attend time-then-space to keep the pair count affordable. Add an action per step and the same model becomes a simulator: a prompt picks the destination, an action is the steering wheel, and the loop only stays honest because it keeps re-grounding on reality.
By the end you will be able to count video tokens and their attention bill by hand (16,384 tokens → 268,435,456 joint pairs → 17,039,360 divided, 15.75×; 202,500 tokens at 360p → 73×); explain what an action buys over a prompt, whether the action is 8 latent codes or a driving vector; trace the VLM → video model → inverse dynamics loop and say how inverse dynamics gets its labels for free; compute compounding drift (2% → 34.6% at 15 steps, 569% at 96, 8.2% with re-grounding every 4); and pick a family for a task from the dated landscape — Genie 3, GWM Worlds 2, GAIA-2, Cosmos-Drive-Dreams, DreamerV3 — while reading the vendor’s own limitations list as the test plan.
01
TWO KINDS OF VIDEO MODEL
A video generator is a movie. A world model is a vehicle.
A prompt picks a destination. An action is the steering wheel. The difference between the two families of video model is not size or sample quality — it is whether anything enters the generation loop after the first frame.
Give a text-to-video model a prompt and it rolls a scene out frame by frame from that one conditioning signal. The 2024 Sora report made this the mainstream architecture: compress video into a latent, decompose it into spacetime patches, denoise the token sequence with a diffusion transformer, and let scale do the rest — up to a minute of 1080p from a sentence. Nothing enters the process after the prompt. If the model starts turning left and you wanted right, the only fix is to generate again. It is the inverse of Lesson 12’s question: video understanding samples frames and asks what happened; a world model consumes frames and asks what happens next — and the next chapter is where the conditioning interface decides which of those two machines you have built.
An action-conditioned world model adds one wire: a control input at every step. Genie’s 2024 paper inferred a discrete latent action between consecutive frames, capped the codebook at 8 codes, and conditioned the next-frame decoder on it — so a user can play the model like a game, one action at a time. Genie 3 made that real-time (24 fps, 720p, 2025), and Runway’s GWM Worlds 2 (September 2026) drives worlds from timestamped text actions plus a camera stream. That is what conditioning buys: not prettier video — controllability, and with controllability, planning.
The third family predates both in spirit. Ha & Schmidhuber’s 2018 World Models paper trained a tiny controller entirely inside its own “hallucinated dream”, then transferred the policy to the real environment. DreamerV3 (2023) scaled that idea: imagine 15 latent steps, learn from the imagination, act, observe, repeat — 150+ tasks with one configuration, all without decoding a single pixel.
pure video generation action-conditioned world model
prompt ─▶ [video DiT] ─▶ 4–8 s frames ─▶ [world model] ─▶ next frames
▲ ▲ │
└─ text, once, at t = 0 └── action ┘ (every step)
the numbers that make it physical
24 fps → 41.7 ms of compute per frame
4 s clip → 96 frames
1 minute of play → 1,440 frames
Genie 1 (Feb 2024) → 11B params, |A| = 8 latent actions, 1 fps
Genie 3 (Aug 2025) → 24 fps, 720p, consistent for a few minutes
Family
What enters, when
What comes out
Dated examples
Can you steer?
Pure video generation
text or image prompt, set once at t = 0
a fixed clip — 4 to 8 seconds typical, a minute at the top
A colleague says a bigger text-to-video model is basically a world model. Which sentence corrects them?
02
ONE CUBE, CUT INTO TOKENS
A video latent is a cube. A video DiT eats it in patches.
Lesson 23 replaced the U-Net with a transformer over image patches. Lesson 28 extends the same block to a third axis — and the bill for that third axis is quadratic in everything.
A video model does not denoise pixels; it denoises a latent produced by a video VAE, a tensor with shape (C, T, H, W). To feed a transformer, that cube is cut into spacetime patches: a 3D convolution with kernel and stride (P_t, P_h, P_w) turns each (P_t × P_h × P_w) block into one token. The token count is arithmetic, not a mystery:
tokens = (T / P_t) · (H / P_h) · (W / P_w)
small latent T = 16, H = 32, W = 32
1 × 1 patches 16 × 32 × 32 = 16,384 tokens
2 × 2 patches 16 × 16 × 16 = 4,096 tokens
+ P_t = 2 8 × 16 × 16 = 2,048 tokens
production clip 150 frames of 480 × 360, patches (2, 8, 8)
75 × 60 × 45 = 202,500 tokens
a still image at 32 × 32 is 1,024 tokens
16 frames of the same grid is 16,384 tokens
joint pairs grow 16² = 256×, from 1.05M to 268,435,456
Positions have to be 3D for the same reason the patches are: token number 300 in a flattened cube is ambiguous unless the model knows its (t, h, w) coordinate. Video DiTs use rotary or learned embeddings applied separately along the three axes — the same trick as the 2D position codes in Lesson 23, now with a third coordinate.
Step 1 · 3D patchify — the stride is the patchpython
A convolution with stride equal to kernel is a patchifier: each output cell is one patch, already projected to the model dimension.
Then comes the expensive part. If every token attends to every other token — full joint attention — the pair count is N², and N just grew by the temporal axis. At the small latent above, 16,384 tokens is 268,435,456 pairs: half a gigabyte for one fp16 score matrix, per head, per layer. Video models therefore divide the attention into two passes: temporal attention at each spatial position, then spatial attention at each timestep. TimeSformer introduced the pattern in 2021; every 2026 video DiT uses a divided or windowed variant.
divided attention at 16 × 32 × 32 (T_tok = 16, H·W = 1,024)
temporal (H·W) · T² = 1,024 × 256 = 262,144 pairs
spatial T · (H·W)² = 16 × 1,048,576 = 16,777,216 pairs
divided total = 17,039,360 pairs
joint total N² = 16,384² = 268,435,456 pairs
ratio = 15.75×
at 5 s 360p (202,500 tokens, 2,700 cells, 75 temporal tokens)
joint 41,006,250,000 pairs → 82.0 GB fp16 if materialised
divided 561,937,500 pairs → 73× cheaper
Cut the cube into tokens
A video latent is a (T, H, W) block of frames. Patchify divides each axis into (P_t, P_h, P_w) blocks and flattens the result into one sequence — the token count every memory bill starts from. Watch the joint-attention pairs explode, then watch divided attention divide them back down.
The source's teaching latent: 16 frames of 32 × 32, each latent cell a token (the prompt says 16,384 tokens). Two knobs do all the work: bigger patches shrink the sequence, but each token now carries more pixels and the decoder has to put them back. Real models balance those two costs — that balance is a compression ratio, not a detail.
Quick check
Full joint attention over a 16 × 32 × 32 latent costs 268,435,456 pairs. Roughly what does divided attention cost, and why is the saving bigger at 360p?
03
EVERY FRAME GETS A STEERING WHEEL
The action is the interface. Eight buttons can be enough.
Conditioning is a design decision: what signal enters the model, in what format, at what rate. Get it right and the same weights become a game; get it wrong and the model is a very expensive slot machine.
The hard part of an action-conditioned model is not the architecture — it is the labels. Internet video has no action annotations: nobody recorded which key was pressed in a clip of a person walking. Genie solved this by inferring the action. A latent action model takes two consecutive frames, predicts the change between them through a small VQ-VAE, and is trained so the dynamics model can reproduce frame t + 1 from frame t plus the inferred code. The codebook is deliberately tiny — |A| = 8 in the paper — and the authors describe interpreting the codes as “learning the buttons on a new controller”. Eight latent buttons, inferred from video nobody labelled.
Interface
How control enters
Dimensionality
Dated example
Latent actions · Genie
A VQ-VAE infers one discrete code between frame t and frame t + 1 from unlabelled video. The decoder conditions on the inferred code.
|A| = 8 codes — small on purpose, so a human can learn the buttons
Genie paper, 23 Feb 2024 (11B parameters; 6.8M curated clips, 30k hours, at 160 × 90 and 10 fps; ~1 fps at inference)
Explicit controls · driving
Structured values enter the latent diffusion model directly: ego speed and steering curvature, 3D boxes for other agents, weather, time of day, road attributes.
one vector per frame, plus scene metadata — no codebook to learn
GAIA-2, 26 Mar 2025; Cosmos-Drive-Dreams, 2025
Text + camera · interactive worlds
A timestamped event stream addresses actions to a subject or to the scene, alongside a per-frame camera translation and rotation.
free-form text plus a 6-number camera pose per frame
GWM Worlds 2, 3 Sep 2026 (720p at 24 fps, 48 kHz audio, no preset session length)
Latent state + action · RL
The world model predicts the next latent state and a reward from the current state and action; the actor and critic learn inside those rollouts.
15-step imagination per update, 64-step training sequences — no pixels decoded
DreamerV3, 10 Jan 2023: 150+ tasks, one configuration, Minecraft diamonds from scratch
Where the action enters — the same wire as the timesteppython
class ActionConditionedDiTBlock(nn.Module):
"""Lesson 23's DiT block, with one extra conditioning vector."""def __init__(self, dim=64, action_dim=8, heads=2):
super().__init__()
self.adaln = nn.Linear(dim, 6 * dim) # shift, scale, gate × 2
self.attn = DividedAttentionBlock(dim, heads) # time then space
self.action_proj = nn.Linear(action_dim, dim)
def forward(self, tokens, grid, action, cond):
c = cond + self.action_proj(action) # (N, dim)
shift1, scale1, gate1, shift2, scale2, gate2 = self.adaln(c).chunk(6, -1)
h = self.norm1(tokens) * (1 + scale1) + shift1
tokens = tokens + gate1 * self.attn(h, grid) # temporal + spatial
h = self.norm2(tokens) * (1 + scale2) + shift2
return tokens + gate2 * self.mlp(h)
# one action per frame, but the *shape* of the code is the interface:# discrete latent |A| = 8 → nn.Embedding(8, dim)# driving vector → nn.Linear(2, dim) (speed, curvature)# text + camera → condition the text tokens and a pose MLP
A structural demo, not a working generator: the action is projected into the same AdaLN conditioning path the timestep and class label use. Learnable buttons, named values and text all arrive through one wire.
DreamerV3 shows the third path on the same principle, at a different scale. Its world model predicts latent states and rewards — no decoder, no pixels — and the policy learns from 15-step imaginations (imag_length: 15 in the reference implementation). It re-reads the real environment between imaginations, which is why a 15-step horizon is enough for 150+ tasks with a single configuration, including collecting diamonds in Minecraft from scratch. The lesson generalises: the horizon only works because the loop re-grounds.
Same start, four predicted futures
The current frame is on the left; every action code is rolled forward through the same decoder on the right. Pick a code, watch its future, then commit it — the world advances one step and the predictions re-plan from the new frame. This is the whole difference between a video and an environment.
ACTION INTERFACE
8 discrete codes, inferred from video
(a VQ codebook, |A| = 8 in Genie)
CHOSEN a₁ · turn left
EFFECT the view pans right; the scene slides across the frame
WORLD STATE (step 0)
pan 0.00 zoom 1.00
door 0% ball 0.00
GAIA-2 conditions on speed + steering
curvature + agent boxes + weather.
The number of action channels is a design decision, not a natural constant: Genie caps the codebook at 8 so a human can learn the buttons. More control buys precision and costs learnability.
Quick check
Genie's latent actions are discovered from video with no action labels. What is the practical limit of that trick?
04
PLAN, IMAGINE, ACT
A VLM plans. A video model imagines. Inverse dynamics moves the arm.
The emerging robotics stack splits one hard problem into three learnable ones, and the middle component is exactly the world model from the last two chapters. Reward shaping disappears; a scheduler takes its place.
Classical robot learning asks a policy to map observations to actions directly, and pays for it with reward engineering and sample-hungry RL. The 2025-2026 alternative borrows a trick from how people plan: describe the goal in language, imagine what fulfilling it would look like, then work out the muscle commands that would produce the imagined motion. Three components, three datasets, one loop:
1. VLM "pick up the red cup" → high-level subgoals
2. video model subgoals + current frame → H imagined future frames
3. inverse dyn. (frame_t, frame_{t+1}) → the motor action between them
then: execute the action → get a real observation → step 1 again
labels per component
VLM language + goal descriptions (the cheap part)
video model unlabelled video (the internet, plus your own logs)
inverse dynamics (observation, action, next observation) triples —
every teleoperation episode is already exactly this
Genie Envisioner (August 2025) is one public instantiation of the pattern: GE-Base is an instruction-conditioned video diffusion model over robotic interactions, GE-Act maps its latent representations to executable action trajectories with a flow-matching decoder, and GE-Sim is the action-conditioned neural simulator that closes the loop; EWMBench measures visual fidelity, physical consistency and instruction-action alignment. Runway’s GWM Robotics (December 2025) sells the same shape as an SDK: rollouts conditioned on robot actions, plus counterfactual trajectories for evaluating a policy without touching hardware.
The third component is a regression on pairspython
class InverseDynamics(nn.Module):
"""(observation_t, observation_t+1) → the action between them."""def __init__(self, obs_dim=128, action_dim=8):
super().__init__()
self.net = nn.Sequential(
nn.Linear(2 * obs_dim, 256), nn.GELU(),
nn.Linear(256, 256), nn.GELU(),
nn.Linear(256, action_dim),
)
def forward(self, obs_t, obs_next):
return self.net(torch.cat([obs_t, obs_next], dim=-1))
for obs_t, action, obs_next in teleop_loader:
loss = F.mse_loss(model(obs_t, obs_next), action)
loss.backward(); opt.step()
# the label is the action the human took — the episode already recorded it.# 50 episodes × 500 steps = 25,000 pairs, zero extra annotation.
A structural demo: a two-observation MLP is enough for the mechanism. Production decoders are flow-matching heads (GE-Act) or diffusion policies, but the supervision is the same recorded triple.
Why inverse dynamics is nearly free to label
The inverse dynamics model answers one question: given two consecutive observations, which action connected them? That is a supervised problem, and robot datasets are already written in its language. A single teleoperation episode is a list of triples:
one episode = 500 steps at 2 cameras, 256 × 256 RGB
observations 500 × 2 × 256 × 256 × 3 = 196,608,000 numbers
actions 500 × 8 (7 joints + gripper) = 4,000 numbers
the action labels exist because a human was driving
50 episodes × 500 steps = 25,000 training pairs
zero extra annotation — the robot already recorded them
inverse dynamics learns f: (o_t, o_{t+1}) → a_t
and at deployment it decodes the video model's imagined pair
The same argument explains why Genie could train a policy from its own latent actions and transfer with almost no expert data: the mapping between an inferred latent action and a real one carries no information about the current observation, so 200 expert samples were enough in the paper’s imitation experiments. Labels are not scarce when the world already wrote them down.
Plan, imagine, act — then decide whether to trust it
The planner aims at the goal using a model that believes its turns are sharper than they really are. In closed-loop mode it re-grounds every step; in open-loop mode it commits to the whole horizon. Watch the dashed imagined path, the solid true path, and the gap between them: that gap is why the loop exists.
the three-component stack
plan goal → high-level subgoal
imagine subgoal → H imagined states
act imagined states → motor action
DreamerV3 imagines 15 latent steps, then acts.
Genie Envisioner's GE-Sim closes this loop
with a video model + an action decoder.
With the bias at zero the two modes look identical. Raise it and the open loop walks wide while the closed loop keeps correcting — which is exactly the trade DreamerV3 and the video-model robotics stacks make: short imaginations, frequent reality checks.
05
WHERE IT BREAKS
Plausible is not correct. And four seconds is not a simulation.
A world model is judged three ways — does it look right, does it answer the right questions, and can anybody afford to run it. The first is measurable today; the other two are where projects die.
Drift. An autoregressive world model conditions every prediction on its own previous output, so a small per-step error becomes a large multi-step error: after h steps the offset is roughly (1 + ε)^h − 1. At ε = 2% that is 34.6% after 15 steps, 228% after 60 and 569% after a 96-frame rollout. Closed-loop control is not a luxury, it is the fix: re-grounding on a real observation every fourth step caps the error at 8.2%. DeepMind describes Genie 3 as consistent “for a few minutes, with visual memory extending as far back as one minute ago”; Runway lists long-term memory as imperfect in GWM Worlds 2. Neither is an infinite world.
Plausibility versus correctness. Sora 2’s September 2025 launch made physical plausibility the headline: rebounding missed shots, modelled weight and balance, failures on purpose. That is a real capability jump — and it is still a statement about a video looking right. A robot arm whose imagined grasp looks smooth can encode an unreachable motion; a driving scene that looks like an icy intersection may contain vehicles that behave nothing like vehicles. Hand-rated plausibility and FVD measure the look; controllability tests — same state, different action, different observation — measure whether the model is actually simulating anything.
Compute cost. The token arithmetic in chapter 02 is the same wall at every scale. Joint attention over one 5-second 360p clip would need an 82.0 GB fp16 score matrix per head; divided attention cuts the pair count 73×, which is why the architecture exists. At inference the real-time budget is brutal: 24 fps means 41.7 ms per frame, and Runway says outright that real-time generation “still trades fidelity for speed”. Open weights do not make it free — Wan2.1’s T2V-1.3B needs 8.19 GB of VRAM and about 4 minutes on an RTX 4090 for a 5-second 480p clip, and HunyuanVideo runs 13B+ parameters.
drift at ε = 2% per step
h = 4 (1.02)⁴ − 1 = 8.2% ← re-ground every 4 steps
h = 15 (1.02)¹⁵ − 1 = 34.6% ← DreamerV3's imagination horizon
h = 24 (1.02)²⁴ − 1 = 60.8% ← one second at 24 fps
h = 60 (1.02)⁶⁰ − 1 = 228%
h = 96 (1.02)⁹⁶ − 1 = 569% ← a 4-second clip, open loop
what the evaluations actually measure
FVD / user study how good it looks
CLIPScore / VQA does it follow the prompt
hand-rated plausibility physics-looking, not physics-true
action → observation does the same state + new action change the future?
the compute bill
real-time at 24 fps 41.7 ms per frame
joint attention, 5 s 360p 82.0 GB fp16 per score matrix
Wan2.1 T2V-1.3B, 5 s 480p 8.19 GB VRAM · ~4 min on an RTX 4090
HunyuanVideo 13B+ parameters
Small errors, compounded
Every autoregressive step conditions on the model’s own last output, so error multiplies: (1 + ε)^h. Drag the horizon and watch the open-loop line leave the chart. Then re-ground: discarding the imagined state and re-reading the real world resets the product, and that is the entire argument for closed-loop control.
ε = 2.0% H = 96 steps (4.0 s)
error at H
re-ground every step 2.0%
re-ground every 4 8.2%
re-ground every 8 17.2%
never (open loop) 569%
peak between resets
every 4 steps 8.2%
frames in 4 s at 24 fps 96
frames in 1 min at 24 fps 1440
Read the sawteeth: the loop is only as good as the last re-grounding. At ε = 2%, 15 steps carry 34.6% error and a 96-frame open-loop rollout carries 569%; re-grounding every fourth step caps it at 8.2%. The ε here is a teaching stand-in — measure the real one on your model and task.
Quick check
Your world model is 98% accurate per step. A teammate wants a 4-second open-loop rollout at 24 fps for planning. What is the error at the end, and what would you change?
06
PICK A FAMILY, THEN A DATE
Ask what the task needs. Then date-stamp the answer.
Interaction, domain conditioning, access model, horizon. Four questions decide the family; the model names change every few months and should be read as examples, not recommendations.
Start with the task, not the leaderboard. If the deliverable is a fixed clip that a human will edit, pure video generation wins on fidelity per dollar: no control interface, no per-frame latency budget. If a user or an agent has to act inside the rollout, you need an action-conditioned world model, and your first specification is the action space — keys, text actions, camera poses, or structured driving values. If the domain is driving, a domain-conditioned model beats a general one because road topology, multi-camera geometry and agent behaviour are the whole problem. If the consumer is a policy, consider a latent world model and skip pixels entirely.
Then check the boring columns: access (API versus open weights), licence (Wan2.1 is Apache-2.0; HunyuanVideo ships under Tencent’s community licence; research previews are gated), and cost (a 1.3B-parameter open video model still needs 8.19 GB of VRAM and about four minutes on an RTX 4090 for a 5-second 480p clip). API-only access is a dependency with an expiry date: Sora 2 launched on 30 September 2025, the app and web experience were discontinued on 26 April 2026, and the API is scheduled to shut down on 24 September 2026 (OpenAI Help Center).
Model
Dated
Family
What it is
Sora 2
30 Sep 2025
pure video generation
Physically more accurate, synchronised dialogue and sound effects. OpenAI discontinued the Sora app and web experience on 26 Apr 2026 and the API sunsets on 24 Sep 2026 (Help Center); the model card remains the reference for plausibility claims.
Genie 3
5 Aug 2025
interactive world model
Text-to-world at 24 fps, 720p, consistent for a few minutes; promptable world events; limited action space; research preview.
Genie Envisioner
7 Aug 2025
robotics world model
Instruction-conditioned video diffusion (GE-Base), a flow-matching action decoder (GE-Act) and an action-conditioned simulator (GE-Sim) with the EWMBench evaluation suite.
GWM-1
11 Dec 2025
interactive world model
Runway's general world model: GWM Worlds, GWM Avatars, GWM Robotics; up to 2 minutes at 720p, controlled by camera pose, robot commands and audio.
Cosmos-Drive-Dreams
2025
driving world model
NVIDIA's controllable, multi-view driving data pipeline built on the Cosmos world foundation models for perception and policy training.
GWM Worlds 2
3 Sep 2026
interactive world model
720p at 24 fps plus 48 kHz audio, no preset session length; driven by a genesis prompt, a first frame and a timestamped event stream.
Every row above is a dated, official claim, linked in the chooser below and in the sources. None of these numbers are a ranking.
Pick a family, then a date
Four tasks, four different families. The question is never “which model is best” — it is whether you need interaction, what your domain conditions on, and whether you can host it. Every example is dated and linked; treat the table as a snapshot, not a ranking.
TASK · A 30-second brand film
One fixed clip, edited later, no interaction. The prompt can describe the whole shot list up front.
FAMILY · Pure video generation
WHY
Nothing needs to steer mid-rollout, so the simplest and highest-fidelity family wins. Spend the compute on pixels, not on control.
CONDITIONING
text prompt (and optionally a first image), set once at t = 0
HORIZON 4-8 s clips typical; a minute at the top of the range
LATENCY offline batches; minutes per clip is acceptable
ACCESS APIs, or open weights you host yourself
TAGS offline · high fidelity · no interaction
Watch out. An API-only model is a dependency with an expiry date: Sora 2 shipped in Sep 2025, the app and web experience were gone by 26 Apr 2026, and the API sunsets 24 Sep 2026. Keep a self-hosted fallback.
OpenAI's flagship video + audio model: stronger physical plausibility (missed shots rebound), synchronised dialogue and sound effects. The Sora app and web experience were discontinued on 26 Apr 2026; the API sunsets 24 Sep 2026 (OpenAI Help Center).
13B+ parameters, open weights under Tencent's community licence — check the terms before shipping commercially.
Dates matter. Sora 2 shipped on 30 September 2025 and its product was discontinued on 26 April 2026; Genie 3 is still a research preview; open weights mean you keep running when an API does not. Choose the family from the task, then the newest model you can honestly evaluate on your own horizon.
Two deliverables carry this lesson out of the notebook. The first is a model picker: given a task, a licence constraint and a latency budget, name the family, the dated example and the fallback. The second is a physical-plausibility check — a short script that runs on any generated video before it ships: does a dropped object keep falling, does an occluded object return, does a ball bounce off the wall it visibly hit, does the same state plus a different action produce a different future? Automate the checks the vendor already told you would fail.
07
CHECK YOURSELF
Six questions. Then the terms worth keeping.
The attention-arithmetic question and the drift question are the two that separate “I watched the videos” from “I can size a deployment”. Answer before you look.
0 / 6 answered · 0 correct
01What distinguishes an action-conditioned world model (Genie, GWM Worlds) from a pure video generator (Sora)?
02A video transformer block uses divided attention instead of full joint attention. What does that mean for a latent of 16 × 32 × 32?
03Sora 2 (30 Sep 2025) advertised better physical plausibility. Which failure modes did that target?
04In the robotics loop — VLM plans, video model simulates, inverse dynamics acts — what is the inverse dynamics model's job?
05A robot team evaluates its world model on 4-second clips and gets excellent plausibility scores. Why is that not enough to trust a 30-step plan?
06Why do autonomous-driving teams invest in world models such as GAIA-2 and Cosmos-Drive-Dreams?
Key terms, demystified
Click a card to swap the lazy description for what it actually means — every definition carries the number that makes it checkable.
Exercises from the lesson
Four problems with exact numbers: count the tokens and the attention bill for a 5-second 360p clip, swap divided attention for joint attention and work the flops, train a minimal latent-action video model, and compute what compounding does to a 96-frame rollout. Try first; a worked answer is one click away.
Compute the token count and the attention bill for a 5-second 360p clip: 150 frames at 480 × 360, patchified with P_t = 2, P_h = 8, P_w = 8. Compare joint against divided attention, and say how much memory one fp16 score matrix would need.Show one worked answer
Grid: T/P_t = 75 temporal tokens, H/P_h = 60, W/P_w = 45, so 75 × 60 × 45 = 202,500 tokens. Joint attention pairs = 202,500² = 41,006,250,000 (4.10 × 10^10); at 2 bytes per score that is 82.0 GB for one attention matrix — before keys, values or activations. Divided attention: temporal = (H·W) × T² = 2,700 × 5,625 = 15,187,500 pairs; spatial = T × (H·W)² = 75 × 7,290,000 = 546,750,000 pairs; total 561,937,500 pairs (5.62 × 10^8). Ratio: 41,006,250,000 / 561,937,500 = 72.97 — about 73× less work, and the saving grows as the clip gets longer because the joint term is quadratic in all three axes. That is why no production video model runs full joint attention at this size.
Swap the divided-attention block in the lesson's TinyVideoDiT for a full joint-attention block. What happens to the parameter count, and what happens to the runtime? Work the flops for d_model = 512 and a 5 s 360p clip.Show one worked answer
Parameter count is nearly unchanged: attention parameters live in the QKV and output projections, which depend on d_model and heads, not on how the pairs are arranged. What changes is compute. Attention costs about 4·N²·d_model flops (2 for QK^T, 2 for the weighted sum). Joint: 4 × 4.100625 × 10^10 × 512 = 8.40 × 10^13 flops ≈ 84 TFLOP per layer. Divided: 4 × 5.619375 × 10^8 × 512 = 1.15 × 10^12 ≈ 1.15 TFLOP per layer — 73× less. At an assumed 100 TFLOP/s effective throughput that is 0.84 s versus 11.5 ms per layer; a depth of 30 layers turns the difference into 25 s against 0.35 s per denoising step. Memory is the harder wall: the joint score matrix alone is 82 GB in fp16, while a divided block never materialises a matrix larger than 546 million pairs.
Build a minimal latent-action video model: collect (frame_t, action_t, frame_{t+1}) triples from any simple 2D game, train a tiny video DiT conditioned on action embeddings, and show that different actions produce different next frames.Show one worked answer
Start with a game whose action is one of a handful of keys — Pong, Snake or a toy grid world with 4 moves — and record frames at a fixed cadence so consecutive pairs line up in time. Encode the action as an embedding (4 one-hot vectors plus a projection, or an MLP on the analog stick), concatenate it to the timestep embedding and inject it into every block's AdaLN path — the same conditioning route text uses in Lesson 23. Train on pairs with the same ε-prediction loss as Lesson 10. The test that matters is the one Genie's authors run: hold the start frame fixed, vary only the action code, and check that the decoded next frames differ in the direction the action implies. A model that ignores its action input will produce near-identical frames for every code — the failure is visible without a benchmark. Only after that works does the latent-action version make sense: train a VQ-VAE to infer a discrete code from (frame_t, frame_{t+1}) and condition on the inferred code, so the same decoder can be played with a gamepad of learned buttons.
A model's per-step prediction error is 2%. Compute the error after DreamerV3's 15-step imagination horizon, after 60 steps, and after a 96-frame open-loop rollout. Then show what re-grounding every 4 steps does.Show one worked answer
Compounding is multiplicative: error(h) = (1 + ε)^h − 1. At ε = 0.02: h = 15 gives 1.02^15 − 1 = 0.3459, so 34.6%; h = 60 gives 1.02^60 − 1 = 2.281, so 228%; h = 96 gives 1.02^96 − 1 = 5.693, so 569% — the imagined scene no longer describes the real one. Re-grounding every k = 4 steps resets the product before it grows: the error at any step is at most 1.02^4 − 1 = 0.0824, so 8.2%. That is the whole argument for closed-loop control: DreamerV3 imagines 15 steps (imag_length: 15 in the official reference config), then acts, observes, and imagines again from the real state. Keep the 2% figure honest: it is a teaching stand-in; the real per-step error has to be measured for a specific model and task, and the drift lab measures exactly this curve live.
Terms this lesson borrows from later lessons (or outside)
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
Diffusion (Phase 4, Lesson 10) — The denoising objective, the ε-prediction loss and the sampler. A video DiT is the same machine with a bigger token space: it learns to denoise a noisy video latent conditioned on text and actions, and sampling runs the reverse process over the whole cube at once or autoregressively.
Video understanding (Phase 4, Lesson 12) — Sampling frames, temporal structure and what changes between them. The world-model decoder and an action-conditioned predictor read the same signal from opposite ends: understanding asks what happened, generation asks what happens next given a control input.
DiT + rectified flow (Phase 4, Lesson 23) — The transformer denoiser with AdaLN conditioning and the flow-matching sampler. Lesson 28 is Lesson 23 with a 3D patchifier and divided attention — the block diagram barely changes; the token space and the conditioning stack do.
Self-attention (Phase 7, Lesson 02) — Queries, keys and the O(N²) pair cost. Every number in the token arithmetic is just N² in disguise, which is why the whole video-model field is a fight over token count and attention pattern.
Latent diffusion (Phase 4, Lesson 10 / Lesson 23) — Compressing pixels before diffusing them. At 48× reduction, a 4 s 24 fps clip of 512 × 512 frames (96 × 786,432 = 75.5M values) becomes 96 × 16,384 = 1.57M latents — the difference between a memory problem and a scheduling problem.
RL agents (later phases) — Policies, rewards and actor-critic learning. DreamerV3 needs them: its world model predicts latent states and rewards in a compact space, and the actor and critic learn inside 15-step imaginations without ever decoding a pixel.
Vision-language models (Phase 4, Lesson 25) — The planner at the top of the robotics stack. A VLM turns 'pick up the red cup' into a sequence of high-level goals; the video model turns each goal into imagined frames; inverse dynamics turns frames into motor commands.
KEEP GOING
A picture is a start. Practice is the rest.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 04, Lesson 28) and the Math Foundations Notebook reference build. The five labs — the world-model loop canvas, the spacetime patch explorer, the action-conditioning simulator, the drift chart and the family chooser — are original to this page, as are the numbers they compute: the token arithmetic (16,384 tokens; 268,435,456 joint pairs against 17,039,360 divided; 202,500 tokens and 73× at 5 s 360p; 82.0 GB fp16), the frame counts (96 frames in 4 s at 24 fps, 1,440 per minute, the 41.7 ms budget), the drift table (2% → 34.6% at 15 steps, 569% at 96, 8.2% re-grounding every 4), the action dimensions (Genie's 8 VQ codes, GAIA-2's speed and steering conditioning, GWM Worlds 2's 720p / 24 fps / 48 kHz stream) and the inverse-dynamics label count (25,000 teleoperation pairs from 50 episodes). Every 2024–2026 model claim is dated and linked to its official page, including the Sora 2 product discontinuation (26 April 2026). The stylised scenes and the 2D room are labelled teaching stand-ins; drift is modelled as (1 + ε)^h.