A video is a sequence of images plus the physics that connects them. Every video model answers one question — where does time enter? Pool it and you are cheap but order-blind; convolve it and you pay the kernel depth; attend over it and you buy the top of the table. Build the sampler, the baseline, the inflation trick and the factorised conv, then read the leaderboards like a practitioner.
2D+pool runs a CNN per sampled frame and averages the features — T × one image, ImageNet weights transfer, order deleted. 3D convolution puts time inside every kernel — k×k becomes k×k×k, 3× the multiply-adds. Spatio-temporal attention tokenises space-time and lets every token see every other — state of the art, quadratic in tokens.
16-frame clip · 2D+pool 7.8 G · R3D-18 40.7 G (5.2×) · MViT_V2_S 64.2 GMean, max and attention pooling all return the same vector for the same frames in any order: pool(f₁…f_T) = pool(f_T…f₁). Kinetics-400 barely notices — most of its classes are visible in one good frame. Something-Something V2, whose labels are mirrored motion sentences, collapses: space-only attention scores 76.9% on K400 and 36.6% on SSv2; temporal attention takes the same backbone to 59.5%.
5-video × 4-clip board: 13/20 clips = 65% clip vs 4/5 = 80% video accuracyInflation copies each 2D kernel along a new time axis (÷ k_t so a constant clip reproduces the 2D response — the boring-video fixed point), which is what gave 3D networks ImageNet weights: I3D took Kinetics top-1 from C3D's 56.1 to 71.1 RGB / 74.2 two-stream. (2+1)D cuts the cube into a 3×3 spatial and a 3×1 temporal conv with BN + ReLU between; the mid channel keeps the budget equal, and the measured result is 67.5% vs R3D-18's 63.2% at the same cost.
3×3 → 3×3×3: 9 unique → 27 slots · 64→64: 82,944 + 27,648 = 110,592 = full 3DA video model is an image model plus one decision — where the time axis enters — and every consequence follows from it: pool it and you are cheap but order-blind, convolve it and motion costs k_t× per kernel, attend over it and you pay quadratically for the top of the leaderboard.
By the end you will be able to predict which family a task needs and what it costs (16 frames of ResNet-18 at 112² = 16 × 0.485 = 7.76 G, vs 40.7 G for R3D-18); write uniform and dense frame samplers and explain the difference a gap of 0.75 s versus 33 ms makes to a 3-slice temporal kernel; count an inflated kernel (9 unique weights in 27 slots, ÷ k_t); derive the (2+1)D mid channel and reproduce the 110,592 = 82,944 + 27,648 identity; implement FramePool, inflate_2d_to_3d and Conv2Plus1D; and read the leaderboards — Kinetics-400 (400 classes, ≈240k train clips), Something-Something V2 (174 classes, 220,847 clips, motion-defined), UCF-101, HMDB-51 and AVA — while telling clip accuracy from video accuracy.
Thirty seconds at 30 fps is 900 frames. Run an image classifier on each one and average the answers and you have a working video model for a whole class of tasks — and a completely blind one for another. Knowing which is which is the first skill this lesson installs.
Start with the arithmetic. A 30-second clip at 30 fps is 900 frames; a 10-second clip is 300. A 2D classifier that costs one forward pass per frame would cost 900 forward passes — and frames 400–900 usually contain the same objects as frames 1–400. Video models almost never look at every frame; they look at a sampled clip of T frames (8, 16, 32, 64) and spend their capacity on the question that matters: when does time enter the computation?
For a large slice of real video tasks the answer is barely at all. “Playing cello”, “eating cake”, “a dog in a park” are object-and-scene problems: one good frame is enough. That is why Kinetics-400 — the standard 400-class action benchmark — has been dominated by models that lean heavily on appearance, and why a per-frame baseline gets within a few points of much more expensive architectures there.
Then there is Something-Something V2. Its labels are sentences like “Pushing something from left to right” with a mirrored twin class “Pushing something from right to left”. Both clips contain the same hand, the same object, the same table, at the same positions — every single frame is nearly identical. The only difference is the direction of travel. No amount of frame quality helps a model that throws the order away, and here is the exact arithmetic of why:
That is not a training failure you can fix with more data or a better backbone. It is a property of the function the network computes: a symmetric pool is invariant to permutation, so reversing the clip cannot change the prediction. Motion-defined labels need an architecture in which the order of frames reaches the output — a convolution over time, or attention over tokens whose positions encode time.
A 2D+pool model sees a 16-frame clip forwards and the same 16 frames backwards. What changes?
Every video architecture answers one question: where does the time axis enter? The answer determines cost, whether ImageNet weights transfer, and which failure modes you inherit. Three families cover the whole space.
2D + pool. Take any 2D CNN — ResNet, EfficientNet, a ViT — run it independently on each sampled frame, then average (or max-pool, or attention-pool) the per-frame embeddings and feed the pooled vector to a classifier. ImageNet transfer is direct, the implementation is four lines, and the cost is T × one image. What you give up is order: the pool is symmetric, so the model sees a bag of appearances with no notion of motion direction.
3D convolution. Replace the (H, W) kernels with (T, H, W) kernels and let the network convolve over space and time. A 3×3×3 filter looks at a 3×3 patch in three neighbouring frames, so motion is modelled natively; stack enough of them and the receptive field grows in time as well as space. The bill arrives at once: a 3×3×3 kernel does 3× the multiply-adds of a 3×3 kernel and stores 3× the weights. I3D’s inflation trick — the subject of chapter 04 — is what made this family trainable, by bootstrapping the 3D kernels from ImageNet weights.
Spatio-temporal transformer. Tokenise the clip into a grid of space-time patches — T frames × N patches per frame — and let attention mix them. Attention has no locality bias, so a token can compare itself with a patch 40 frames away, and the same architecture handles clips over a minute long. The catch is arithmetic: joint attention over all T·N tokens costs O((T·N)²). TimeSformer’s divided attention splits each block into temporal attention (same patch, all frames) and spatial attention (same frame, all patches), turning the cost into O(T² + N²) and keeping state-of-the-art accuracy. Its ablations are the cleanest evidence for this whole lesson: space-only attention scores 76.9 on Kinetics-400 and 36.6 on Something-Something V2 — a 1.1-point loss on the appearance-heavy benchmark and a 40.3-point collapse on the motion one.
Two subtleties hide in that table. First, real 3D networks are not uniformly 3× a 2D network, because the temporal resolution is downsampled as the network deepens (R3D-18 strided convolutions cut 16 frames to 8, 4 and 2 by the last stage) and because some layers stay spatial-only. The measured 5.2× for R3D-18 against a per-frame baseline is the honest number; the 3× kernel rule is the mental model. Second, the families are not ranked by spend: MViT_V2_S buys 80.8% for 8.3× the baseline’s compute, but at the cheap end S3D reaches 68.4% for 18.0 G — less than half of R3D-18’s 40.7 G for a 5-point-better score. Choosing the right block inside a family matters as much as choosing the family.
The three families on one axis. Flip between compute, parameters and accuracy; every published number is from torchvision’s Kinetics-400 table (video-level, 16-frame clips, 112×112 crops). The 2D+pool row is computed live from ResNet-18’s per-frame cost and is the only estimate on the page.
Cost units are torchvision’s: one multiply-add counts as one op, measured on a 16-frame clip at the 112×112 crop — not the 224² image size. Comparing 2D+pool at 224² with 3D models at 112² would understate the bill by 4×.
Say the 2D backbone costs C per frame and T = 16. The three families differ only in how that per-frame cost is multiplied:
The point is not the estimate’s precision. It is that the 2D+pool bill scales linearly in T (double the frames, double the cost), the 3D bill scales linearly in T and in the temporal kernel depth, and the transformer bill scales quadratically in the token count — which is why sampling, pooling and attention-pattern choice are the levers you actually tune.
A team classifies 12 appearance-heavy categories in broadcast sports footage (tennis, swimming, gymnastics…) and wants the cheapest strong baseline next week. Which family should they start with?
A 10-second clip at 30 fps is 300 frames; a model sees a sampled handful. Whether those frames are spread across the whole clip or drawn from one contiguous window changes what the temporal kernels can possibly learn — and it is the bug most video pipelines ship with.
The standard practice is to sample T frames per clip with T ∈ {8, 16, 32, 64} — 8 to 16 for most modern models, 64 in the original I3D, whose 64-frame clip at 25 fps covers 2.56 seconds of real time. Three strategies are in common use:
Uniform sampling picks T frames evenly across the clip: 300 frames at T = 8 gives indices 0, 37, 75, …, 262, i.e. a sampled frame every 1.25 s. It maximises coverage and is the default for 2D+pool. Dense sampling picks a contiguous window of T neighbouring frames — a 300-frame clip at T = 8 gives, say, frames 114–121, one every 33 ms. This is the default for 3D convolutions, because a 3-slice temporal kernel can only compare adjacent samples: on a uniform sample it windows over 2.5 s of real time (two 1.25 s gaps), on a dense sample over 67 ms. Multi-clip sampling draws several windows from the same video, classifies each, and averages the predictions — a test-time trick that trades inference cost for stability.
Pooling sits at the end of the 2D+pool pipeline. Mean pooling is the default, max pooling keeps the strongest evidence per feature, and attention pooling learns which frames matter. All three are permutation-invariant in the same sense: reorder the frames and the pooled vector is unchanged. That is a feature when the task is appearance and a hard ceiling when the task is motion.
A 6-second clip is 180 frames; no model sees all of them. Pick a strategy and T, then watch which frames survive, how far apart they land in real time, and what a 3-slice temporal kernel can possibly learn. The ball is the whole feature on purpose — it makes pooling and its order-invariance visible.
Uniform sampling is the 2D+pool default: cover the whole clip, accept the gaps.
One more decision hides in plain sight: what counts as a clip at inference time. Training sees one sampled window; evaluation can cheaply see five or ten, score each, and average the 400-class score vectors into one video-level prediction. That is exactly what the next chapter’s metric reader exposes — and it is why clip accuracy and video accuracy are two different numbers. Higher frame counts do not automatically win either: 96-frame clips are where TimeSformer starts to shine, but for a 3D conv on a small dataset, longer clips mostly buy new ways to overfit.
A 3D network cannot use ImageNet weights — unless you copy each 2D kernel along a new time axis. That one paragraph is I3D, and it is the reason 3D convolution became practical instead of a research curiosity.
In 2017 Carreira and Zisserman asked a simple question: if a 2D network trained on ImageNet already knows edges, textures and objects, why would you train a video network from random weights? Their answer was to take a strong 2D architecture and make every filter cubic. From the paper, verbatim:
“Filters are typically square and we just make them cubic — N × N filters become N × N × N.”
“This can be achieved, thanks to linearity, by repeating the weights of the 2D filters N times along the time dimension, and rescaling them by dividing by N. This ensures that the convolutional filter response is the same.”
The second sentence is the load-bearing one. Copying a 3×3 kernel into a 3×3×3 kernel gives you 9 unique weights in 27 slots. Without the division by N, the response of the new 3D kernel to a constant input is 3× the response of the old 2D kernel — the system calls this the boring-video fixed point: an image repeated into a video must produce exactly the same activations as the image did. Dividing by N restores that identity, which is what keeps batch-norm statistics and learning rates valid on the first forward pass.
What inflation does not give you is motion. Every slice of an inflated kernel is identical, and a kernel with identical time slices computes a temporal average — a symmetric, order-invariant function. An inflated network starts at exactly the 2D network applied per frame and pooled over time. Order sensitivity is learned during fine-tuning on video, when gradient descent pulls the slices apart because Kinetics rewards motion. That is the precise sense in which inflation is a starting point, and it explains why I3D needed both the trick and a 240k-clip dataset to beat the 2D baselines.
The results justified the extra compute. On Kinetics, the paper’s own comparison table reads: a C3D-like 3D ConvNet at 56.1% top-1 (79M parameters, 16-frame inputs), a 2D two-stream network at 62.2% (12M parameters, one RGB frame plus optical flow), and two-stream I3D at 74.2% — 71.1% with RGB alone — from 25M parameters and 64-frame clips. The 3D family stopped being the expensive loser the moment its weights could start from ImageNet.
I3D’s trick in one picture: copy a 2D kernel along a new time axis, divide by the number of copies, and the response to a constant (“boring”) clip is exactly the 2D response — so the ImageNet weights are a valid starting point for a 3D network. Flip the slices to “learned” to see what fine-tuning adds.
The division is not cosmetic: a 3× larger activation entering the next batch-norm layer would shift every statistic on the first forward pass. Dividing by k_t restores the exact 2D response, which is what the paper calls the boring-video fixed point.
The paper’s reasoning is three lines of algebra. Let x be a still image and let the 3D input be that image repeated across k_t frames. The 3D kernel’s weight tensor is the 2D kernel w repeated along time and divided by k_t:
Because every layer’s output is constant in time for a constant input, pointwise non-linearities and average/max pools return exactly what they returned for the single image, and the whole network inherits the fixed point. That is why the ImageNet weights are not merely “close enough”: at initialisation, the inflated network computes precisely the 2D network’s function on boring videos. Everything the fine-tuning run adds from there is the temporal part — and it adds it without destroying the spatial features that made the 2D weights worth borrowing.
In inflation, why divide the repeated weights by k_t?
A 3D kernel mixes space and time in one operation: 27 weights that have to learn appearance and motion together. The (2+1)D block cuts the cube into a spatial square and a temporal stick, puts a non-linearity between them, and keeps the parameter budget — the same trick, with a better learning problem inside.
A full 3×3×3 convolution answers one question per output: “what combination of this 3×3×3 neighbourhood matters?” Space and time are entangled in the same 27 weights. Tran and colleagues’ 2018 study — A Closer Look at Spatiotemporal Convolutions — asked what happens if you factorise that cube into a spatial 1×3×3 convolution (one frame at a time) followed by a temporal 3×1×1 convolution (one pixel at a time, across frames), with batch norm and ReLU in between.
The factorisation splits the learning problem in a useful way: the spatial half is a genuine 2D convolution, so it can be initialised from ImageNet weights exactly as in I3D; the temporal half is 1D and cheap, so it can be trained from scratch on video. And because the block now has two non-linearities instead of one, the function it can represent is strictly richer per unit of compute. The authors choose the number of channels in the middle — the mid width — so the two stages together cost about what the full 3×3×3 convolution cost:
Read that arithmetic carefully, because it corrects a widespread misreading: (2+1)D is not primarily a parameter-saving trick. With the paper’s mid-channel rule it spends the same budget as the full 3D kernel — for 64 channels, 110,592 weights either way. What you buy with the same money is a better-behaved learning problem and an extra non-linearity. The measured outcome is the cleanest same-cost comparison in video: on Kinetics-400 with torchvision’s published weights, R3D-18 scores 63.2% with 33.4M parameters and 40.70 G per clip, while R(2+1)D-18 scores 67.5% with 31.5M parameters and 40.52 G — 4.26 points for slightly less compute. The paper’s own summary is the same sentence: roughly the same cost as R3D, higher accuracy.
Two practical notes before the lab. The factorised temporal convolution is where the time lives: with equal spatial and temporal strides, torchvision’s Conv2Plus1D puts the spatial stride on the 1×3×3 stage and the temporal stride on the 3×1×1 stage, so a stage transition downsamples space and time independently. And the rule generalises: the same factorisation idea shows up in later efficient video networks (S3D, X3D) as separable convolutions, and it is the reason “3D conv” in modern papers usually means something more structured than a plain cube.
A (2+1)D block replaces one k×k×k convolution with a spatial k×k then a temporal k×1×1, with BN + ReLU in between. The paper’s mid-channel rule keeps the two stages inside the same parameter budget as the full 3D kernel — so the extra non-linearity is close to free. Pick a layer, then move the mid channel and watch the budget.
3×3×3 either way: the split does not change the receptive field, only how the weights are arranged and how many non-linearities sit inside it.
Move the slider down to a 0.25× or 0.5× mid width to see the cheaper, narrower split; leave it at 1× to see how the paper’s rule spends the same budget as a full 3×3×3 without making the middle narrower than the 3D kernel’s channel pair.
You want a factored conv with the same receptive field (k×k×k) and about the same parameter count as the full cube. If the spatial stage has mid output channels, its weight count is in · mid · k²; the temporal stage is mid · out · k. Set their sum equal to the full cube:
The rule keeps the middle from being narrower than the full convolution’s channel pair would be, which is what the naive split (mid = in) gives up: 64→64 with mid = 64 costs only 49,152 weights, 2.25× cheaper, but the temporal stage then sees a representation half the width the cube would have built. The paper’s rule is a capacity-preserving choice, and the reason its networks are not cheaper than R3D — just better per unit of compute.
The whole lesson compiles to four short functions. Write them once and you can read a video model’s code the way you read a ResNet: the sampler says what it sees, the pooling says whether it can see order, and the convolution says what it costs.
Start with the sampler, because it is upstream of everything. These are the source’s two functions, faithful except for the guard rails:
import numpy as np
def sample_uniform(num_frames_total, T):
if num_frames_total <= 0:
raise ValueError(f"num_frames_total must be positive, got {num_frames_total}")
if num_frames_total <= T:
return list(range(num_frames_total)) + [num_frames_total - 1] * (T - num_frames_total)
step = num_frames_total / T
return [int(i * step) for i in range(T)]
def sample_dense(num_frames_total, T, rng=None):
if num_frames_total <= 0:
raise ValueError(f"num_frames_total must be positive, got {num_frames_total}")
rng = rng or np.random.default_rng()
if num_frames_total <= T:
return list(range(num_frames_total)) + [num_frames_total - 1] * (T - num_frames_total)
start = int(rng.integers(0, num_frames_total - T + 1))
return list(range(start, start + T))
print(sample_uniform(300, 8))
# → [0, 37, 75, 112, 150, 187, 225, 262]
print(sample_dense(300, 8, np.random.default_rng(0)))
# → a contiguous window [start, start + 1, …, start + 7], start ∈ [0, 292]Now the baseline. A 2D ResNet-18 with its global average pool kept, a small linear head on top, and a mean over the T per-frame feature vectors in the middle:
import torch
import torch.nn as nn
from torchvision.models import resnet18, ResNet18_Weights
class FramePool(nn.Module):
def __init__(self, num_classes=400, pretrained=True):
super().__init__()
weights = ResNet18_Weights.IMAGENET1K_V1 if pretrained else None
backbone = resnet18(weights=weights)
self.features = nn.Sequential(*(list(backbone.children())[:-1])) # global avg pool kept
self.head = nn.Linear(512, num_classes)
def forward(self, x):
# x: (N, T, 3, H, W)
N, T = x.shape[:2]
x = x.reshape(N * T, *x.shape[2:])
feats = self.features(x).view(N, T, -1)
pooled = feats.mean(dim=1) # ← order-invariant by construction
return self.head(pooled)
model = FramePool(num_classes=10, pretrained=False)
x = torch.randn(2, 8, 3, 224, 224)
print(f"output: {model(x).shape}") # torch.Size([2, 10])
print(f"params: {sum(p.numel() for p in model.parameters()):,}")Then the two tricks that make a 3D network trainable and cheap. The inflation function takes a Conv2d and returns a Conv3d whose weights are the 2D weights repeated along a new time axis and divided by the time kernel:
import torch.nn as nn
def inflate_2d_to_3d(conv2d, time_kernel=3):
out_c, in_c, kh, kw = conv2d.weight.shape
pad_h = conv2d.padding[0] if isinstance(conv2d.padding, tuple) else conv2d.padding
pad_w = conv2d.padding[1] if isinstance(conv2d.padding, tuple) else conv2d.padding
stride_h = conv2d.stride[0] if isinstance(conv2d.stride, tuple) else conv2d.stride
stride_w = conv2d.stride[1] if isinstance(conv2d.stride, tuple) else conv2d.stride
has_bias = conv2d.bias is not None
conv3d = nn.Conv3d(
in_c, out_c,
kernel_size=(time_kernel, kh, kw),
padding=(time_kernel // 2, pad_h, pad_w),
stride=(1, stride_h, stride_w), # time is never strided here
bias=has_bias,
)
weight_3d = conv2d.weight.data.unsqueeze(2).repeat(1, 1, time_kernel, 1, 1) / time_kernel
conv3d.weight.data = weight_3d # ← 27 slots, 9 unique weights
if has_bias:
conv3d.bias.data = conv2d.bias.data.clone()
return conv3d
c2d = nn.Conv2d(3, 16, kernel_size=3, padding=1, bias=False)
c3d = inflate_2d_to_3d(c2d, time_kernel=3)
print(tuple(c2d.weight.shape)) # (16, 3, 3, 3) → 432 weights
print(tuple(c3d.weight.shape)) # (16, 3, 3, 3, 3) → 1,296 slots
print(tuple(c3d(torch.randn(1, 3, 8, 32, 32)).shape)) # (1, 16, 8, 32, 32)import torch.nn as nn
class Conv2Plus1D(nn.Module):
def __init__(self, in_c, out_c, kernel_size=3):
super().__init__()
mid_c = (in_c * out_c * kernel_size ** 3) // (in_c * kernel_size ** 2 + out_c * kernel_size)
mid_c = max(8, mid_c)
self.spatial = nn.Conv3d(in_c, mid_c, kernel_size=(1, kernel_size, kernel_size),
padding=(0, kernel_size // 2, kernel_size // 2), bias=False)
self.bn = nn.BatchNorm3d(mid_c)
self.act = nn.ReLU(inplace=True)
self.temporal = nn.Conv3d(mid_c, out_c, kernel_size=(kernel_size, 1, 1),
padding=(kernel_size // 2, 0, 0), bias=False)
def forward(self, x):
return self.temporal(self.act(self.bn(self.spatial(x))))
c = Conv2Plus1D(3, 16)
print(tuple(c(torch.randn(1, 3, 8, 32, 32)).shape)) # (1, 16, 8, 32, 32)
print(f"params: {sum(p.numel() for p in c.parameters()):,}")Use it. Two libraries cover production video work. torchvision.models.video ships R(2+1)D, MC3, S3D, MViT and Swin3D with Kinetics-400 weights behind the same weights= API as the image models — r2plus1d_18(weights="KINETICS400_V1") and you are fine-tuning a motion model. Meta’s pytorchvideo adds the data side: Kinetics, SSv2 and AVA loaders plus the standard transforms. For video-language work — captioning, video QA — the transformers library carries VideoMAE, VideoLLaMA and InternVideo, and those models are where spatio-temporal attention stops being a classifier and starts being a conversation partner.
Six datasets and two accuracy numbers are all you need to read any video paper’s results table — and to tell whether a headline gain came from a better model or from a more generous evaluation protocol.
Kinetics-400 is the ImageNet of video: 400 action classes, roughly 240,000 training clips and 20,000 validation clips of about 10 seconds each, crawled from YouTube (306,245 clips in the original release). Kinetics-600 and Kinetics-700 extend the same collection class by class to about 650,000 clips. Its classes are mostly things you can name from a good still frame — playing cello, eating cake, petting a dog — which is why it rewards appearance models so strongly.
Something-Something V2 is the counterweight: 174 classes and 220,847 clips of scripted hand-object interactions whose labels are motion sentences with mirrored twins. “Pushing something from left to right” and “pushing something from right to left” are the same pixels in every frame. The cleanest published proof of what that costs an appearance-only model comes from TimeSformer’s ablation: the same backbone with space-only attention scores 76.9% on Kinetics-400 and 36.6% on SSv2; adding temporal attention moves those to 78.0% and 59.5%. One dataset barely notices the time axis is gone; the other loses 40 points.
The older benchmarks still appear in tables. UCF-101 (101 classes, 13,320 clips, 27 hours) and HMDB-51 (51 classes, 6,766 clips) are small enough to train on a single GPU and small enough to overfit; they are useful as transfer targets — two-stream I3D reaches 93.4% on UCF-101 after Kinetics pretraining — not as pretraining corpora. AVA changes the task: 80 atomic visual actions densely annotated in 430 fifteen-minute movie clips, 1.62M labels, asking who did what in which second. Classification “what is this clip” is the first rung; localisation in space and time is the next one.
First the leaderboards: which datasets reward appearance and which one only motion can solve. Then the metric that confuses everyone on first contact — a clip is one sampled window, a video is the average across windows. Click any clip to flip its prediction and watch the two numbers move apart.
The ImageNet of video: 400 action classes crawled from YouTube (306,245 clips in the release). 'Playing cello', 'eating cake', 'petting dog' are object-and-scene problems in disguise. Unit: ~10 s YouTube clips.
Kinetics-600 and 700 extend the same crawl class by class; the original I3D paper pretrained on Kinetics-400 and beat every 2D model on UCF-101 and HMDB-51 by transferring. Unit: ~10 s YouTube clips.
Labels are sentences like 'Pushing something from left to right' with a mirrored twin. The two clips contain the same objects in every frame — only the order differs. Space-only attention drops to 36.6% here while scoring 76.9% on Kinetics-400. Unit: short scripted clips.
The pre-Kinetics benchmark. I3D reaches 93.4% (two-stream) on it after Kinetics pretraining — evidence that video pretraining transfers the way ImageNet pretraining does. Unit: YouTube clips.
51 motion classes with 1,000+ clips each, extracted from films. Small enough that a 2D+pool baseline with a strong ImageNet backbone can be competitive. Unit: movies + YouTube.
Atomic visual actions: not 'what class is this clip' but 'which person is doing which action in which second'. Localisation in space and time is the next problem after classification. Unit: 1.62M action labels.
Dropping temporal attention costs 1.1 points on Kinetics-400 and 40.3 points on Something-Something V2. That gap is the whole reason the dataset exists.
The two-class simplification is deliberate: each clip carries a score for the correct class and for its strongest wrong class, and a video is correct when the averaged correct score wins. Real evaluations average the full 400-class score vectors — same idea, more numbers.
Two habits separate a careful video evaluation from a sloppy one. First, always report both numbers and say how many clips per video you averaged, because a single clip-level number hides the variance and a video-level number flatters the model with test-time augmentation. Second, look at both datasets if your task involves motion at all: a model that gains three points on Kinetics and nothing on SSv2 has probably learned better objects, not better motion, and your deployment task decides which of those you needed.
Your Kinetics-400 model reports 76% clip accuracy and 82% video accuracy. What does the 6-point gap tell you?
Answer before you look. The order-invariance question, the inflation question and the clip-vs-video question are the three that separate a memorised list of model names from a mental model you can use on the next video architecture you meet.
0 / 5 answered · 0 correct
01Why does a 2D+pool video model fail on the Something-Something V2 dataset?
02What is I3D's inflation trick?
03A (2+1)D factorised convolution splits a 3D conv into which two operations?
04In a video transformer, what does 'divided attention' mean?
05Your Kinetics-400 model reports 76% clip accuracy and 82% video accuracy. What does the gap tell you?
Click a card to swap the lazy description for what it actually means.
Three problems with exact numbers — cost a 2D+pool baseline against a 3D one, prove that appearance alone cannot solve a motion task, and build the (2+1)D version that can. Try first; a worked answer is one click away.
Take the standard video crop of 112×112. torchvision publishes ResNet-18 at 1.81 G multiply-adds per 224² image; the same layer stack at 112² costs 1.81 / 4 ≈ 0.485 G per frame because every spatial dimension halves (the pooling layout makes that a slight under-estimate, but it is the right order). FramePool with T = 8 costs 8 × 0.485 = 3.88 G per clip; with T = 16 it is 7.76 G. Naively inflating every 3×3 conv multiplies each layer's cost by k_t = 3, giving 3.88 × 3 = 11.6 G at T = 8, or ≈23.3 G at T = 16. The measured torchvision R3D-18 costs 40.70 G on a 16-frame clip — about 20.4 G per 8-frame clip, 5.3× the FramePool bill. The ratio is 3–5× rather than a clean 3× for two reasons. First, real 3D networks downsample time as they deepen: R3D-18's stage-transition strides take 16 frames to 8, 4 and 2 by the last stage, so the deep, wide layers run on far fewer frames than the naive estimate assumes. Second, the 2D counterpart in deployment usually runs batched frames on a GPU for a fraction of the memory traffic per frame, while the 3D model must materialise the T×H×W volume at once. The number to remember: same crop, same clip length, roughly 5× the compute for native motion.
Generate three classes by drawing a random starting position, then stepping the ball by a fixed velocity each frame; label by direction. Balance the classes so each direction sees the same distribution of positions. Train FramePool with the clip sampler set to dense windows and cross-entropy. The expected result is chance: because the two mirrored trajectories visit the same set of positions arranged differently, and FramePool's prediction is a function of the multiset of per-frame features, the model can only learn position priors, not direction — with balanced positions it lands near 33%. Prove it, do not just measure it: evaluate the test set twice, once with frames in order and once with the frames of every clip reversed, and print max |logits(x) − logits(x_reversed)|. For FramePool it is exactly 0 — the two evaluations produce bit-identical tensors. Then confirm the dataset is not broken by training a dense 3D baseline (or replacing FramePool's mean with the Conv2Plus1D block from the build chapter): it can reach 90%+ because it reads the change between adjacent frames. The pair of numbers — 0 difference and 33% accuracy on one side, 90%+ on the other — is the whole argument for temporal modelling.
Three steps. (1) Start from torchvision's ResNet-18 and swap each BasicBlock's 3×3 Conv2d for the factorised block: spatial 1×3×3 to a mid width computed by mid = in·out·27/(in·9 + out·3), then BN + ReLU, then temporal 3×1×1 to the block's output width, with the stride split — spatial stride on the first stage, temporal stride on the second, exactly as torchvision's own r2plus1d_18 does. The stem becomes the R(2+1)D version: a 7×7 spatial conv with time stride 1, then a 3×1×1 temporal conv. (2) Inflate: copy the ImageNet stem's 7×7 weights into the spatial half of the new stem (the temporal half starts as the identity-ish average), and for every factorised block copy the 2D kernel into the spatial stage, initialising the temporal stage with w/3 in each of its three time slices — the boring-video fixed point. (3) Train on the same clips and sampler as exercise 2 with a smaller learning rate on the pretrained spatial weights than on the temporal ones. Expected outcome: the motion dataset becomes solvable, because the temporal 3×1×1 stage can learn the first difference that distinguishes the mirrored classes, and accuracy lands far above FramePool's ~33%. The shortcut is to notice that torchvision ships the finished object: r2plus1d_18(weights='KINETICS400_V1') is exactly this network pretrained on Kinetics-400 — the exercise's value is in seeing the two stages, the mid-channel arithmetic, and where the 2D weights land inside a 3D network.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 04, Lesson 12) and the Math Foundations Notebook reference build. The five labs (the canvas approach comparator, the canvas frame-sampling lab with the reverse-clip test, the canvas inflation visualizer with the boring-video check, the (2+1)D splitter with the mid-channel budget, and the dataset + clip-vs-video metric reader) are original to this page, as are the order-invariance arithmetic with its four-frame numeric check, the measured cost table (2D+pool 16 × 0.485 = 7.8 G, R3D-18 40.7 G ≈ 5.2×, R(2+1)D-18 40.52 G, S3D 17.98 G, MViT_V2_S 64.22 G, Swin3D_B 140.67 G), the inflation bookkeeping (9 unique weights in 27 slots, ÷ k_t, the fixed point derived and checked numerically), the (2+1)D mid-channel arithmetic (64→64: mid 144, 82,944 + 27,648 = 110,592 = the full cube), the R3D-18 63.2% vs R(2+1)D-18 67.5% same-cost pair, I3D's Kinetics table (56.1 / 62.2 / 67.2 / 71.1 / 74.2), the space-only-attention ablation (76.9 K400 / 36.6 SSv2 vs 78.0 / 59.5 divided), the sampling-gap arithmetic (0.75 s uniform vs 33 ms dense for a 3-slice kernel), the dataset census (Kinetics-400 ≈240k train, Kinetics-700 ≈650,317, SSv2 220,847 clips across 174 classes, UCF-101 13,320, HMDB-51 6,766, AVA 430 videos / 80 actions / 1.62M labels), the 5-video × 4-clip metric board (13/20 = 65% clip vs 4/5 = 80% video), and the INFLATE REPEATS, (2+1)D SPLITS memory hook. Every number shown is computed live by the labs or verified by hand in the prose.