Pipeline Parallel and Bubble Analysis
Tensor parallelism splits the matrix multiply across ranks. Pipeline parallelism splits the model across ranks, one stage per rank. Microbatches flow through the pipeline. The empty time at the start and end is the bubble; minimising it is the whole craft.
Visual edition planned
This lesson isn’t interactive yet.
It is part of the curriculum and will get the same treatment as Phase 1 — a visual cover, hands-on labs, derivations with numeric checks and a quiz. Until then, the original lesson is the best place to read it:
Part of Phase 19 — Capstone Projects. Use the previous / next cards below to keep browsing the phase.