A matrix multiply is billions of independent little sums — so the machine that does them thousands at a time wins. This lesson sets up the three ways to get that machine (your own GPU, free Colab, or a rented one), benchmarks CPU against GPU honestly, and works out how much model actually fits in memory.
Training is dominated by matrix multiplication, and one n×n matmul costs 2n³ floating-point operations — for n = 5000 that is 2.5×10¹¹ of them, all independent. A central processing unit (CPU) runs a few very fast cores one step at a time; a graphics processing unit (GPU) runs thousands of slower lanes at once. That is the whole trade: latency versus throughput, and throughput is what training needs.
2n³ FLOPs · n = 5000 → 2.5×10¹¹ per matmulA local NVIDIA GPU costs nothing per hour if you already own it; Google Colab's free T4 needs no setup at all; cloud providers rent everything from a T4 to an A100 for roughly $0.20–2.00 per hour (the source's range — prices change). The right answer depends on how often you train: daily → local, occasionally → Colab, seriously → rent.
$0 local · $0 Colab · ≈ $0.20–2.00/hr rented (example)Storing a 7B model in fp16 takes 14 GB. Training it is a different bill: fp32 gradients (28 GB) and Adam's two fp32 moments (56 GB) ride along, plus activations that grow with the batch. The 2-bytes-per-parameter rule gets you 12B parameters on a 24 GB card for inference — and about 1.7B for Adam training.
7B: 14 GB stored → ≈ 98 GB to train with AdamA GPU does not compute faster — it computes wider: thousands of small operations happen at once, which is exactly what matrix multiplication is; and the first wall you hit is not speed but memory, because training stores far more per parameter than inference does.
By the end you will be able to verify a GPU with nvidia-smi and torch.cuda, read the Apple Silicon MPS path and the no-GPU fallback, run the source’s 5000×5000 benchmark with torch.cuda.synchronize() and explain why the barrier matters, compute a training-memory budget (7B in fp16 is 14 GB, but Adam training is ≈ 98 GB), decide between a local card, Colab and a rental, and write one script that picks its device at runtime instead of assuming one.
Phases 1–3 of this course run fine on a CPU. But training CNNs, transformers and LLMs is a wall of matrix multiplication, and the source’s framing says it plainly: a run that takes 8 hours on a CPU takes about 10 minutes on a GPU. Why? Not because a GPU is faster at any one thing — because it does thousands of things at once.
Every layer of a neural network is a matrix multiplication in disguise. The forward pass is y = xW + b; the backward pass is more of the same with transposed shapes; even attention is a few matmuls glued together. One multiply of an n×n matrix by another costs 2n³ floating-point operations — counting a multiply and an add separately, as the standard bookkeeping does. For n = 5000:
Here is the part that makes hardware matter: every one of those 2.5×10¹¹ operations is independent. Each entry of the output is its own dot product; nothing waits for anything else. Work like this maps badly onto a CPU — a handful of very fast cores, each optimised for doing one complicated thing after another — and perfectly onto a GPU, which is thousands of small arithmetic lanes built to run the same instruction across a sea of data.
Do the crude arithmetic: 8 CPU cores at ~4 GHz give about 8 × 4×10⁹ ≈ 3.2×10¹⁰ lane-cycles per second, while a modest data-center GPU with 2,560 lanes at ~1.6 GHz gives 2,560 × 1.6×10⁹ ≈ 4.1×10¹² — roughly 100× more parallel width. A GPU lane is individually slower and simpler, which is why a GPU loses on small one-off operations and wins by a lot on big ones. This is a teaching estimate, not a benchmark: real speedups depend on the card, the precision, memory bandwidth and how well the library parallelises your exact shapes.
One more source detail worth noticing: the lesson text benchmarks a 5000×5000 matmul, while the bundled gpu_check.py uses size = 4000. That is 2 × 4000³ = 1.28×10¹¹ FLOPs — about half the work, from a 20% smaller matrix, because the cost grows with the cube of the size. Doubling the matrix multiplies the FLOPs by eight.
This single ratio explains the whole lesson. Small matmuls leave both machines idling, so the fixed costs — kernel launch, host-to-device transfer, Python overhead — dominate and the GPU can even lose. Large matmuls give the GPU enough independent work to fill every lane, and the fixed costs vanish into the noise. Training loops are the large case by construction: the same big weights are reused for every step, so the GPU never runs out of work.
The same matrix multiply, split across CPU cores and GPU lanes. Everything here is a teaching simulation — but the shape is the real story: small matrices can be slower on a GPU, and big ones win by a lot.
This model charges the GPU one tick of launch overhead, so at the smallest sizes the win disappears entirely. Real hardware adds library warm-up, memory-bandwidth limits and kernel-launch queues on top — which is exactly why the source’s benchmark prints whatever ratio your machine actually produces.
Why does splitting a 5000×5000 matrix multiply across thousands of GPU lanes work so well?
You do not need to buy anything to start training on a GPU. The source lays out three routes: a local NVIDIA card, Google Colab’s free tier, or a rented cloud instance. Each one trades money against setup and flexibility in a different way.
Route 1 — a local NVIDIA GPU. If the machine on your desk already has an NVIDIA card, the marginal cost is $0 and the loop is the fastest: no upload, no SSH, no session timeout. The setup is a driver plus the CUDA toolkit — CUDA (Compute Unified Device Architecture) is NVIDIA’s platform for running general-purpose code on its GPUs — and its deep-learning library, cuDNN (CUDA Deep Neural Network library); in practice a prebuilt PyTorch wheel bundles what it needs, so “install” usually means picking the right wheel. The catch is the hardware you already own: an 8 GB card caps you at small models, and driver-versus-toolkit mismatches are a classic first-day problem.
Route 2 — Google Colab, free tier. No setup at all: open a notebook, choose Runtime → Change runtime type → T4 GPU, and verify from inside the notebook. It is the right answer for “I have no GPU and I want to try this today”, and the lessons that need one link a Colab notebook. Free has limits: sessions are recycled, GPUs are shared and queued, and a long training job is not safe there. Treat Colab as a lab bench, not a server.
Route 3 — a rented cloud GPU. Lambda, RunPod and Vast.ai rent GPU instances by the hour, roughly $0.20–2.00/hour in the source’s range, priced by card class and on-demand versus spot. You get a real machine over SSH (Secure Shell), install what you need, and run as long as you pay. This is the route for serious training — and the one that can quietly cost real money, so checkpoints and a spending cap come first.
| route | cost | setup | best for | watch out |
|---|---|---|---|---|
| Local NVIDIA GPU | $0/hr (you own it) | Driver + CUDA toolkit (cuDNN via wheel) | Regular use, large local datasets | Capped by your card; version mismatches |
| Google Colab, free | $0 | None — pick T4 GPU at runtime | Quick experiments, no GPU at home | Sessions time out; no guarantee of a GPU |
| Rented cloud GPU | ≈ $0.20–2.00/hr | SSH + install | Serious training, large models | The meter runs; preemption on spot |
| Apple Silicon (MPS) | $0/hr (you own it) | Nothing beyond PyTorch | Learning, small models, quiet local dev | Not CUDA; op coverage varies; shared memory |
# Runtime → Change runtime type → T4 GPU, then in a cell:
!nvidia-smi
import torch
print(torch.cuda.get_device_name(0), torch.cuda.get_device_properties(0).total_memory / 1e9)ssh user@your-gpu-instance
pip install torch torchvision torchaudio
python -c "import torch; print(torch.cuda.get_device_name(0))"
# then: clone your code or upload it, and start with a smoke test
# before launching the multi-hour run you are paying forGPU time is a meter running. Pick a card class, set the hours and the number of runs, and see what a training sweep costs. Every rate here is an example for teaching, not a quote.
fine-tunes small models; serves a 7B model with room to spare
| class | VRAM | on-demand | spot |
|---|---|---|---|
| T4-class · 16 GB | 16 GB | $0.30/h | $0.15/h |
| A10 / L4-class · 24 GB | 24 GB | $0.60/h | $0.30/h |
| A100 80 GB-class | 80 GB | $1.60/h | $0.80/h |
Spot instances are cheaper because they can be taken away mid-run — which is why checkpointing is not optional on spot. On-demand costs more and still needs a spending cap: a forgotten instance bills by the hour, not by the idea.
Two minutes of checking saves an afternoon of debugging. The source’s first move is nvidia-smi, then a short PyTorch report — and both have exact outputs you can read like a receipt.
nvidia-smi is the NVIDIA System Management Interface: a command-line tool that talks to the driver, not to Python. It answers the question “is there a GPU here at all, and what is it doing?” before any framework gets involved.
nvidia-smi
# the output is a receipt for your hardware:
# driver version the software that talks to the card
# CUDA version the highest CUDA API this driver supports
# GPU name e.g. NVIDIA GeForce RTX 4090
# memory used / total VRAM
# utilization % of the last second the GPU was busy
# processes which PIDs are using the card right nowOnce the driver is fine, ask PyTorch the same questions in code — this is the source’s device report, extended with the memory line:
import torch
print(f"CUDA available: {torch.cuda.is_available()}")
print(f"CUDA version: {torch.version.cuda}")
if torch.cuda.is_available():
print(f"GPU: {torch.cuda.get_device_name(0)}")
props = torch.cuda.get_device_properties(0)
print(f"Memory: {props.total_memory / 1e9:.1f} GB")
print(f"Compute capability: {props.major}.{props.minor}")
# Apple Silicon has its own backend — Metal Performance Shaders (MPS)
print(f"MPS available: {torch.backends.mps.is_available()}")
print(f"MPS built: {torch.backends.mps.is_built()}")| probe | NVIDIA workstation | Apple Silicon Mac | CPU-only laptop |
|---|---|---|---|
nvidia-smi | driver + CUDA 12.4 + card + VRAM | command not found | command not found |
torch.cuda.is_available() | True | False | False |
torch.version.cuda | “12.4” | None | None |
torch.cuda.get_device_name(0) | NVIDIA GeForce RTX 4090 | — (no CUDA device) | — (no CUDA device) |
total_memory / 1e9 | 25.8 GB (a 24 GiB card) | — | — |
torch.backends.mps.is_available() | False | True | False |
torch.device(…) | cuda | cpu (MPS is opt-in) | cpu |
The no-GPU path is a first-class path. The source is explicit: most lessons work on CPU, and the ones that need a GPU say so and include Colab links. If you are on an Apple Silicon Mac, PyTorch ships a Metal-backed backend (MPS — Metal Performance Shaders) that accelerates a useful subset of operations on macOS 12.3+, but it is not CUDA and it is not picked up automatically.
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(f"Using: {device}")
# on a Mac this prints "cpu" — MPS is not part of the cuda-or-cpu idiom.
# The next chapter shows the three-way version.Your friend has an Apple Silicon Mac with an M2 chip and a recent PyTorch. What do these two lines print?
One script, three machines. Step through it line by line and watch the same code land on CUDA, MPS or CPU — including the errors you get when you ask for a backend that is not there.
nvidia-smiprint(torch.cuda.is_available())print(torch.version.cuda, torch.cuda.get_device_name(0))print(torch.backends.mps.is_available())print(round(torch.cuda.get_device_properties(0).total_memory / 1e9, 1))print(torch.device("cuda" if torch.cuda.is_available() else "cpu"))t = torch.randn(3, 3).to("cuda"); print(t.device)torch.cuda.synchronize(); print("kernel finished")Linux box with an RTX 4090 (24 GiB) and the NVIDIA driver installed.
The errors here are the real ones you will meet, though the exact wording shifts between PyTorch versions. Notice the Mac: the usual cuda if available else cpu line never touches MPS — Apple’s backend is opt-in.
The source’s benchmark looks like a beginner’s five-liner, and one line in it is doing something subtle: torch.cuda.synchronize(). Delete it and the GPU appears a thousand times faster — because the stopwatch never waited for the work.
GPU operations are asynchronous. When Python says c_gpu = a_gpu @ b_gpu, it enqueues a kernel on the device’s work queue and returns immediately; the multiply happens later, on the GPU. That is a feature — the host can prepare the next batch while the device chews — but it breaks naive timing.
The source’s fix brackets the measured region with a barrier on both sides: synchronize before starting the timer (so pending copies are done), and synchronize before stopping it (so the multiply is done). The measured interval is then work, not enqueue.
import torch
import time
size = 5000
a_cpu = torch.randn(size, size)
b_cpu = torch.randn(size, size)
start = time.time()
c_cpu = a_cpu @ b_cpu
cpu_time = time.time() - start
print(f"CPU: {cpu_time:.3f}s")
if torch.cuda.is_available():
a_gpu = a_cpu.to("cuda")
b_gpu = b_cpu.to("cuda")
torch.cuda.synchronize() # wait for the copies to land
start = time.time()
c_gpu = a_gpu @ b_gpu
torch.cuda.synchronize() # wait for the multiply to finish
gpu_time = time.time() - start
print(f"GPU: {gpu_time:.3f}s")
print(f"Speedup: {cpu_time / gpu_time:.0f}x")Teaching estimates for one 5000×5000 fp32 matmul, chosen so every figure below can be checked by hand. They are example rates, not a measurement of your machine:
The illusion is not small — it is the difference between “the GPU is magic” and “the GPU is about 24× faster here”. And the honest number is the one you can act on: it tells you that a 10-hour CPU run becomes ~25 minutes, not 36 seconds, which is the difference between a coffee break and a nap.
Two footnotes that make real timings even messier. First, the first GPU call is slow: libraries initialise, kernels get compiled and tuned, so benchmarks warm up with dummy iterations before recording. Second, for precise timing use torch.cuda.Event rather than time.time(); events are recorded on the GPU timeline itself. The barrier is still the point: without it, you are timing Python.
A GPU call returns before the work is done. Toggle synchronize() and watch the stopwatch: without it the timer stops while the kernel is still running.
The same pitfall hides everywhere: timing with time.time() and no barrier, comparing an async run to a sync one, or benchmarking the first call while the GPU compiles kernels. Warm up, synchronize, then measure.
You time a GPU matmul and get 0.006 s, then print a 900× speedup over the CPU. What most likely went wrong?
VRAM — video RAM, the memory on the GPU, separate from your system’s RAM — is the wall. The source gives the standard rule of thumb: 2 bytes per parameter for fp16. It is a good rule and a dangerous one, because it answers the inference question, while training asks a much more expensive one.
The source’s arithmetic is simple enough to do in your head. A model with 7 billion parameters in fp16 (16-bit floating point — half the storage of its 32-bit sibling fp32, with minimal accuracy loss for inference) uses:
Now the trap this lesson exists to disarm: that 14 GB figure is weights only. During training, every parameter also carries a gradient, optimizers keep their own state, and the forward pass leaves activations behind for the backward pass to use. The moment you add those, the picture changes by an order of magnitude.
A second example at a size people actually fine-tune on a single consumer card — 1.5B parameters, fp16, batch 1:
| card | fp16 inference (2 B/param) | train with SGD (6 B/param) | train with Adam (14 B/param) |
|---|---|---|---|
| 8 GB | 4 B params | 1.3 B params | 0.6 B params |
| 12 GB | 6 B params | 2.0 B params | 0.9 B params |
| 16 GB | 8 B params | 2.7 B params | 1.1 B params |
| 24 GB | 12 B params | 4.0 B params | 1.7 B params |
| 80 GB | 40 B params | 13.3 B params | 5.7 B params |
When the budget does not fit, engineers change the budget. Gradient checkpointing stores a few activations and recomputes the rest during the backward pass, trading compute for memory (the planner’s ÷8 is a teaching estimate). Parameter-efficient fine-tuning — LoRA (Low-Rank Adaptation) and its quantized cousin QLoRA — freezes the base weights and trains small adapter matrices instead: a 7B base in 4-bit form plus adapters is a well-trodden 24 GB recipe, because the frozen base costs about 1 byte per weight and far fewer parameters carry optimizer state. Sharding — FSDP (PyTorch’s Fully Sharded Data Parallel) or ZeRO (DeepSpeed’s Zero Redundancy Optimizer) — splits the 98 GB across several cards. And gradient accumulation gives you the gradient of a large batch while only ever holding the activations of a small one.
Weights are only the first line of the bill. Move the sliders and watch gradients, optimizer states and activations stack up — and find out which of the five card sizes can actually hold the run.
The rule of thumb answers “will it run?”. Training asks a harder question — every parameter also carries a gradient, Adam carries two more numbers, and activations grow with the batch. That is how a 14 GB model becomes a 98 GB training job.
A 7B model in fp16 is 14 GB, so it loads on a 16 GB T4. Can you train it there with Adam?
The source’s device line — torch.device(“cuda” if torch.cuda.is_available() else “cpu”) — is the smallest habit with the biggest payoff. It is what lets the same notebook run on a T4, a Mac, and the laptop you are reading this on.
Hard-coding .cuda() is the classic mistake: it works on exactly one kind of machine and fails everywhere else with an error that says nothing about the fix. The pattern that survives every environment is ask, then use: probe the backends at runtime, pick one, and move both the model and every batch onto it.
import torch
def pick_device() -> torch.device:
if torch.cuda.is_available(): # NVIDIA
return torch.device("cuda")
if torch.backends.mps.is_available(): # Apple Silicon (opt-in)
return torch.device("mps")
return torch.device("cpu")
device = pick_device()
print(f"Using: {device}")
model = model.to(device) # move the model once
for x, y in loader: # move every batch to the same place
x, y = x.to(device), y.to(device)
...
# checkpoints: load them wherever you are
state = torch.load("checkpoint.pt", map_location=device)Three habits make the pattern stick. First, never call .cuda() outside a probe — use .to(device) everywhere. Second, branch the timing code too: torch.cuda.synchronize() for NVIDIA, torch.mps.synchronize() for Apple, nothing on CPU — which is why the benchmark from the last chapter wraps its barrier in an if torch.cuda.is_available(). Third, let environment variables steer, not code: the same script should run in CI (continuous integration) on CPU, in Colab on a T4, and on your Mac without a single edit.
Answer before you look. The synchronize question and the VRAM question are the two that separate “I read the lesson” from “I can budget a training run”.
0 / 5 answered · 0 correct
01Why is a GPU faster than a CPU for training neural networks?
02What does VRAM refer to?
03What command verifies that your NVIDIA GPU is detected and shows its current status?
04When benchmarking GPU vs CPU matrix multiplication, why must you call torch.cuda.synchronize() before measuring GPU time?
05Using the fp16 rule of thumb, approximately how many parameters can fit in 24 GB of VRAM?
Click a card to swap the lazy description for what it actually means.
Three problems with exact numbers — run the benchmark and read the ratio, take it to Colab if you have no GPU, and turn your card’s memory into a parameter budget. Try first; a worked answer is one click away.
Expect these shapes, not exact numbers — every machine differs. The benchmark builds two size×size fp32 matrices, times a_cpu @ b_cpu with time.time(), then (if CUDA exists) copies both to the GPU, calls torch.cuda.synchronize() before starting the timer, multiplies, synchronizes again and stops the timer. Read the output in three parts. (1) The CPU number: on a laptop, a 5000×5000 matmul is usually several seconds (the lesson's teaching estimate is 5.0 s at ≈50 GFLOP/s — a quarter of a trillion FLOPs at 5×10¹⁰ per second). (2) The GPU number: a fraction of a second on a modern card (the teaching estimate is 0.20 s), and it should include the 5 ms launch overhead rather than hide it. (3) The ratio: expect double digits, e.g. 5.00 / 0.205 ≈ 24×, and be suspicious of three- or four-digit ratios — that is the missing-synchronize signature. If you double size to 10,000, the FLOPs go ×8 (2n³) while the operand bytes only ×4, so the GPU's advantage grows: more independent work per launch. Two caveats to check before believing your own numbers: warm up the GPU first (the first kernel pays library initialisation), and make sure both synchronize() calls are present. If the GPU branch never runs (no CUDA), the script prints the CPU line only — that is the designed fallback, not a failure.
In Colab: File → New notebook, then Runtime → Change runtime type → T4 GPU, then paste the benchmark and run it with !nvidia-smi first to confirm the hardware. The free T4 has 16 GB of VRAM and its fp32 throughput is a fraction of a data-centre card's — the teaching estimate of 1.25 TFLOP/s effective gives 2.5×10¹¹ / 1.25×10¹² ≈ 0.20 s, but a realistic free-tier run can be slower because the CPU runtime it pairs with is also fast and because the shared GPU may be busy. Compare like with like: same size, same precision, same synchronize() placement. Two Colab-specific traps: the runtime is recycled when you are idle, so re-run the checks rather than trusting a stale state; and the 'CPU' you compare against is Colab's host, not your laptop — to compare your laptop, export the CPU time from your local run and compare against the notebook's GPU number. The lesson's point is not the exact ratio but the order of magnitude: minutes instead of hours, at $0.
Run torch.cuda.get_device_properties(0).total_memory / 1e9. A free Colab T4 reports about 15.8–16.0 GB; a 24 GiB RTX 4090 reports about 25.8 GB (decimal GB, because 1 GiB = 1.0737×10⁹ bytes). Then three divisions. Inference: capacity ÷ 2 bytes per parameter — 16 GB → 8B parameters, 24 GB → 12B, 80 GB → 40B. Training with SGD: capacity ÷ (2 + 4) bytes per parameter = capacity ÷ 6 — 16 GB → 2.7B, 24 GB → 4.0B, 80 GB → 13.3B. Training with Adam: capacity ÷ (2 + 4 + 8) = capacity ÷ 14 — 16 GB → 1.1B, 24 GB → 1.7B, 80 GB → 5.7B. Subtract room for activations (roughly 0.5 GB at 7B params and batch 1, scaling with batch) plus the CUDA context, so leave a gigabyte or two of headroom. Cross-check one line with the full budget: a 7B model is 14 GB of fp16 weights, 28 GB of fp32 gradients and 56 GB of Adam moments ≈ 98.5 GB — which is why the honest answer for a 16 GB T4 is 'serve 7B, train about 1B'. When the budget does not fit, change the budget: gradient checkpointing shrinks activations, LoRA/QLoRA trains adapters instead of all parameters, and FSDP/ZeRO shards the 98 GB across cards.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text, quiz and the benchmark code are adapted from AI Engineering from Scratch (Phase 00, Lesson 03) and the Math Foundations Notebook reference build. The five labs (the simulated throughput race, the VRAM budget planner, the cloud cost calculator, the device-dispatch simulator and the timing-pitfall lab) are original to this page, as are the matmul arithmetic (2n³, 2.5×10¹¹ FLOPs for 5000², 100 MB per fp32 operand), the rough lane-width comparison, the source's 5000-versus-4000 size check, the async-illusion numbers (CPU 5.0 s, kernel 0.20 s, launch 5 ms → “1000×” untimed versus ≈24× honest, a 41× illusion), the full 7B training budget (14 GB weights + 28 GB fp32 gradients + 56 GB Adam moments + ≈0.5 GB activations ≈ 98.5 GB) with the 1.5B second example and the 8/12/16/24/80 GB capacity table, the ≈16-bytes-per-parameter mixed-precision cross-check, the Apple Silicon MPS path, the three-way device dispatch, the CUDA-version triage, the LoRA/QLoRA/checkpointing/sharding escape hatches, and the memory hooks. Every hourly rate and every simulated timing is labelled as a teaching estimate, and the labs compute their numbers live.