Transformers Deep Dive
The architecture that changed everything. Understand every layer.
16 lessons · 0 visual stories
0/16 complete · 0%
#LessonTypeLanguagesTimeOpen
01Why Transformers — The Problems with RNNsRNNs process tokens one at a time. Transformers process all tokens at once. That single architectural bet changed every scaling curve in deep learning after 2017.LearnPython45 min↗ Official lesson02Self-Attention from ScratchAttention is a lookup table where every word asks "who matters to me?" - and learns the answer.BuildPython75 min↗ Official lesson03Multi-Head AttentionOne attention head learns one relation at a time. Eight heads learn eight. Heads are free. Take more of them.BuildPython75 min↗ Official lesson04Positional Encoding — Sinusoidal, RoPE, ALiBiAttention is permutation-invariant. "The cat sat on the mat" and "mat the on sat cat the" produce the same output without positional signal. Three algorithms fix it — each with a different bet on what "position" means.BuildPython45 min↗ Official lesson05The Full Transformer — Encoder + DecoderAttention is the star. Everything else — residuals, normalization, feed-forward, cross-attention — is the scaffolding that lets you stack it deep.BuildPython75 min↗ Official lesson06BERT — Masked Language ModelingGPT predicts the next word. BERT predicts a missing word. One sentence of difference — and half a decade of everything embedding-shaped.BuildPython45 min↗ Official lesson07GPT — Causal Language ModelingBERT sees both sides. GPT sees only the past. The triangle mask is the most consequential single line of code in modern AI.BuildPython75 min↗ Official lesson08T5, BART — Encoder-Decoder ModelsEncoders understand. Decoders generate. Put them back together and you get a model built for input → output tasks: translate, summarize, rewrite, transcribe.LearnPython45 min↗ Official lesson09Vision Transformers (ViT)An image is a grid of patches. A sentence is a grid of tokens. The same transformer eats both.BuildPython45 min↗ Official lesson10Audio Transformers — Whisper ArchitectureAudio is an image of frequency over time. Whisper is a ViT that eats mel spectrograms and speaks back.LearnPython45 min↗ Official lesson11Mixture of Experts (MoE)A dense 70B transformer activates every parameter for every token. A 671B MoE activates only 37B per token and beats it on every benchmark. Sparsity is the most important scaling idea of the decade.BuildPython45 min↗ Official lesson12KV Cache, Flash Attention & Inference OptimizationTraining is parallel and FLOP-bound. Inference is serial and memory-bound. Different bottleneck, different tricks.BuildPython75 min↗ Official lesson13Scaling LawsThe 2020 Kaplan paper said: bigger model, lower loss. The 2022 Hoffmann paper said: you were under-training. Compute goes into two buckets — parameters and tokens — and the split is not obvious.LearnPython45 min↗ Official lesson14Build a Transformer from Scratch — The CapstoneThirteen lessons. One model. No shortcuts.BuildPython120 min↗ Official lesson15Attention Variants — Sliding Window, Sparse, DifferentialFull attention is a circle. Every token sees every token, and memory pays the price. Four variants bend the shape of the circle and recover half the cost.BuildPython60 min↗ Official lesson16Speculative Decoding — Draft, Verify, RepeatAutoregressive decoding is serial. Each token waits for the previous one. Speculative decoding breaks the chain: a cheap model drafts N tokens, the expensive model verifies all N in one forward pass. When the draft is right you paid one big forward for N generations.BuildPython60 min↗ Official lesson