Attention Variants — Sliding Window, Sparse, Differential
Full attention is a circle. Every token sees every token, and memory pays the price. Four variants bend the shape of the circle and recover half the cost.
Visual edition planned
This lesson isn’t interactive yet.
It is part of the curriculum and will get the same treatment as Phase 1 — a visual cover, hands-on labs, derivations with numeric checks and a quiz. Until then, the original lesson is the best place to read it:
Part of Phase 07 — Transformers Deep Dive. Use the previous / next cards below to keep browsing the phase.