Capstone Projects
Prove everything you learned. Build portfolio-grade systems.
85 lessons · 0 visual stories
0/85 complete · 0%
#LessonTypeLanguagesTimeOpen
01Capstone 01 — Terminal-Native Coding AgentBy 2026 the shape of a coding agent is settled. A TUI harness, a stateful plan, a sandboxed tool surface, a loop that plans, acts, observes, recovers. Claude Code, Cursor 3, and OpenCode all look the same from 50 feet. This capstone asks you to build one end to end — CLI in, pull request out — and measure it against mini-swe-agent and Live-SWE-agent on SWE-bench Pro. You will learn why the hard part is not the model call but the tool loop, the sandbox, and the cost ceiling on a 50-turn run.Capstone—35 min↗ Official lesson02Capstone 02 — RAG over Codebase (Cross-Repo Semantic Search)Every serious engineering org in 2026 runs an internal code search that understands meaning, not just strings. Sourcegraph Amp, Cursor's codebase answers, Augment's enterprise graph, Aider's repomap, Pinterest's internal MCP — same shape. Ingest many repos, parse with tree-sitter, embed function- and class-level chunks, hybrid-search, re-rank, answer with citations. This capstone asks you to build one that handles 2M lines of code across 10 repos and survives incremental re-indexing on every git push.Capstone—30 min↗ Official lesson03Capstone 03 — Real-Time Voice Assistant (ASR to LLM to TTS)A voice agent that feels right has end-to-end latency under 800ms, knows when you have stopped talking, handles barge-in, and can call a tool without stalling. Retell, Vapi, LiveKit Agents, and Pipecat all hit this bar in 2026. They do it with the same shape: a streaming ASR, a turn-detector, a streaming LLM, and a streaming TTS, all wired through WebRTC with aggressive latency budgets at every hop. Build one, measure WER and MOS and false-cutoff rate, and run it under packet loss.Capstone—30 min↗ Official lesson04Capstone 04 — Multimodal Document QA (Vision-First PDF, Tables, Charts)The 2026 document-QA frontier moved away from OCR-then-text and toward vision-first late interaction. ColPali, ColQwen2.5, and ColQwen3-omni treat each PDF page as an image, embed it with multi-vector late interaction, and let the query attend to patches directly. On financial 10-Ks, scientific papers, and handwritten notes this pattern beats OCR-first by a large margin. Build the pipeline end to end on 10k pages and publish the side-by-side against OCR-then-text.Capstone—30 min↗ Official lesson05Capstone 05 — Autonomous Research Agent (AI-Scientist Class)Sakana's AI-Scientist-v2 published full papers. Agent Laboratory ran the experiments. Allen AI shared traces. The 2026 shape is plan-execute-verify tree search over experiments, budgeted cost, sandboxed code execution, a vision-feedback LaTeX writer, and an automated NeurIPS-style reviewer ensemble. The capstone is to build one, run it end to end within $30 per paper, and survive the sandbox-escape red team that Sakana documented.Capstone—40 min↗ Official lesson06Capstone 06 — DevOps Troubleshooting Agent for KubernetesAWS's DevOps Agent went GA, Resolve AI published its K8s playbooks, NeuBird demoed semantic monitoring, and Metoro tied AI SRE to per-service SLOs. The production shape is settled: an alert webhook fires, an agent reads telemetry, walks a graph of K8s objects, ranks root-cause hypotheses, and posts a Slack brief with approval buttons. Read-only by default. Every remediation gated by a human. This capstone is that agent, evaluated on 20 synthetic incidents and compared against AWS's Agent on three shared cases.Capstone—30 min↗ Official lesson07Capstone 07 — End-to-End Fine-Tuning Pipeline (Data to SFT to DPO to Serve)An 8B model trained on your own data, DPO-aligned on your own preferences, quantized, speculative-decoded, and served at measurable $/1M tokens. The 2026 open stack is Axolotl v0.8, TRL 0.15, Unsloth for iteration, GPTQ/AWQ/GGUF for quantization, vLLM 0.7 with EAGLE-3 for serving. The capstone is to run the whole pipeline reproducibly — YAML in, served endpoint out — and publish a model card under the 2026 Model Openness Framework.Capstone—35 min↗ Official lesson08Capstone 08 — Production RAG Chatbot for a Regulated VerticalHarvey, Glean, Mendable, and LlamaCloud all run the same production shape in 2026. Ingest with docling or Unstructured and ColPali for visuals. Hybrid search. Re-rank with bge-reranker-v2-gemma. Synthesize with Claude Sonnet 4.7 using prompt caching at 60-80% hit rate. Guard with Llama Guard 4 and NeMo Guardrails. Watch with Langfuse and Phoenix. Grade with RAGAS on a 200-question golden set. Build one in a regulated domain (legal, clinical, insurance), and the capstone is passing the golden set, the red team, and the drift dashboard.Capstone—30 min↗ Official lesson09Capstone 09 — Code Migration Agent (Repo-Level Language / Runtime Upgrade)Amazon's MigrationBench (Java 8 to 17) and Google's App Engine Py2-to-Py3 migrator set the 2026 bar. Moderne's OpenRewrite does deterministic AST rewrites at scale. Grit targets the same problem with codemod-style DSL. The production pattern combines both: a deterministic substrate for safe rewrites plus an agent layer for the ambiguous cases, a sandbox for per-branch builds, and a test harness that flips green before the PR opens. The capstone is to migrate 50 real repos and publish a pass rate with a failure taxonomy.Capstone—30 min↗ Official lesson10Capstone 10 — Multi-Agent Software Engineering TeamThe 2026 shape of a multi-agent engineering team has converged: an architect plans, N coders work in parallel worktrees, a reviewer gates, a tester verifies. SWE-AF's factory architecture, MetaGPT's role-based prompting, AutoGen 0.4's typed actor graph, Cognition's Devin, and Factory's Droids all landed on it independently. Parallel worktrees convert wall-clock into throughput. Shared state and handoff protocols become the failure surface. The capstone is to build the team, evaluate on SWE-bench Pro, and report which handoffs break and how often.Capstone—40 min↗ Official lesson11Capstone 11 — LLM Observability & Eval DashboardLangfuse went open-core. Arize Phoenix published the 2026 GenAI semconv mappings. Helicone and Braintrust both doubled down on per-user cost attribution. Traceloop's OpenLLMetry became the de-facto SDK instrumentation. The production shape is ClickHouse for traces, Postgres for metadata, Next.js for UI, and a small army of eval jobs (DeepEval, RAGAS, LLM-judge) running over sampled traces. Build one self-hosted, ingest from at least four SDK families, and demonstrate catching an injected regression in under five minutes.Capstone—25 min↗ Official lesson12Capstone 12 — Video Understanding Pipeline (Scene, QA, Search)Twelve Labs productized Marengo + Pegasus. VideoDB shipped the CRUD-for-video API. AI2's Molmo 2 published open VLM checkpoints. Gemini long-context handles hours of video natively. TimeLens-100K defined temporal grounding at scale. The 2026 pipeline is settled: scene segmentation, per-scene caption + embedding, transcript alignment, multi-vector index, and a query that answers with (start, end) timestamps plus frame previews. The capstone is ingesting 100 hours, hitting public benchmarks, and measuring hallucination on counting and action questions.Capstone—30 min↗ Official lesson13Capstone 13: Stateless MCP Server with Registry and GovernanceProduction MCP is not one server process. It is a chain of contracts: publishable metadata, live discovery, a stateless request envelope, authorization, policy, audit, and deployment evidence.Capstone—25 min↗ Official lesson14Capstone 14 — Speculative-Decoding Inference ServerSpeculative decoding — a cheap draft proposes tokens, the target model verifies them in one pass — is now a production-ready optimization, not a research trick. EAGLE-3 in vLLM 0.7 ships 2.5-3x throughput on real traffic. P-EAGLE (AWS 2026) pushed parallel speculation even further. SGLang's SpecForge trained draft heads at scale. Red Hat's Speculators hub published aligned drafts for common open models. TensorRT-LLM made speculative decoding first-class on NVIDIA. The 2026 production serving stack is vLLM or SGLang with EAGLE-family drafts, FP8 or INT4 quantization, and HPA on queue-wait. This capstone is to serve two open models at 2.5x+ baseline throughput with a full tail-latency report.Capstone—30 min↗ Official lesson15Capstone 15 — Constitutional Safety Harness + Red-Team RangeAnthropic's Constitutional Classifiers, Meta's Llama Guard 4, Google's ShieldGemma-2, NVIDIA's Nemotron 3 Content Safety, and X-Guard for multilingual coverage defined the 2026 safety-classifier stack. garak, PyRIT, NVIDIA Aegis, and promptfoo became the standard adversarial evaluation tools. NeMo Guardrails v0.12 ties them into a production pipeline. This capstone wires all of it together: a layered safety harness around a target app, an autonomous red-team agent running 6+ attack families, and a constitutional self-critique run that produces a measurable harmlessness delta.Capstone—25 min↗ Official lesson16Capstone 16 — GitHub Issue-to-PR Autonomous AgentLabel an issue, get a PR — the 2026 product shape for autonomous coding agents: run an agent in a cloud sandbox, verify tests pass, and post a review-ready PR with rationale. AWS Remote SWE Agents, Cursor Background Agents, OpenAI Codex cloud, and Google Jules all ship it. The hard parts are reproducing the repo's build environment automatically, preventing credential leakage, enforcing per-repo budgets, and making sure the agent cannot force-push. This capstone builds the self-hosted version and compares it on cost and pass rate to the hosted alternatives.Capstone—30 min↗ Official lesson17Capstone 17 — Personal AI Tutor (Adaptive, Multimodal, with Memory)Khanmigo (Khan Academy), Duolingo Max, Google LearnLM / Gemini for Education, Quizlet Q-Chat, and Synthesis Tutor all shipped adaptive multimodal tutoring at scale in 2026. The common shape is a Socratic policy (never just dump the answer), a learner model that updates after every interaction (Bayesian knowledge tracing style), voice + text + photo-math input, curriculum graph retrieval, spaced-repetition scheduling, and hard safety filters for age-appropriate content. The capstone is to ship a subject-specific tutor (K-12 algebra or intro Python), run a two-week efficacy study with 10 learners, and pass a content-safety audit.Capstone—30 min↗ Official lesson20Agent Harness Loop ContractThe harness is the agent. The model is a coprocessor. This lesson freezes the loop contract you can wire any model into.BuildPython90 min↗ Official lesson21Tool Registry with Schema ValidationA tool the agent cannot validate is a tool the agent cannot call. Build the registry and the schema checker before you build the tools.BuildPython90 min↗ Official lesson22JSON-RPC 2.0 Over Newline-Delimited StdioThe transport between a model client and a tool server is JSON-RPC over stdio. Hand-rolling it once teaches you what every framing layer is paying for.BuildPython90 min↗ Official lesson23Function Call DispatcherThe dispatcher is where the harness pays for every promise the schema made. Timeouts, retries, dedupe, error mapping. All on one seam.BuildPython90 min↗ Official lesson24Plan-Execute Control FlowA plan that cannot survive a failure is a script. A script that can replan is an agent. Build the replanner first.BuildPython90 min↗ Official lesson25Capstone Lesson 25: Verification Gates and the Observation BudgetAn agent harness without a verification layer is a wish in a trenchcoat. This lesson builds the deterministic gate chain that decides whether a tool call is allowed to fire, how much of its output the agent is allowed to see, and when the loop has to stop because the agent has read too much. The chain is a function of small, named gates plus an observation ledger that tracks every token the model has been shown.Build—90 min↗ Official lesson26Capstone Lesson 26: Sandbox Runner with Denylist and Path JailThe verification gate decides whether a tool call should run. The sandbox decides what happens when it does. This lesson ships a subprocess runner that refuses dangerous executables, refuses dangerous argv shapes, jails every file path to a project root, truncates oversized output, and kills runaway processes on a wall-clock timeout. It is the second of two layers that sit between the model and the operating system.Build—90 min↗ Official lesson27Capstone Lesson 27: Eval Harness with Fixture TasksA coding agent is only as good as the suite of tasks you measure it against. This lesson builds an evaluation harness that takes a folder of fixture tasks, runs each through a candidate agent, scores pass or fail through a deterministic verifier, and aggregates the results into pass@1, pass@k, mean latency, and mean cost. The harness is the source of truth that lets you tell a regression from a refactor.Build—90 min↗ Official lesson28Capstone Lesson 28: Observability with OTel GenAI Spans and Prometheus MetricsAn agent harness without observability is a black box that costs money. This lesson hand-rolls a span builder that emits records compliant with the OpenTelemetry GenAI semantic conventions, writes them to a JSON-Lines file one span per line, and exposes counters and histograms in Prometheus text format. The whole thing is stdlib Python and runs offline.Build—90 min↗ Official lesson29Capstone Lesson 29: End-to-End Coding Agent on the HarnessTrack A's payoff. This lesson stitches the gate chain, the sandbox, the eval harness, and the OTel spans into one working coding agent that fixes a real (small, fixture-scale) bug in a multi-file Python project. The agent is a deterministic policy, not an LLM; the substitution makes the lesson reproducible and shows that the harness was the interesting part all along. The contract is identical: a real model plugs in at the policy seam.Build—90 min↗ Official lesson30BPE Tokenizer From ScratchBytes in, ids out, ids back to the same bytes. Build the tokenizer that every modern text model still starts from.BuildPython90 min↗ Official lesson31Tokenized Dataset with Sliding WindowA pretraining run is a function from token ids to gradients. This lesson builds the conveyor that feeds the ids in.BuildPython90 min↗ Official lesson32Token and Positional EmbeddingsIds are integers. The model wants vectors. Two lookup tables sit between them, and the choice of the positional one shapes what the model can learn.BuildPython90 min↗ Official lesson33Multi-Head Self-AttentionOne linear projection, three views, H parallel heads, one mask. The attention block as the model actually uses it.BuildPython90 min↗ Official lesson34Transformer Block from ScratchOne block is the unit of every modern decoder LLM. Layer norm, multi head attention, residual, MLP, residual. The pre-LN variant trains stably without warmup. The post-LN variant is what the original paper shipped. This lesson builds both, side by side, and shows which one survives a 12 layer stack at common learning rates.BuildPython90 min↗ Official lesson35GPT Model AssemblyTwelve blocks stacked, a token embedding, a learned position embedding, a final LayerNorm, and a tied language model head. That is the entire 124 million parameter GPT model. This lesson assembles those pieces into a working class, counts the parameters to confirm the model matches the reference 124M shape, and generates text with multinomial sampling, temperature, and top-k.BuildPython90 min↗ Official lesson36Training Loop and EvaluationA loop that does not measure is a loop that lies. This lesson builds the training loop that drives the GPT model: AdamW with weight decay split, a warmup plus cosine learning rate schedule, a `calc_loss_batch` helper, an `evaluate_model` pass on held out data, a `generate_and_print_sample` qualitative probe every K steps, and a JSONL log of losses you can plot after. The same skeleton trains every decoder LLM you will ever build.BuildPython90 min↗ Official lesson37Loading Pretrained WeightsTraining a 124 million parameter model from scratch is a budget decision; loading a published checkpoint is a Tuesday. This lesson loads pretrained GPT-2 style weights from a safetensors file into the exact architecture from lesson 35, walks the parameter name mapping piece by piece, and sanity generates a continuation to prove the load worked. No network, no third party loaders, no opaque magic.BuildPython90 min↗ Official lesson38Capstone Lesson 38: Classifier Fine-Tuning by Head SwapTrack B's first capstone. A pretrained language model is a stack of self-attention blocks ending in a token-prediction head. When you want spam vs ham, the head is wrong but the body is mostly right. This lesson rips the head off, glues a two-class linear layer onto the pooled representation, and trains the classifier two different ways: final-layer only, and full fine-tuning. The eval is precision, recall, and F1 on a held-out split. You learn what each strategy buys you and what it costs.Build—90 min↗ Official lesson39Capstone Lesson 39: Instruction Tuning by Supervised Fine-TuningA pretrained base model can extend a sequence but cannot follow an instruction. Supervised fine-tuning is the smallest change that fixes this: feed the model paired examples of an instruction and a desired response, and train the body to predict the response tokens. The trick is that you only want the loss to count the response, not the instruction. This lesson builds an Alpaca-style SFT loop with a custom collate function that masks instruction tokens with `ignore_index=-100`, trains on 200 instruction-response pairs, and evaluates on a held-out split using exact-match.Build—90 min↗ Official lesson40Capstone Lesson 40: Direct Preference Optimization from ScratchReward models and PPO are the classical RLHF stack. DPO collapses that stack into a single supervised loss that fits a policy directly against preference pairs. This lesson derives the DPO loss from the reward-difference identity, ships a working reference model plus policy model, computes per-token log-probabilities, and trains a tiny transformer on a preference fixture of chosen and rejected completions. Tests pin the loss math and the gradient direction so you know the implementation matches the paper.Build—90 min↗ Official lesson41Capstone Lesson 41: Full Evaluation PipelineTraining is the part you can monitor with loss curves. Evaluation is the part you have to design. This lesson builds a unified eval pipeline that takes any trained language model, runs four heterogeneous evals on it, aggregates the results into a per-task report, and ships a local mock LLM-as-judge so the loop runs without a network. The four evals cover the dimensions every shipping model needs: language modelling (perplexity), short-form correctness (exact-match), open-form similarity (token F1), and qualitative scoring (judge).Build—90 min↗ Official lesson42Large Corpus DownloaderTraining a language model begins long before the first forward pass. The corpus has to land on disk, decompressed, deduplicated, and addressable, with the resume story already worked out before the network drops at 4 percent. This lesson builds a streaming downloader that pulls compressed shards, decompresses on the fly with Zstandard, fingerprints near-duplicates via MinHash plus locality-sensitive hashing, and writes a shard manifest the rest of the pipeline can trust.BuildPython90 min↗ Official lesson43HDF5 Tokenized CorpusThe downloaded corpus has to land in a layout the trainer can stream from at line speed. JSONL on disk does not survive 16 dataloader workers. HDF5 with a resizable, chunked integer dataset does. This lesson builds streaming tokenization into a resizable HDF5 dataset, sharded write across multiple files, memory-mapped read at training time, and a sliding-window dataloader that produces fixed-length sequences with the right packing.BuildPython90 min↗ Official lesson44Cosine LR with Linear WarmupThe learning-rate schedule is the second most important decision after the loss function. AdamW with a cosine decay and a linear warmup is the modern default for language-model training because it lets the model see a small effective step size during the brittle first thousand updates, ramps up to a configured peak, and decays smoothly back toward zero. This lesson builds that schedule, plots the curve over training steps, logs gradient norms next to the schedule, and proves the schedule honors warmup, peak, and decay boundaries.BuildPython90 min↗ Official lesson45Gradient Clipping and Mixed PrecisionThe optimizer and schedule from the previous lesson assume gradients are sane. They usually are not. A single bad batch can spike the gradient norm by three orders of magnitude. Mixed-precision training amplifies this by introducing FP16 overflow on the loss side. This lesson builds the two safety belts that production training cannot ship without: gradient clipping to a configured global L2 norm, and a mixed-precision loop with autocast and GradScaler that detects NaN and Inf, skips the step cleanly, and logs the scaling factor for forensics.BuildPython90 min↗ Official lesson46Gradient AccumulationTrain at an effective batch you cannot afford, one micro-batch at a time. Scale the loss, hold the optimizer step, and let the gradients pile up.BuildPython90 min↗ Official lesson47Checkpoint Save and ResumeTrain interrupts kill runs; checkpoints let them continue. Save model, optimizer, scheduler, loss history, step counter, and RNG state, atomically, so a kill at any moment leaves a valid file on disk.BuildPython90 min↗ Official lesson48Distributed Data Parallel and FSDP from ScratchMulti-rank training is two collectives and one rule. Broadcast the parameters at startup, average the gradients after backward, never let the ranks disagree about what step they are on.BuildPython90 min↗ Official lesson49Language Model Evaluation HarnessA model that does well on a task you cannot define is a model that does well by accident. The harness is the task definition, the metric, the runner, and the leaderboard, in one short, swappable shape.BuildPython90 min↗ Official lesson50Hypothesis GeneratorA research agent that asks the same question twice is wasting tokens. The trick is forcing each draft to land somewhere new.BuildPython90 min↗ Official lesson51Literature RetrievalA hypothesis is cheap. Knowing whether someone already proved it is the expensive part. Build the retrieval layer that answers that question before the runner spins up a sandbox.BuildPython90 min↗ Official lesson52Experiment RunnerThe loop is only as honest as its measurements. Build the runner that takes a spec, executes it in a sandboxed subprocess, and emits a json metrics blob the evaluator can trust.BuildPython90 min↗ Official lesson53Result EvaluatorThe runner produced numbers. The evaluator decides whether those numbers are an improvement, a regression, or noise. Build the verdict path that turns metrics into a one line conclusion.BuildPython90 min↗ Official lesson54Paper WriterA LaTeX skeleton is a contract between the researcher and the typesetter. If the contract is broken the document does not compile, and the failure is loud. Build the skeleton first, then fill it.BuildPython90 min↗ Official lesson55Critic LoopA critic that returns "looks good" the first time is broken. A critic that always returns "needs work" is broken. The interesting critic is the one that converges, and you have to engineer convergence.BuildPython90 min↗ Official lesson56Iteration SchedulerA research loop without a scheduler is a queue with delusions. The scheduler is where the loop decides what to stop exploring, and that decision is the whole game.BuildPython90 min↗ Official lesson57End-to-End Research DemoA demo is the place where every contract you wrote earlier has to compose. If any one of them leaks, the demo is the lesson that catches it.BuildPython90 min↗ Official lesson58Vision Encoder PatchesA vision model that reads pixels needs a tokenizer for pixels. Patch embedding is that tokenizer. Cut the image into a grid of squares, flatten each square, project it through one linear layer, then add a 2D position signal so the transformer knows where each square sat in the original image.BuildPython90 min↗ Official lesson59Vision Transformer EncoderPatches alone do not see. A 12-layer pre-LN transformer with 12 attention heads turns the sequence of patch tokens into a sequence of contextual tokens, with the CLS token pooling whole-image features in its final hidden state. This lesson is the engine room of every modern vision-language model.BuildPython90 min↗ Official lesson60Projection Layer for Modality AlignmentA vision encoder produces image tokens. A text decoder consumes text tokens. The two live in different vector spaces. A small two-layer MLP projects image tokens into the text embedding space, and a cosine alignment loss against a paired caption pulls the two spaces into agreement. That projection is the smallest piece of a vision-language model and the one that matters most for transfer.BuildPython90 min↗ Official lesson61Cross-Attention FusionThe projection layer aligns one image vector with one caption vector. A real vision-language decoder needs every text token to attend to every patch token, so the model can ground each word in a region. Cross-attention is how that grounding happens. The text queries; the vision keys and values answer. This lesson builds the cross-attention block, the causal text self-attention, and the mask shapes that keep both legal.BuildPython90 min↗ Official lesson62Vision-Language PretrainingThe encoder, projection, and decoder are wired. Now train them together. Two objectives drive learning: a contrastive image-text loss (InfoNCE) that pulls matching pairs together in the joint embedding space, and a language modeling loss that asks the decoder to caption each image. Combined, they teach the network both to find the right image for a caption and to write a caption for the image.BuildPython90 min↗ Official lesson63Multimodal EvaluationTraining is half the loop. The other half is measurement. This lesson builds three evaluation surfaces from primitives: image-caption retrieval reported as R@1, R@5, R@10; visual question answering reported as exact match accuracy; and image captioning reported as BLEU-4. Each metric is a function over the model's outputs and a synthetic eval suite that runs in seconds.BuildPython90 min↗ Official lesson64Chunking Strategies, ComparedChunking decides what your retriever can ever surface. Get the boundaries wrong and no embedding model, no reranker, no LLM can repair the damage downstream.BuildPython90 min↗ Official lesson65Hybrid Retrieval with BM25 and Dense EmbeddingsLexical and semantic retrieval fail on opposite query distributions. Hybrid retrieval with reciprocal rank fusion does not interpolate, it votes - and the vote wins on every query class.BuildPython90 min↗ Official lesson66Cross-Encoder RerankerA bi-encoder embeds query and document independently. A cross-encoder concatenates them and reads both at once. The cross-encoder is the smartest reader and the slowest. Used as a second stage on the bi-encoder's top-k, it pays for itself.BuildPython90 min↗ Official lesson67Query Rewriting: HyDE, Multi-Query, and DecompositionThe query the user types is not the query your retriever wants. Rewriting bridges the gap before retrieval, so the index sees something closer to what the answer looks like.BuildPython90 min↗ Official lesson68RAG Evaluation: Precision, Recall, MRR, nDCG, Faithfulness, Answer RelevanceIf you cannot grade your retrieval and your answer at the same time, you cannot ship the system. The two are not the same metric and the same prompt fails on different axes.BuildPython90 min↗ Official lesson69End-to-End RAG SystemSix lessons of components. One pipeline. One eval loop. One self-terminating demo. This is the system you ship.BuildPython90 min↗ Official lesson70Task Spec FormatAn eval harness is only as good as the contract its tasks honour. Freeze the JSONL shape and the metric vocabulary before you write a single scoring function.BuildPython90 min↗ Official lesson71Classical MetricsBLEU, ROUGE-L, F1, exact-match, accuracy. Five metrics that still account for most published LLM eval numbers. Implement each from first principles so you know what the number means.BuildPython90 min↗ Official lesson72Code Exec MetricGenerated code is right when it passes the tests. The eval harness has to extract code, run it without crashing the host, and tally pass-rates honestly. This lesson builds that surface.BuildPython90 min↗ Official lesson73Perplexity and CalibrationIf your model says 90 percent confident on a thousand answers and gets six hundred right, it is not well calibrated. Calibration is half of trustworthy eval. The other half is perplexity, which tells you whether the model thinks the held-out text is plausible at all.BuildPython90 min↗ Official lesson74Leaderboard AggregationPer-task scores are easy. Per-model rankings across heterogeneous tasks are harder. Statistical significance on a thousand-prediction leaderboard is the part everyone skips. This lesson does not skip it.BuildPython90 min↗ Official lesson75End-to-End Eval RunnerFive lessons of plumbing, one lesson to glue them. The runner reads the task spec from lesson 70, calls a model through an adapter, scores with lessons 71 and 72, attaches the calibration report from lesson 73, and emits the leaderboard from lesson 74. Demo self-terminates.BuildPython90 min↗ Official lesson76Collective Ops From ScratchThe four collective operations that hold distributed training together are allreduce, broadcast, allgather, and reduce_scatter. Every other primitive a training framework offers is a wrapper around these. Build them once over a `multiprocessing.Queue` mesh, verify them against a reference implementation, and the rest of the track becomes plumbing.BuildPython90 min↗ Official lesson77Data Parallel DDP From ScratchDistributedDataParallel is a hook on top of allreduce. Wrap a model, broadcast the initial parameters from rank 0 so every rank starts identical, install a backward hook on every parameter that issues an allreduce of the gradient, and the rest is gradient descent. The whole pattern is 200 lines.BuildPython90 min↗ Official lesson78ZeRO Optimizer State ShardingAdam stores two moment estimates per parameter, both in float32. A 7B-parameter model carries 56 GB of optimiser state. ZeRO stage 1 shards that across N ranks; each rank owns 1/N of the optimiser. After the local step the updated parameter shards broadcast back, every rank reconstructs the full model, and the next step begins. The win is a linear memory drop on the largest single allocation in the training stack.BuildPython90 min↗ Official lesson79Pipeline Parallel and Bubble AnalysisTensor parallelism splits the matrix multiply across ranks. Pipeline parallelism splits the model across ranks, one stage per rank. Microbatches flow through the pipeline. The empty time at the start and end is the bubble; minimising it is the whole craft.BuildPython90 min↗ Official lesson80Sharded Checkpoint and Atomic ResumeA 70B-parameter training job is paused by a node failure every few hours. The checkpoint format decides whether you lose 30 minutes or 30 hours. A sharded checkpoint writes every rank's shard in parallel and records ownership in a manifest. Resume loads each rank's shard from its own file, reconstructs the state on the same world size, and the optimiser steps as if nothing happened. Atomic write keeps a half-finished checkpoint from poisoning the next resume.BuildPython90 min↗ Official lesson81End-to-End Distributed TrainingLessons 76 through 80 each built one piece. This is the assembly: a tiny GPT trained across 4 simulated ranks with DDP for gradient sync, ZeRO-1 for optimiser-state sharding, and a sharded checkpoint at the halfway mark. The demo runs 20 steps, self-terminates, prints a loss curve plus a memory profile, and writes a resumable checkpoint.BuildPython90 min↗ Official lesson82Capstone 82 — Jailbreak TaxonomyA safety harness without a taxonomy is a coin flip. Name the attack before you defend it.BuildPython90 min↗ Official lesson83Capstone 83 — Prompt Injection DetectorA detector is a function from prompt to confidence and category. Anything else is a vibe.BuildPython90 min↗ Official lesson84Capstone 84 — Refusal EvaluationHelpfulness on benign prompts and refusal on harmful prompts are two metrics, not one. Measure both.BuildPython90 min↗ Official lesson85Capstone 85 — Content Classifier IntegrationClassifiers on the output side answer a different question than rules on the input side. Both need a policy router.BuildPython90 min↗ Official lesson86Capstone 86 — Constitutional Rules EngineA rule is a name, a predicate, and an explanation. Anything missing one of those three is a vibe, not a rule.BuildPython, YAML90 min↗ Official lesson87Capstone 87 — End-to-End Safety GatePre-gen, during-gen, post-gen. Three checkpoints, one verdict, an audit trail per request.BuildPython90 min↗ Official lesson