Customer AI Deployment
Discover a customer workflow, reduce its riskiest assumptions, and carry a useful AI system through measurement and rollout.
#LessonPhaseTimeRequirementOpen
01Define the Outcome Before You Choose the OutputFast implementation increases the penalty for choosing the wrong problem. Shape the outcome first so speed points in the right direction.Phase 1460 minRequired↗ Official lesson02Discover the Workflow People Actually PerformRequirements are not waiting in a meeting to be collected. They are scattered across actions, workarounds, records, and disagreements.Phase 1470 minRequired↗ Official lesson03Map Assumptions and Resolve the Riskiest One FirstA roadmap hides uncertainty inside features. An assumption map exposes what must be true before those features deserve to exist.Phase 1465 minRequired↗ Official lesson04Choose the Smallest Slice That Can Change the DecisionSmall is useful only when it proves something important. A tiny build that cannot change the next decision is merely incomplete.Phase 1465 minRequired↗ Official lesson05Write Specifications That Preserve JudgmentA useful specification fixes invariants and evidence while leaving reversible implementation choices open. It is a decision boundary, not a screenplay.Phase 1475 minRequired↗ Official lesson06RAG (Retrieval-Augmented Generation)Your LLM knows everything up to its training cutoff. It knows nothing about your company's docs, your codebase, or last week's meeting notes. RAG solves this by retrieving relevant documents and stuffing them into the prompt. It's the most deployed pattern in production AI. If you build one thing from this course, build a RAG pipeline.Phase 1190 minRequired↗ Official lesson07Evaluation & Testing LLM ApplicationsYou would never deploy a web app without tests. You would never ship a database migration without a rollback plan. But right now, most teams ship LLM applications by reading 10 outputs and saying "yeah, looks good." That is not evaluation. That is hope. Hope is not an engineering practice. Every prompt change, every model swap, every temperature tweak changes your output distribution in ways you cannot predict by reading a handful of examples. Evaluation is the only thing standing between your application and silent degradation.Phase 1145 minRequired↗ Official lesson08Building a Production LLM ApplicationYou have built prompts, embeddings, RAG pipelines, function calling, caching layers, and guardrails. Separately. In isolation. Like practicing guitar scales without ever playing a song. This lesson is the song. You will wire every component from Lessons 01-12 into a single production-ready service. Not a toy. Not a demo. A system that handles real traffic, fails gracefully, streams tokens, tracks costs, and survives its first 10,000 users.Phase 11120 minRequired↗ Official lesson09Design Success Metrics Before the Result ExistsMeasurement should answer a decision, not decorate a dashboard. Start with the goal, derive questions, then choose the smallest metrics that answer them.Phase 1470 minRequired↗ Official lesson10Choose Prototype, Pilot, or Production DeliberatelyThese are different learning environments, not levels of polish. Choose the stage that answers the current unknown with the least unnecessary consequence.Phase 1470 minRequired↗ Official lesson11Shadow Traffic, Canary Rollout, and Progressive Deployment for LLMsLLM rollouts combine the hardest parts of software deployment: no unit tests, diffuse failure modes, delayed signals. The sequence is (1) shadow mode — duplicate prod requests to candidate model, log, compare with zero user impact; catches obvious distribution issues but is not a quality guarantee; (2) canary rollout — progressive traffic shift 10% → 25% → 50% → 75% → 100% with gates at each step; track latency percentiles, cost/request, error/refusal rate, output length distribution, user-feedback rate; (3) A/B testing for distinct alternatives after stability confirmed. Non-determinism is irreducible — up to 15% accuracy variation across runs with identical inputs due to GPU FP non-associativity plus batch-size variance. Cost is a variable, not constant — a 20% better model can be 3x more expensive per call. Rollback speed is decisive: if rollback requires redeploy, you are too slow. Policy lives in config/flags; model lives in registry with pinned digests; rollback = flip policy + revert threshold + pin old model in seconds.Phase 1760 minRequired↗ Official lesson12Build a Feedback Ratchet with Ownership and RetirementShipping closes one build loop and opens the learning loop. Evidence must change the system or it becomes telemetry nobody owns.Phase 1475 minRequired↗ Official lesson