Reinforcement Learning
Agents that learn by doing. The foundation of RLHF.
12 lessons · 0 visual stories
0/12 complete · 0%
#LessonTypeLanguagesTimeOpen
01MDPs, States, Actions & RewardsA Markov Decision Process is five things: states, actions, transitions, rewards, a discount. Everything in RL — Q-learning, PPO, DPO, GRPO — optimizes over this shape. Learn it once, read the rest of reinforcement learning for free.LearnPython45 min↗ Official lesson02Dynamic Programming — Policy Iteration & Value IterationDynamic programming is RL with cheating. You already know the transition and reward functions; you just iterate the Bellman equation until `V` or `π` stops moving. It is the benchmark every sampling-based method tries to approach.BuildPython75 min↗ Official lesson03Monte Carlo Methods — Learning from Complete EpisodesDynamic programming needs a model. Monte Carlo needs nothing but episodes. Run the policy, watch the returns, average them. The simplest idea in RL — and the one that unlocks everything downstream.BuildPython75 min↗ Official lesson04Temporal Difference — Q-Learning & SARSAMonte Carlo waits until the episode ends. TD updates after every step by bootstrapping the next value estimate. Q-learning is off-policy and optimistic; SARSA is on-policy and cautious. Both are one line of code. Both underpin every deep-RL method in this phase.BuildPython75 min↗ Official lesson05Deep Q-Networks (DQN)2013: Mnih trained one Q-learning network on raw pixels, beat every classical RL agent on seven Atari games. 2015: extended to 49 games, published in Nature, sparked the deep-RL era. DQN is Q-learning plus three tricks that make function approximation stable.BuildPython75 min↗ Official lesson06Policy Gradient — REINFORCE from ScratchStop estimating value. Parameterize the policy directly, compute the gradient of expected return, step uphill. Williams (1992) wrote it in one theorem. It is why PPO, GRPO, and every LLM RL loop exist.BuildPython75 min↗ Official lesson07Actor-Critic — A2C and A3CREINFORCE is noisy. Add a critic that learns `V̂(s)`, subtract it from the return, and you get an advantage that has the same expectation but far lower variance. That is actor-critic. A2C runs it synchronously; A3C runs it across threads. Both are the mental model for every modern deep-RL method.BuildPython75 min↗ Official lesson08Proximal Policy Optimization (PPO)A2C throws away each rollout after one update. PPO wraps the policy gradient in a clipped importance ratio so you can do 10+ epochs on the same data without the policy exploding. Schulman et al. (2017). Still the default policy-gradient algorithm in 2026.BuildPython75 min↗ Official lesson09Reward Modeling & RLHFHumans cannot write a reward function for "good assistant response," but they can compare two responses and pick the better one. Fit a reward model to those comparisons, then RL the language model against it. Christiano 2017. InstructGPT 2022. The recipe that turned GPT-3 into ChatGPT. In 2026 it is mostly being replaced by DPO — but the mental model stays.BuildPython45 min↗ Official lesson10Multi-Agent RLSingle-agent RL assumes the environment is stationary. Put two learning agents in the same world and that assumption breaks: each agent is part of the other's environment, and both are changing. Multi-agent RL is the set of tricks to make learning converge when the Markov assumption no longer holds.BuildPython45 min↗ Official lesson11Sim-to-Real TransferA policy trained in a simulator that fails on hardware is a policy that memorized the simulator. Domain randomization, domain adaptation, and system identification are the three tools to make learned controllers cross the reality gap.LearnPython45 min↗ Official lesson12RL for Games — AlphaZero, MuZero, and the LLM-Reasoning Era1992: TD-Gammon beat human champions at backgammon with pure TD. 2016: AlphaGo beat Lee Sedol. 2017: AlphaZero dominated chess, shogi, and Go from scratch. 2024: DeepSeek-R1 proved the same recipe, with GRPO replacing PPO, works on reasoning. Games are the benchmark that drives every breakthrough in this phase.BuildPython75 min↗ Official lesson