Reinforcement Learning Foundations

What you’ll be able to do

1. LO1Analyse the exploration–exploitation trade-off, and implement and compare ε-greedy, UCB and Thompson sampling by their regret.
2. LO2Estimate value functions from experience with Monte Carlo, TD(0), n-step and TD(λ) methods, explain their bias–variance trade-offs, and evaluate one policy from another's data with importance sampling.
3. LO3Derive, implement and compare model-free control methods (SARSA, Q-learning, Expected SARSA and Double Q-learning), including on- against off-policy learning, reward shaping and the handling of termination and truncation.
4. LO4Apply value-function approximation, explain the deadly triad, and build a deep Q-network with experience replay and a target network that learns a control task on one CPU core.
5. LO5Derive the policy-gradient theorem, implement REINFORCE with baselines and actor-critic methods, and explain how trust regions and PPO's clipped objective keep policy updates safe.
6. LO6Plan with learned models (Dyna, Monte Carlo tree search), and learn from fixed data with offline RL and imitation learning (behaviour cloning, DAgger), explaining extrapolation error and compounding errors.
7. LO7Design and report RL experiments honestly: seeded runs, held-out start states, checkpoints chosen on validation episodes, standard errors and robust summaries, and recognise when a problem needs RL at all.

Chapters

6 chapters · each section ends with a short check

1. 01Bandits: learning to choose under uncertainty
2. 02Learning values from experience: Monte Carlo and temporal-difference prediction
3. 03Model-free control: SARSA, Q-learning and their relatives
4. 04Function approximation and deep Q-networks
5. 05Policy gradients and actor-critic methods
6. 06Learning with models, logged data and experts

Graded labs

1. L1

   Bandit algorithms and an honest comparison

   After chapter 1 · about 5 hours · Seeded Bernoulli bandit instances · Own work, CC BY 4.0 · Runs on a CPU

2. L2

   Prediction and control in grid worlds

   After chapter 3 · about 6 hours · Five-state random walk, the A/B batch example, 4 x 12 cliff walking and the maximisation-bias MDP · Course own work, CC BY 4.0 · Runs on a CPU

3. L3

   A deep Q-network for cart-pole on one core

   After chapter 4 · about 6 hours · Cart-pole environment · Course own work, CC BY 4.0 · Runs on a CPU

4. L4

   Policy gradients and actor-critic on cart-pole

   After chapter 5 · about 5 hours · Cart-pole · Course own work, CC BY 4.0 · Runs on a CPU

5. L5

   Learning from a model, from logs and from an expert

   After chapter 6 · about 5 hours · Dyna maze · Course own work, CC BY 4.0 · Runs on a CPU

Before you start

How it’s taught

You read short sections written for the page. Each ends with a check, and in-browser exercises let you practise. Graded labs run on your own cloud lab machine, CPU-friendly.

No videos. No live classes. No AI tutor.

How it works→