This course is about the algorithms and techniques that drive experimentation in online platforms. At many firms, thousands of experiments are run daily to gather the vast amount of data needed to enable data-driven decision making. For any one of these experiments, an experimenter faces the challenge of deciding what data to collect, how long to run the experiment, and how to glean insights from the data collected. As a result, there is an increased demand by practitioners for methods and algorithms that deliver statistically sound results faster and with less opportunity cost. This course develops a modern toolbox of experimentation, with a focus on adaptive experimental design. Rooted in classical statistical and machine learning techniques, AED decides what future data to collect based on past measurements in a closed loop. Due to both theoretical gains and empirical success, AED has quickly become one of the most commonly employed algorithmic paradigms in practice with a promise of cutting experimentation time by up to half. However, practitioners who employ AED blindly can easily bias their results, or potentially not collect the data needed to make any useful inferences.
Students who take this course will learn how to a) design choose and develop adaptive experiments in practice, b) analyze the data coming from these experiments, c) conduct research in cutting-edge directions in AED.
Topics include:
Class sessions will be a mix of lecture and discussion of papers. There will be homeworks focused on implementing algorithms.
Class Session: MW 1-2:30, Paccar Hall 490
Class website: lalitjain.com/aed MyPlan link: https://myplan.uw.edu/course/#/courses/MKTG596
Class Notes: Link
Homeworks All homeworks should be completed in groups. You have 3 weeks to complete each homework. No late assignment will be accepted without prior consent from the Professor.
Projects Project drafts are due June 1st. Final versions are due June 9th. All projects should
| Date | Lecture | Topics | Resources |
|---|---|---|---|
| Mar 27 | Pillar 1 | A/B Testing, Lower bounds, SPRT | Sequential Analysis |
| Mar 29 | Pillar 2 | Multi-armed bandits | |
| Apr 3 | Pillar 2 Continued | Multi-Armed Bandits | Bandit Algorithms |
| Apr 5 | Pillar 3 | Martingales, MSPRT | Peeking at A/B Tests, Time-uniform Chernoff bounds via nonnegative supermartingales, |
| Apr 10 | Pillar 3 Continued | Anytime Confidence Intervals/Least Squares | Spotify Blog Post |
| Apr 12 | NO CLASS | ||
| Apr 17 | Linear Bandits | Optimism and OFUL | Chapter 19,20 of the Bandit Book |
| Apr 19 | Bayesian Methods | Thompson Sampling, Langevin Dynamics | Learning to Optimize via Posterior Sampling, Chapters 35-36 of the Bandit Book |
| Apr 24 | Continue Thompson Sampling | Thompson Sampling | Colab Link |
| Apr 26 | Posterior Sampling and Uncertainy Quantification | ||
| May 1 | Contextual Bandits | Policy Based vs Model Based | Handwritten Notes, Chapter 18 of Bandit Book |
| May 3 | Contextual Bandits | $\tau$-Greedy | Notes by Alekh and Sham, Taming the Monster, Instance Dependent Bounds |
| May 8 | Contextual Bandits | SquareCB | Beyond UCB, Bypassing the Monster, Online Learning Notes, Prediction, Learning and Games Chapter 2 |
| May 10 | Off-Policy Evaluation | IPS and Friends | Course Notes, Recsys 2021 Tutorial |
| May 15 | Guest Lecture | ||
| May 17 | NO CLASS | ||
| May 22 | Non-Stationarity | ||
| May 24 | Non-Stationarity | ||
| May 29/31 | Group Meetings |