Lalit Jain

MKTG 596: Adaptive Experimentation in Practice

This course is about the algorithms and techniques that drive experimentation in online platforms. At many firms, thousands of experiments are run daily to gather the vast amount of data needed to enable data-driven decision making. For any one of these experiments, an experimenter faces the challenge of deciding what data to collect, how long to run the experiment, and how to glean insights from the data collected. As a result, there is an increased demand by practitioners for methods and algorithms that deliver statistically sound results faster and with less opportunity cost. This course develops a modern toolbox of experimentation, with a focus on adaptive experimental design. Rooted in classical statistical and machine learning techniques, AED decides what future data to collect based on past measurements in a closed loop. Due to both theoretical gains and empirical success, AED has quickly become one of the most commonly employed algorithmic paradigms in practice with a promise of cutting experimentation time by up to half. However, practitioners who employ AED blindly can easily bias their results, or potentially not collect the data needed to make any useful inferences.

Students who take this course will learn how to a) design choose and develop adaptive experiments in practice, b) analyze the data coming from these experiments, c) conduct research in cutting-edge directions in AED.

Topics include:

Class sessions will be a mix of lecture and discussion of papers. There will be homeworks focused on implementing algorithms.

Class Session: MW 1-2:30, Paccar Hall 490

Class website: MyPlan link:

Class Notes: Link

Homeworks All homeworks should be completed in groups. You have 3 weeks to complete each homework. No late assignment will be accepted without prior consent from the Professor.

Projects Project drafts are due June 1st. Final versions are due June 9th. All projects should

Date Lecture Topics Resources
Mar 27 Pillar 1 A/B Testing, Lower bounds, SPRT Sequential Analysis
Mar 29 Pillar 2 Multi-armed bandits  
Apr 3 Pillar 2 Continued Multi-Armed Bandits Bandit Algorithms
Apr 5 Pillar 3 Martingales, MSPRT Peeking at A/B Tests, Time-uniform Chernoff bounds via nonnegative supermartingales,
Apr 10 Pillar 3 Continued Anytime Confidence Intervals/Least Squares Spotify Blog Post
Apr 12 NO CLASS    
Apr 17 Linear Bandits Optimism and OFUL Chapter 19,20 of the Bandit Book
Apr 19 Bayesian Methods Thompson Sampling, Langevin Dynamics Learning to Optimize via Posterior Sampling, Chapters 35-36 of the Bandit Book
Apr 24 Continue Thompson Sampling Thompson Sampling Colab Link
Apr 26 Posterior Sampling and Uncertainy Quantification    
May 1 Contextual Bandits Policy Based vs Model Based Handwritten Notes, Chapter 18 of Bandit Book
May 3 Contextual Bandits $\tau$-Greedy Notes by Alekh and Sham, Taming the Monster, Instance Dependent Bounds
May 8 Contextual Bandits SquareCB Beyond UCB, Bypassing the Monster, Online Learning Notes, Prediction, Learning and Games Chapter 2
May 10 Off-Policy Evaluation IPS and Friends Course Notes, Recsys 2021 Tutorial
May 15 Guest Lecture    
May 17 NO CLASS    
May 22 Non-Stationarity    
May 24 Non-Stationarity    
May 29/31 Group Meetings