R Tutorial: Designing an Experiment - Power Analysis

Want to learn more? Take the full course at https://learn.datacamp.com/courses/ab-testing-in-r at your own pace. More than a video, you'll learn hands-on coding & quickly apply skills to your daily work. --- Now that we have a good sense of our baseline numbers we're ready to design our experiment. Here we'll use our knowledge of seasonality along with power analysis to figure out how long we need to run our experiment. In preparing our experiment we learned about historical conversion rates. On average conversion rates are about 28%, but that can change throughout the year. What does this mean for building our experiment? Well, it would be bad to run the control condition in August and the test condition September, because the control may look better simply due to seasonality, not because it's actually a better condition. This is why A/B experiments try to run both conditions simultaneously, to ensure both conditions are exposed to similar seasonal variables. We also need to consider seasonal effects for knowing how we expect our control condition to perform. If the experiment is run in January we expect the control to have a conversion rate of roughly 20%, but if it's run in August the control should be closer to 50%. With this knowledge, we use a power analysis to determine how long we should run our experiment. Experiment length is one of the big questions in A/B testing. If you stop too soon you may not get enough data to see an effect. Too long and you may waste valuable resources on a failed experiment. One way to safeguard against this is with a power analysis. A power analysis will tell you how many data points (or your sample size) that you need to be sure an effect is real. Once you have your sample size, you can figure out how long you will need to run the experiment to get your number of required data points. This will depend on variables such as how many websites hits you get per day. Running a power analysis is also good because it makes you