Statistical Power, Clearly Explained!!!

Key Takeaways

this clouds outside but who cares it's time for stat quest stat quest hello I'm Josh starburns welcome to stat quest today we're gonna talk about statistical power and it's gonna be clearly explained note this stat quest assumes that you are already familiar with p-values if not check out the quests this stat quest also assumes that you are already familiar with the normal distribution if not check out the quest let's start with two distributions the one on the Left represents the weights of mice on a special diet and the one on the right represents the weights of mice eating normal Mouse food since there's only a little bit of overlap it's pretty easy to see the difference between these two diets most of the mice on the special diet weigh less than the mice on the normal diet and if we collect a small set of measurements from the special diet and another small set of measurements from the normal diet and plot these points on a graph and compare their means then in this case we'll get a p-value equal to zero point zero zero zero four and this small p-value less than 0.05 would cause us to correctly reject the null hypothesis that both sets of data came from the same distribution in other words if this distribution which is somewhere between the special and normal diets said all data a tums from me then the small p-value would say in a loud and confident voice I reject your hypothesis if we repeated this experiment a bunch of times there's a high probability that each statistical test will correctly give us a small p-value in other words there is a high probability that the null hypothesis that all of the data came from the same distribution will be correctly rejected but every once in a while we will get something like this where the data overlap and when this happens we will get a large p-value greater than 0.05 and that means that even though we know the data came from two different distributions we cannot correctly reject the null hypothesis that all of the data comes from the same distribution so the large p-value says in a very small and meek voice dang I can't reject the null hypothesis that said because these distributions are so far apart and there is so little overlap the probability of correctly rejecting the null hypothesis is high power is the probability that we will correctly reject the null hypothesis alternatively you could say that power is the probability that we will correctly get a small p-value in this example because we have a high probability of correctly getting a small p-value and rejecting the null hypothesis we have a large amount of power BAM [Music] note if there was no difference between the special diet and the normal diet and they both shared the same distribution and we collected one set of measurements for mice on the special diet and one set of measurements for mice on the normal diet then the null hypothesis that both datasets came from the same distribution would be true in this case there is no such thing as correctly rejecting the null hypothesis so the concept of power the probability that we will correctly reject the null hypothesis doesn't apply in this situation contrast if this special diet wasn't very good at helping mice lose weight but it still made a difference then even though there is a lot of overlap we have two distinct distributions and that means power the probability that we correctly reject the null hypothesis applause if we were to weigh three mice on the special diet and three mice on the normal diet and plot these points on a graph and compare their averages then in this case we will get a p-value equal to 0.34 that means we will fail to reject the null hypothesis that both groups come from the same distribution this is a bummer because in this case we know the data comes from two different distributions and when we repeat this many times most of the time we will get a large p-value and fail to reject the null hypothesis however every once in a while we will get something like this where the data do not overlap and we will correctly get a small p-value when this happens even though the null hypothesis says all day today comes from me the small p-value will say in a loud and confident voice I reject the null hypothesis so out of all these tests this was the only one that gave us a p-value small enough that we correctly rejected the null hypothesis and that means when there is a lot of overlap between the two distributions and we have a small sample size we have a relatively low power medium BAM the good news is that we can always increase power by increasing the number of measurements we collect and a power analysis will tell us how many measurements we need to collect to have a good amount of power shameless self-promotion we'll talk more about how and why we can increase power in the stat quest on power analyses BAM in summary power is the probability that we will correctly reject the null hypothesis when we have two distributions that have very little overlap we will have a lot of power because there is a high probability that we will correctly reject the null hypothesis however when the two distributions overlap a lot and if we have a small sample size we will have a small amount of power however if we want more power we can increase the sample size lastly a power analysis will tell us how many measurements to collect to have a good amount of power double BAM hooray we've made it to the end of another exciting stat quest if you like this stat quest and want to see more please subscribe and if you want to support stack quest consider contributing to my patreon campaign becoming a channel member buying one or two of my original songs or a t-shirt or a hoodie or just donate the links are in the description below alright until next time quest on