The Beta Distribution : Data Science Basics

Key Takeaways

what is up friends so today we're going to be talking about one of my favorite distributions in all of statistics and data science the beta distribution now truth is I didn't learn about this distribution until kind of later in my journey through data science and machine learning but once I did learn about it and I learned about the main application of the beta distribution it quickly became one of my probably like top three distributions in all of data science so if you agree after this video let me know in the comments or just tell me what your top three distributions and all of data science are but let's get right into it so the fun example today is that you just started a probability course so probability 101 and here's your super cool professor it's three days into the course so it's been Monday Tuesday and Wednesday and you've realized a very curious thing you've realized that your professor only wears red shoes or blue shoes to class so either red or blue and so this question starts forming in your mind of can I estimate can I well estimate the probability that the professor would wear red shoes on any given day so just a couple days in the course and based on some prior knowledge the best you can do right now for estimating this probability is basically just look at the number of times in these three days that the professor has worn red shoes which is just Monday and Tuesday twice divided by the total number of days that have gone by in the course so far which is again just these three days so right now your best guess which we call this sample proportion P hat R is equal to 2 divided by 3 or about 67 percent now a couple more days go by in the course and you learn about this thing called the error associated with the sample proportion you learned that this thing is called a point estimate just a single answer to that question of what's the probability the professor would wear red shoes but you can actually do better you can come up with a error a standard deviation around this point estimate and you learn that the formula to do so this might seem familiar to some of you from previous courses if not don't worry I'll explain the intuition here you learn that the standard deviation is the square root of this sampler proportion we just computed times 1 minus that sample proportion we just computed all divided by the number of samples that that proportion is based on which in this case is just three days so far the main intuition in this formula is really in the denominator so you see that as the denominator increases so right now n is 3 which is a pretty small number but let's say 100 days have gone by in the course that denominator would be 100 when the denominator of a fraction goes up the entire fraction goes down and in that case in this case the fraction is representing your error which makes total intuitive sense the more samples you have in calculating the sample proportion the lower your error should be and that totally all checks out but for now just observing three days you see your error associated with this estimate of 67 percent the error is about 27 so pretty big error which is great in your estimation of probability because now you can tell someone well my best guess is 67 but I'm going to have to say that comes with an error of plus or minus 27 so now whoever you tell this to has a much better understanding about what the situation is but that kind of gets your gears turning you say that hmm I've just learned that the main unit of probability in statistics is something called a distribution and I used to have a point estimate now I have this interval estimate can I do better can I get a full distribution for estimating this probability you're asking for a full distribution whose probabilities are going to help you have a best guess about a probability itself and that's why a lot of times you'll see this kind of Catchphrase about the beta distribution saying it's the probability of probabilities that's not a very helpful phrase but it is catchy but basically what it's trying to say is that the beta distribution is a distribution that helps us model probabilities that we don't know about it gives us the probability that a certain probability we're trying to estimate is at different values now that probably still seems extremely confusing because as I said it I got confused too and so let's continue on and see what that exactly means so what we're saying is that I want a distribution who's going to satisfy these three requirements crucially I need this distribution to only take positive mass or positive density between zero and one because this distribution that I would like to come up with is going to help me model a probability and since it's going to help me model a probability it doesn't make sense for this distribution to take any values less than zero or greater than one because probabilities can only be between zero and one so that's requirement number one for whatever distribution I'm going to use to model this probability the others are that I want the mean of this distribution so the center of mass of this distribution to be close to the sample proportion and the intuition behind this is that well the sample proportion should continue being my best guess if I had to give someone a single answer to this question and therefore I would like that to be close to the mean of this distribution at the end of the day similarly I would like the standard deviation of this distribution to be close to this empirical standard deviation this standard error of the sample proportion that I came up with here so I would like all this information I knew before to line up to be consistent with this distribution that I come up with now which is going to give me even more information about what this probability might be as the number of days in the course goes on and so around the middle of the course you finally learn about a distribution that meets all of these criteria and that is exactly the beta distribution so let's look at a couple examples of the beta distribution to understand how it actually works the beta distribution is characterized by two parameters Alpha and beta so Alpha is going to be consistently written in these red in red here so for example in this beta distribution here's Alpha and beta is equal to two now what do alpha and beta mean in the context of this problem and other problems that you would use the beta distribution for Alpha is related to the number of successes in your problem and beta is related to the number of failures in your problem now these successes and failures are a very generic term that help you apply this to lots of different use cases so you can find the right words in your domain in this case we're considering every time the professor wears red shoes as a success because that's the probability we're trying to model and we can consider every time the professor wears blue shoes which is the other possibility as are failures so they're not exactly successes and failures but you can see how they would match in the context of for example if we're using this to predict the probability that a team is going to win a basketball game in that case the success and failure terminology would just be more spot on but same idea here so more specifically Alpha is the number of successes you've observed so far plus one okay so Alpha is the number of successes plus one which makes sense why after three days Alpha is equal to three because as we said so far we observe the professor wearing red shoes two times and two plus one is equal to three similarly beta is the number of failures you've observed so far plus one so so far after three days we observed the professor wearing blue shoes one time one plus one equals two so that's how we got to we're going to model the distribution of the probability the professor would wear red shoes after three days as a beta 3 2 distribution now the big question is what does a beta 3 2 distribution look like well it looks like this so let me put a little bit more information here so on the left axis here we have 0 over here we have one and so we see that that first criteria we wanted to meet is met there is no mass of this distribution there's no density of this distribution on the left of zero or to the right of one so that first thing is met now we'll come back to whether our other two criteria about the mean being approximately equal to the sample proportion and the standard deviation being approximately equal to the standard error of the sample proportion we'll come to that on the right panel here but as a teaser for why things are looking good if we calculate the mode of this distribution so the mode of a distribution is just the peak the highest point in the distribution so that is where this gray dash line is the mode of this distribution is 0.67 which is actually the sample proportion itself so that makes us feel really warm and fuzzy inside that the most likely value of this distribution is actually the sample proportion is the empirical proportion of times that the professor has worn red shoes now let's say 10 days have gone by in the course so now let's say that the number of times the professor has worn red shoes is three therefore Alpha is equal to that plus one so Alpha is equal to four in this example let's say the number of times the professor wore blue shoes is seven which is ten minus three and therefore beta is going to be that number of failures seven plus one which is eight so now we have a beta for eight distribution and you see that that looks like this again makes us feel warm and fuzzy inside because the mode of this distribution is at 0.3 which is exactly the sampler proportion in this case of the number of times the professor wore red shoes now the other thing I want to note the other thing that should make us feel warm and fuzzy inside is that after three days our distribution looked like this after 10 days our distribution looks like this besides just the kind of General place the Peaks are at the other difference between these two distributions is that after 10 days the standard deviation has gotten a lot smaller you see that this distribution is a lot tighter around its peak than the distribution was after just three days and that totally lines up with everything we know so far we know that as more and more days go by in the course our distribution whatever it is should get Tighter and Tighter and Tighter around the most likely value because we're just more certain after 10 days than we are after three days about our estimate being correct and let's say now it's the last day of the course it's day 100 of the course you just took your final exam and you aced it and you found that your final reading on the number of times the professor wore red shoes is 35 therefore if we take Alpha as that plus one we get 36 and the final reading of the number of times Professor wore blue shoes is 65 so beta is 66 and we see that we have an even tighter distribution here so we see the mode of this distribution is again the sampler proportion of 0.35 and crucially we see that the standard deviation has gotten even tighter than after 10 days and if there was more days in the course let's say we were on day like 1000 of the course or uh even more than that you would see that it just gets Tighter and Tighter and Tighter so the beta distribution seems to hit a lot of the intuitive conditions we would require out of a probability distribution who is trying to measure probabilities themselves so hopefully it's more clear now what people mean when they say that the beta distribution is a probability of probabilities that again is kind of a useless catchphrase but it is cool to say and I hope that it's a little bit more clear what's going on now let's go into some of the mathematical properties of the beta distribution we kind of showed visually things make sense but mathematically are things sane the mean of the beta distribution which is different from the mode so the mode is the peak but if you look at this distribution for example the mean is going to be to the left of that because this distribution is skewed to the left which is going to pull that mean to the left of the mode the mean of the beta distribution is Alpha over Alpha plus beta so in this case it would be three divided by the sum of these two numbers which is 5 which is 0.6 which as we expected is to the left of 0.67 and so if we break down Alpha and beta in terms of the actual number of times the professor wore red and blue we get the alpha is red plus one again written down here red plus one and beta is blue plus one now even though this isn't exactly R over R plus b which is the mode we see that as the sample size gets larger and larger so as r and v get larger and larger the limit that this approaches is exactly going to be R over R plus b because these ones just become more irrelevant if you think back to your study of limits as the variables R and B get larger and larger these ones kind of they're influence vanishes and so the mean converges to the mode in that case looking at if the standard deviation also makes sense to us standard deviation in this case actually variance formula for the beta distribution looks a little complicated but we're going to break it down about the same way it's equal to Alpha times beta divided by quantity Alpha plus beta squared times quantity Alpha plus beta plus one what a mouthful but we're going to break it down the same way I'm going to take one term of alpha divided by one of these Alpha over betas that gives us Reds plus one over Reds plus Blues plus two I'm going to take the beta and take the other term of alpha plus beta which gives me Blues plus one divided by Reds plus Blues plus two and all that's remaining is a 1 divided by Alpha plus beta plus one by the way what is Alpha plus beta well that's going to be the number of Reds plus one plus the number of Blues plus one and there's another plus one here so we have three plus ones total so we get up three here and all that's remaining is Reds plus Blues in the denominator Reds plus Blues is just the total number of days that I've gone by because on any given day it's either red or blue so if I add up Reds and blues I get total number of days so I just put n here for Simplicity also because I was running out of space on the page so hopefully that's clear there now as complicated as this looks we can do the same limit trick as we did before so what does the variance approach as the Reds and blues and N gets bigger and bigger so the number of data points we observe gets larger and larger well all these constants cancel out again so this one becomes Irrelevant this two becomes irrelevant same same same and so what you're left with is red over red plus blue which is the sample proportion times blue over red plus blue which is 1 minus the sample proportion all divided by n if we look back to our understanding about the variance before we went into this whole beta distribution we saw that its sample proportion times one minus sample proportion all divided by n and we feel very good that that's exactly what we got here again this is the variance so you could just take the square root and it would look exactly like it did on the previous page and that is it folks that is the beta distribution in a nutshell the reason I think the beta distribution is so cool one of my top distributions in all of data science is that it is a distribution that helps us understand what value a probability itself might be and therefore it lends itself very well to the field of statistics and data science called Bayesian statistics it lends itself to Bayesian statistics because in Bayesian statistics we have some kind of prior of what we think a parameter might be for example in this problem before observing any data at all I would put equal probability that the probability of the professor wears red shoes on any given day is between zero and one just equally likely I have no idea I've never seen any data but as Bayesian statistics goes it takes this prior and as You observe more data about the world as three days go by as 10 days go by as hundred days go by you're able to update that prior in intuitive ways in order to reflect your best understanding based on a combination of that prior and Based on data you've observed in the real world that's why the beta distribution is just awesome in my opinion it really lends itself to the study of Bayesian statistics and understanding the study of uncertainty itself so hopefully you agree hopefully you think the beta distribution is pretty cool if you have any questions or comments at all please feel free to post them below thanks for watching this video like And subscribe for more just like this I will see you all next time