Probability Density Functions - EXPLAINED!

Key Takeaways

hello everyone and welcome to another episode of Code Emporium where we're going to talk about probability density functions and its relationship with continuous random variables fun topic so let's get started with some math so to kind of motivate this entire discussion I want to tie kind of these Arcane Concepts in mathematics to something that's easy to understand in the real world so let's conduct a simple experiment now this experiment is a simple one which involves people watching so take a pen and paper go out to the main street in front of your house and then just watch people pass by and every time someone passes by you you want to just write some information about them so the first thing you would write is the color of their hair and the second thing you would write is the approximate height that you think they are just by eyeballing them so for example somebody now crosses you so you write down one person ID let's say that they had brown hair and the height of this person was 165 centimeters great so now some time passes by and the second person now crosses you and this person had black hair and this person was 160 centimeters approximately in height and now let's say that we document this for about 10 minutes and that will conclude our experiment and so what we end up with is a list of some observations and some data and where there is data we can now ask some interesting questions so for example one question that we can ask is how many people did we actually see during the duration of our experiment and maybe another question that's pretty straightforward from this observation list is what is the average height of people that passed you and for some of these you know we have pretty straightforward answers I'm going to introduce some mathematical notation right now and say let's say how many people actually walked past you will represent the outcome with some variable Omega one and this is going to be I don't know maybe 10 people past us so Omega 1 is 10 people and let's also say that I want to represent this question with a random variable we're going to call it X1 and random variables are essentially functions that take in the outcome of an experiment and map it to some measurable quantity so it's a function so we'll write it in some functional notation here taking in the outcome of the experiment which is Omega 1 and we're going to map it to an integer which is 10 because 10 people passed us in much similar way we can also write the answer to this question what is the average height of people who passed you so let's say that this is now the outcome is represented with Omega 2 and let's say in this specific experiment uh we saw that the average height was 165.32 centimeters and so if we represent this question with another random variable which we call X2 it is a function that takes in the outcome of the experiment and it will map it to some measurable quantity so let's say that it is 165.32 so this is a cool thing with random variables in general random variables are functions that allow you to take real world experiments and actually map them to a mathematical number of a quantity and because now that we're dealing with numbers we can perform some fun mathematics with these so these are simple questions but we can now answer more complex questions too like you know this is the output or outcome for one experiment but you know how do we conducted the experiment differently or maybe you know in another world we we might not have seen the average height be 165.32 centimeters and so I want to kind of ask more complicated questions with regards to what could have been possible or probable and so let's actually talk about a specific question now as I clear the screen so now let's say that for our third question we want to ask around probability so what is the probability that the average person's height across the entire experiment would have been over 170 centimeters now we already have of some random variable that really can already help out here so the random variable we defined was X2 that is the height of people or the average height of all the people in a single experiment we call that with a random variable X2 and you know it is a function so it takes in the outcome this Omega 2 is this outcome of a single experiment but you know we could have conducted this experiment in many ways and there could have been many possible outcomes for this um like one of the possible outcomes was that the average height of all the people in our 10 minute experiment could have been instead of 165.32 it could have been 165 or 164 centimeters or 161 centimeters or 171 centimeters so many values in fact let's write out a few of those values so something that we saw was you know let's say it's 165 centimeters so because random variables map the outcome to a actual measurable quantity I'm only writing numbers here so it could have been 165 it could have been 166. it could have been 167 and so on but that said it could have also been you know between these two numbers of 165 and 166. it could have been 165.1 165.2 165.3 these could have been all possible outcomes of the experiment that we conducted but what we actually got was something like 165.32 but it could have been even like you know some other small number too like 165.33 and so on so what's very apparent from this list is that there is an infinite number of values that could be present in this list and because we can also get values between you know any two numbers that you see you could always find a value that's in between them for example in between these two values we can get 165.3278472 and this is kind of why this entire set is an uncountable set it is not a subset of like natural numbers for example and because of that the random variable X2 we call this random variable a continuous random variable to learn more specifically about continuous random variables I have an entire video on discrete and continuous random variables so I do highly recommend you check that out now to answer this kind of question up here we might actually find it easier if we can somehow graph this out into a probability distribution function so let's actually try creating a probability distribution function with the random variable X2 knowing that it is a continuous random variable now before moving forward I wanted to give a quick shout out to the sponsor of this video Coursera so videos like this are very mathematical and I can't include all the mathematical detail as I would I talk about how this math is useful in machine learning but if you do want to know more about the rigorous mathematics itself I do recommend certain courses on Coursera so for example you can check out the mathematics for machine learning course by Imperial College of London where they have three courses in a specialization they offer on top of that you can also check out an advanced statistics course by John Hopkins University and I'll link all of these in the description down below of really interesting courses that I think are worth it if you're not interested in just the mathematics and you want to go towards more in the Practical machine learning and deep learning space with some integration of mathematics I have also listed some recommended courses down in the description below it helps you because you get a seven day free trial with a amazing knowledge and it also helps me because I do get some Kickback and it supports me on this channel and helps me make more videos like this so if you please do like these videos please do consider checking out the courses down in the description I know you won't regret it and with that let's get back to the video in this section we are going to see why we need a probability density function to begin with so let's start the discussion by drawing an x-axis and then we have a y-axis so the x-axis is going to be X2 for our random variable and it's going to contain the Heights in centimeters so I'm just going to write the integer values over here of 165 166 and 167. and this goes on and let's say that the probability distribution I'm just going to represent it with rectangular bars as we would for like a discrete random variable which is like a count right so maybe for 165 we have a bar that looks something like this for 166 we have a bar that looks like this and for 167 we have a bar that probably looks like this now the y-axis over here represents a probability but this overall graph is actually not a great representation for X2 because the X2 here clearly contains it can take on values that are just not present in this x-axis for example what if it was 165.32 like we actually observed but we don't really have a bar for that and in fact even if we were to divide each of these rectangular bars vertically you're not really going to be able to come to a situation where X2 has all of the values that it can take and the only way you can do that is if each of these bars is infinitesimally small in width and if we do make these bars infinitesimally small in width then you'll see that it follows a very smooth curve so let's actually draw this out and so the graph might look something like this where the x-axis is the variable X2 and the y-axis is some probability here now in in this graph on the left side If This Were A continuous random variable let's say that this was the correct representation we often talk about probability mass and probability density so this here let's say that if we wanted the 166 this bar over here let me change the color to to Yellow this bar over here would represent a mass in fact the area of this would represent a mass and since each of these units is one you could say that well this length the width is one but the height is some probability and if the total area is a mass we kind of represent this y-axis with probability Mass notation so that's a small P subscript random variable right and this would be a probability Mass function or a graphical representation of a probability Mass function but we already know that this doesn't really hold well because the values are not everything that X2 should be now each of these bars here is a an area that we can just say is like a mass in physics so what that means is that you can take this bar put it on a weighing scale and whatever weight you get is equivalent to like the area that this bar is but now if you squish this to an infinitesimally small width well you end up with a situation where let's say let me change the color again here to Yellow you end up with a situation where the bar is just going to be so infinitesimally small and because of that the area is going to be well it's going to converge to the value zero and because of this the bar over here for a continuous random variable will have a mass of zero and so talking about probability mass for a continuous random variable isn't very helpful but what can be helpful instead is instead of looking at specific point wise estimations we would look at the probability estimations across an interval so by an interval let's say we shade this region over here now the interval is going to be the width of this new bar you can consider this as like a stack of tiny infinitesimal bars and well if you put all these bars on a weighing scale we will get like a weight that is corresponding to the the mass of this area that's shaded out but here the interval here is like some volume so this is going to be like I don't know this is let's say that this width over here is a volume and what that means is well in probability in physics theory mass is equal to well mass is a product of density and volume and density is the division of like mass divided by volume so if the area is considered like a mass and the the width is considered which is an interval is considered as a volume then that leaves this x-axis part to be labeled as a density and so we kind of name it instead of in the same way so that is represented by a small f with a subscript S2 to show that this here is a probability density function and this is kind of why we talk about probability density functions with respect to continuous random variables because they can take values clearly which are greater than or equal to zero so I hope this analogy of probability mass for discrete random variables works well and the probability density function for continuous random variables makes more sense and I hope it also makes sense of why we need the probability density function in the continuous case now with this intuition out of the way I want to look at the properties of these probability density functions and continuous random variables but from a little more mathematical perspective so let's get to it so the first property that we're going to look at is that the probability mass of a continuous random variable X2 is going to be 0 at any point now to explain why this is the case mathematically I want to make use of a concept called the cumulative probability distribution function so it's going to be represented as P of X2 less than or equal to X and this is the cumulative probability distribution and if you were to graph it out it would be the area under the probability density function that is less than or equal to this value of x so if you want to now represent probability Mass probability mass is going to be well let's say at a specific point let me represent it with notation p x 2 is probability mass at the specific point x this is going to be equal to well it's going to be the cumulative probability of x 2 is less than or equal to x minus the cumulative probability that X2 is less than or equal to some value that's just slightly less than x so I'm going to call that small value let's say Delta X so if I just write it out just like this Delta X could literally be anything it's just a Delta after all but we know that with continuous random variables we need to make sure that this Delta X is honestly an infinitesimally small value and so I want to find the value of this entire function when Delta X becomes an infinite intestimally small value as it becomes closer and closer to zero and this is why we introduce the fun concept of limits so limit as Delta X becomes zero but from the positive direction so I put a little plus as um a superscript here now if we do some substitutions over here with applying the limit then you will see that the probability Mass function of X2 at the point x is going to be zero because substituting Delta X for zero these two terms cancel out and you are left with this final form for all values of X and this is the mathematical representation of exactly what this statement says and I hope this is clear how it links to our image that we saw previously over here where well it makes sense because if we were to just draw a bar that looks like this the probability mass for this specific point is going to be zero because the area of that bar is or it converges to the value of zero as the interval becomes infinitesimally small intense towards zero so graphically and mathematically I hope property one makes sense now let's move on to the second property and this states that the probability density function is a value that is greater than or equal to zero now in order to prove this out mathematically let's restate the the definition of density and density is going to be Mass over volume but from a mathematics standpoint it's going to be the area under probability curve divided by the interval let's now write all of this in mathematical notation so that's F of X2 the value of x is going to be equal to well so probably the mass we already defined in the previous section so let's write that out in terms of the cumulative probability distribution function and we're going to divide this by the interval itself and this interval is going to be of infinitesimally small value which in this case is going to be Delta X and because Delta X is an infinitesimally small value that will almost see go to zero and we want to find the value of this entire function as it convert as Delta X converges to zero we are going to use the concept of limits and for any math nerd out there we all know that this limit with um this fraction over here is the fundamental definition of a derivative and so what we can say is that f x 2 of X is going to be equal to the derivative of the cumulative distribution function for x and I'm going to box this now what this implies over here is that well we already know that the cumulative distribution function this is going to be a value that's greater than or equal to zero itself and because of that the derivative of something that is some positive or zero or positive value is also going to be overall greater than or equal to zero and hence the probability density of X2 is going to be greater than or equal to 0 for any value that it takes so mathematically I hope that this is sound now let's move on to property three property three is that the probability density of X2 at every point is going to sum to 1. to prove this property out now let's take the result of what we found from the previous section which is that the probability density function f of x 2 for the value X is the derivative with respect to x for the cumulative probability distribution function now let's integrate this value on both sides with respect to X so let me write that out in steps and then we'll end up with well the integral of f x 2 for X DX is equal to P of X2 is less than or equal to X and because this cumulative distribution function is from negative Infinity to X we write that as the bounds of this integral as well now that we have this equation what we want to do is well let's take X to be Infinity and then we can substitute it in this equation and because this is a cumulative distribution function over all possible values of X2 this is itself going to be equal to 1. and this right here is the mathematical statement of our property above so I hope now that the idea of an experiment how it's related to continuous random variables and how and why we use probability density functions along with their properties makes much more sense now so I've actually listed all of this out in a accompanying blog posts along with some applications in the field of machine learning so if you're interested please do check the description down below for my medium article and please do give me a follow on medium and also here on YouTube and I appreciate your support thank you all so much for watching and I'll see you in the next one bye-bye