Statistics & Probability for Data Science | Data Science Tutorial | Edureka Rewind

Key Takeaways

a very good afternoon all of you so welcome to the webinar on statistics and probability for data science my name is Vidya and I am going to be teaching you or giving you a very high level information about statistics and probability that is required for data science so before we get started I would like you all to go through the edura master class page where we have all our members who are registered in this and this is a place where you will be getting some latest courses that are available on our Technologies as well as about the webinar information so let's get started statistics and probability for data science so we we're going to talk about probability and introduction to probability that is required for data science and also a introduction to statistics that is required for data science I'm sure we've all covered about probability in our school as well as in our College days right what is probability it's basically a measure of How likely an event will occur so what is that suppose I'm asking you to find out a probability of if I'm going to toss a coin what is the probability that we are going to get the head right so what are the total outcomes that is possible so we have two choices right we have two options one is called the head option and the other one is called the tail option so among this head and tail so they become part of our denominator so that means these are the sample spaces that we have the total options that we have when you toss a coin are two so what is the probability of getting a head you only have two choices head and a tail so the probability of getting a head is one so that means it is 1 by two so this is how we work on probability the next is let's understand some terminologies that is used in probability so we have something called sample space the one I just mentioned what are all the uh possible outcomes that you can get when you toss a coin you have only two possible outcomes called head and so that is your sample space what is what is your sample space when you're going to toss a dice so what are the options you have you have six options so your denominator is going to be six that is your sample what is random experiment an experiment or a process for which the outcome cannot be predicted with Clarity so you cannot tell what is going to happen or what would be the number when you're going to toss a coin or when you're going to put a dies right so that is called a random experiment a event a event is nothing but one or more outcomes of the experiment so if I say what is the probability of getting a five so getting a five is basically event that is going to happen among all the sample spaces I have I have six sample spaces for a coin and getting the the chance of getting a five is one so the CH the probability is 1 by six so that is a event so let's move to our next slide what are the different types of probabilities we have a couple of types of probabilities called as joint probability and disjoint probability joint probabilities are certain The Joint events are events which has something in common so I'm going to say a student who can get 100 marks in statistics and 100 marks in probability can this event can this event happen at this same time for a particular student yes it can happen that is called as joint probability another example I can give you is if I'm asking you to find uh uh in a pack of in a deck of cards if I'm asking you to get a a and number four red and a four so can you get a red and a four on the same card yes you can get so that is called as joint probability what is disjoint these two two things can never happen at the same time I want a card to be a and a four that will never happen a card can either be a A or it can be a four so this is not possible so these are called as disjoint events so we just discussed about the different types of probabilities we do have um something called as probability distribution so what is probability distribution we have something called as probability density function it is actually called as PDF or also called it as a pmf we have normal distribution and we also have Central limit theorem so let's get uh into it and see what these mean so what is this uh probability or called as a PDF so what is this PDF probability distribution function so here so this is going to be this is our graph this is our normal curve so what is normal curve I'll explain in our next slide this is our graph right so here in this normal curve what is the area bounded by the density function and x-axis is equal to one so the probability that a random variable assumes a value between a and b and is equal to the area under the PDF bounded by A and B so here what is this range of data what is this data here what is the data between this particular range for example let's assume that I'm going to give you um this graph is plotted for the marks of the students we have a student who's got 95 Marks here I have a student who's got say 20 Marks here the least is 20 the highest is 95 and here we were going to have marks let's say in the range of 45 to 75 so if you see the majority of the people are lying in this average range so if I ask you to find out this particular range of data A to B that is called as a probability density function the higher values of 95 or 90 or any higher values will be on the rightmost side and lower values people who are getting least marks will also be on the uh Lower Side with minimal population the amount the maximum number of people are going to be in the average range so this is actually called as a normal distribution curve okay so what is probability uh normal distribution the one I just mentioned is our normal distribution so here we going to say the random variable X so the probability distribution that Associates some variable a variable That Is Random with a cumulative probability the formula for our normal distribution is 1 divided by Sigma multiplied by square otk of 2 piun * e ^ of x - mu by 2 Sigma squ the formula may sound um slightly confusing but let me explain what these terms mean here so here we have our x x is our random variable what is Mu for any data I need to find out something called mu mu is going to be the mean next is Sigma Sigma is my standard deviation I'm going to give you an example of mean and uh standard deviation in our upcoming slide so for any data so where I give you an example of the person's Mark in the range of 90 uh 20 to 95 we do have other students whose marks are plotted out here some students Mark is here a student has got Marks here a few more students have got Marks here and so on so now this is our me so this is your average Mark for example let's say the average Mark is 65 average Mark is 65 then there are students in the range of let's say 45 to 75 what is this Sigma here Sigma indicates standard deviation how far is the data away from the mean let's say we have a student who's got 75 how far is this 75 away from the mean it is 10 units away from the mean if it is going to be 10 units here it has to be 10 units here as well that means this is going to be a range of say 50 5 to 75 so this is Sigma Sigma means standard deviation that is one standard deviation away from the mean so this entire range is one standard deviation away from the mean so we do have two standard deviation and three standard deviation as well so you go two standard deviation on the right and two standard deviation on the left we're going to get the next value called as Mu + 2 Sigma so this is Mu Min - 2 Sigma because in a normal distribution curve we expect it to be we assume that the values are going to be symmetric so the both sides the left side and the right side would be symmetric so here we have mu + 2 Sigma here it is Mu minus 2 Sigma how much is this particular value away from the mean how much is this particular value away from the mean that is called as Mu + 2 Sigma or mu+ Sigma how much much deviation do we have from your average value right so for a normal random um variable we usually assume that the mean is going to be at zero and variance is equal to 1 so that is why we have mu+ Sigma or mu+ 2 Sigma and the other values so here we have the mean which is going to determine the location of the center of the graph and the standard deviation determines the height of the graph so how far is the standard deviation do we have too much deviation in our data or do we have minimal deviation in our data so if you look at this both are normal distribution curve both are normally distributed having the same both are symmetric if your standard deviation is large the curve kinds of kind of gets elongated suppose our mean here is say 10 and I have a standard deviation of let's say four units so that means I'm going to have 14 on this side and six on this side so this is how my data is going to be but if my uh deviation standard deviation is small I'm going to have 10 here and I have only two standard deviation so that means I'm talking about a range of 8 to 12 my Curve will look taller and narrower right so these are some of the um measures of um central tendency so we will see that in the upcoming slides and we have something called as Central limit theorem so what is this Central limit theorem before we move into Central limit theorem I need to explain some more terms on statistics so let me complete that and get back to Central limit theorem um we have we have probability different types of probability is marginal probability joint probability and conditional probability joint probability is something we've already seen so let's see what is marginal probability and conditional probability marginal probability is our basic probability so it is a probability of occurrence of a single event so here we are checking uh what is the probability of getting this particular card it is three and heart so we have a total of 52 we have a deck of 52 cards and what is the probability that we are going to get this value so we going to express it as probability of a is Sigma I equal to 1 2 k p of x i so if I'm very particular I want a three and a heart then it is going to be 1 by 52 if we want only a heart here then it is going to be 4 by 52 right the next is joint probability so joint probability is a measure of two events happening at the same time so here we actually want a card uh the probability that a card is an Ace and a heart so how many cards are going to be ace and a heart so we have a total of 13 um heart cards and out of which one is Ace so it is going to be 1 by 52 what is conditional probability so conditional probability means a probability of an event is depending on event that has already happened so let me give you an example if I ask what is the probability of play suppose if I'm asking what is the probability that you're going to go out and play now the answer is 1 by two because you you may play or you may not play so the probability is 1 by two so this conditional probability the value or the probability value increases based on a prior event that has happened what is that the probability we are try basically trying to increase the chances of my probability so I will say p of play I have some prior event I already know something is already already happened right I'm going to say rain so this is p of play Given rain right so the given rain means it is currently raining what is the probability that you're going to play outside so now probably we will not say 50 50% chances we probably may say 20% chance that we can actually play this is 20 is just a random number I'm trying to take a smaller value so that is called as conditional probability so here I'm going to say p of B given a so given a means a event has already happened based on that tell me what is the probability of event B happening so if I have to link this with your joint probability we are going to say a joint probability B divided by P of a so this is how we are going to Define um a p of B given a all right so I I just we just discussed about probability and then we spoke about joint probability then we spoke about conditional probability and we have something called base theorem base theorem was found by a mathematician and uh called base and hence the name called base theorem this base theorem uses this concept of conditional probability so here I'm saying what is the chance or what is the probability of a happening given B so B event has already happened what is the chance what is the probability of event a occurring so this is actually called as posterior which means the probability of occurrence of a given B this is what we are trying to find out for this the formula the base theorem formula is probability of a divided by probability of B so we know what is probability of event a separate what is the probability of event B separately multiplied by P of B given a so this P of g b given a is referred to as likelihood ratio which measures the probability of B given a so we are just reversing this values out here multiplied by P of a p of a is nothing but the actual probability of getting a a value divided by probability of B so why do we need probability it is going to help us optimize our model loot of our algorithms uh mainly when you move into supervised or machine learning algorithms they do use probability and when you're going to uh calculate some laws or some how your model is performing how how is these or how are these machine learning um uh performance uh how are the machine learning models performing then we need probability okay all like I said all our models are built out on probability okay guys so now that I've just given you a brief introduction about probability let's move on to statistics so what is statistics this is an area of Applied Mathematics concerned with data collection analysis interpretation and presentation statistics is a really important in terms of machine learning or our data science right so uh how are you going to to check because we need statistics because let me explain that so we have something called population right so there is something called population and there's something called sample for example I want um let's see what's the example given here okay suppose I want to find out what is the average age a person would live in India average age for all people living in India I want to find out what could be the average age do you think it is is possible for us to go calculate everyone's gather everyone's age and say uh take all the age divided by the total number of people in India and find out the age by then maybe we would have lost two people and probably 10 new babies were born so the moment you complete this calculation the data is already changed the population that means every individual you have the data is already changed so whatever you average you have actually derived is basically not correct so it is impossible for us to perform something on population so population is something a collection or a set of individuals or objects or events whose properties needs to be analyzed so we cannot work on a population so what are we going to do we're going to take some sample of data so probably we will pick some um sample of people across all the states not everyone we're going to pick some sample of people across all the states to understand their lifestyle to understand the age to understand how healthy they are to understand if there is any problem in the environment which is causing giving them some health issues right that is why the age they are they not they are average uh age is not 70 and it is only 65 in that particular State we're going to collect samples of data so sample is basically a subset of my population probably I'm going to do some random sampling so instead of taking everyone from here I will probably do this I'm going to say pick this person I'm going to pick this I'm going to pick this and I'm going to pick this one thing before I do a sample I should ensure that this sample that I picked is a representative of my population I just can't go pick people from just this area and say this is my sample right maybe these people belong to one state or maybe this is a different state of data that has no relation with this data so my data the way I need to pick up is I need to ensure that I it is a a sample is a representative of my population so we have different types of sampling so let's get directly into the sampling first one is called as random sampling so in random sampling you're just going to pick some people at random and this becomes your random sample what is systematic sample you're going to say pick every second row of data or pick every fourth person so this is my fourth person this is a fourth person fourth person fourth person and so on this is called as systematic sampling all right next is stratified sampling so stratified sampling we will what we here do here is the subset of population should share at least one common characteristics so we have a group of people here we are going to take some characteristics as common so let's consider gender so we've already separated the male data set and a female data set pick a few data points from the male subset pick a few data from the female subset and that becomes our stratified sampling so we do have some different types of Statistics called as descriptive statistics and inferential statistics what is descriptive statistics like I already told you we already have the data with us we already have some data with us how are you going to transfer or uh understand the properties of those population so I already have so much of data points what what are the properties of these data points how is this data distributed how is what is the average age of these people a lot of statistical inferences we can get out of this data that is called as descriptive statistics so what is the next one then so inferential statistics so inferential statistics is what we will be using mostly in our data science because we will not have the entire population of data so this is your population you're going to get only sample so you're going to work with multiple sample data sets so what do I mean by multiple sample data sets here this inference means can I infer something about this population using this particular sample I'm trying to let's say the average age of um people living in India is let's say 73 73 years can my sample get me this age which is closer to 73 maybe my sample will not give me 73 it says let's say 71.8 right or it can even say 74.2 it may not be closer to uh it's not exactly 73 but am I able to get some range which is actually closer to my population mean this is my mean population mean and this is my sample mean am I anywhere closer to this particular mean is what we are going to infer that is called as INF differential statistics so here I'm going to uh draw a conclusion about my population from my sample okay so we need we we need to do a lot of tests and understand the relation and perform some statistical tests to arrive at this particular value so here uh the descriptive statistics we have two things called as measures of central tendency and measures of variability what is measures of central tendency we have mean median and and mode we are talking about the population data that you already have for this particular data I'm going to calculate my mean median and mode how are you going to calculate your mean median and mode we will just see that with a example next is measures of variability we can also call this as measures of dispersion how is the data spread we've already found the mean the median and the mode how is the data spread so now my data is spread like this right how is my spread of data so my data is basically spreading from this side to this side or my data is just so compact but my spread is narrow so my range is only so much so this is called the spread of data so we do have certain things called range interquartile range variance and standard deviations that is the measures that is used to understand the spread of data so here I'm sure you all already know what is uh how do you calculate the mean of a value right mean is basically average so in this example we are just taking this particular column called HP this is called as a car's data set and here we've taken the HP value horsepower 110 110 93 96 how do you calculate the average add all these values divided by the total number of rows which is eight and you get the average of 103 this is called as the mean how do we find out the median for median we need to find out it is basically the measure of central value one thing you have to remember in median is the data should be sorted either in ascending or in descending order here we have taken the miles per gallage uh M miles per gallon uh for the car so we have we have 21 21 21.3 23 and so on first sort the data in ascending or descending order and take the middle value if you have odd number of values then the middle value will be your median if you have even number of values in our case we have eight values then you have to take the middle representative the two values here you're going to find the average of those two so add these two values and divided by two that would be the median what is mode mode is nothing but the frequency how many times is this particular value occurring in cylinder if you see we have six cylinder and we have four cylinder what is occurring more times six cylinder is occurring more times so we going to say six is our mode the next comes to measures of spread what is range range is basically the maximum value minus the minimum value the example I gave you I had a student whose the lowest Mark was 20 and we had the highest Mark is 95 the range is nothing but 95 minus 20 which is 75 so we have data spread across 75 points or 75 units that's called as spread of data what is mode mode is nothing but frequency mode is nothing but frequency suppose I have um let's say these are my marks okay 95 93 95 94 okay now if you look at these are the marks of five students okay which is which Mark is occurring more times three students have got 95 this is your this Mark 95 is your mode okay median I'll show you the previous slide let me show you the previous slide yeah so median is basically the central value okay so now if you see you have some values here 21 21 21.3 and so on right if this is odd number of values let's say I'm going to give you something like this 1 2 3 4 five I have five data points the middle value there are five data points right just take the middle value this is your median if I give you even number of values I'm giving you six data points then you need to take the middle two values and find the average so the average is 3 + 4 / 2 which is say 3.5 so let's see something called as inter quartile range what is inter quartile range here I am going to divide my data into five different units so what I mean here is I have some data points here right I have data points here this is my minimum value this is my maximum value I'm sure you know how to find out minimum and maximum and you also know what is your median value median is that 50 percentile right apart from this I'm also going to find out something called as 25th percentile and 75th percentile how many people let's assume that these are the marks say this is zero this is say 100 maximum Mark so this is zero this is 100 I have 50 I have 25 and 75 so this is called as interquartile range that middle 50% of the data so you'll have 50% of data in this range you have 25% here and you have 25% here clear so this is called as inter quartile range I'm going to divide it into quarters say five sets of values I have here so this is 0 to 25 this is 25 to 50 This Is 50 to 75 this is 75 to 100 how many people have got above 75 percentage of marks how many people or how many students have got less than 25 percentile so this is where we use the interquartile range so that is your interquartile range the next is standard deviation so what is standard uh deviation I told you it is the spread of data so variance is first uh let's look at variance here we have how much is our data differing from your expected value I wanted a age of say 70 3 I got a value of let's say 72 how much is this 72 differing from this value or how is your data spread is called as variance and standard deviation variance is nothing but it is called as s squ if I do a square root of this then I'm going to get my standard deviation this is my actual value minus xar so this xar is my 73 and this is the value that I got divided by your total number of data points I need to do this for every data point let's say I have 100 data points I need to repeat this for 100 data points and square root this or let's say this is your variance and I'm going to square root my variance to get standard deviation okay so here I have my variance and square root of that variance is called as your standard deviation so it's a measure of dispers version of a set of data from its mean how is the spread of data so here I have some uh uh example to calculate my standard deviation and your mean so uh you know how to calculate the mean right we have a list of numbers here add all the numbers divide by the count you're going to get 20 which is to 20 data points add all these values and divided by 20 you're going to get a mean value of seven seven is the mean or the average of all these values all right so standard deviation I need to apply it to the formula what formula do we need to apply XI I already know XI is going to be seven right so I'm going to do this calculation for each of my data point so these are my data points here mu is s right XI is 9 and mu is 7even so I need to do this for each of the data point so for this data point we have 9 - 7 then I have 2 - 7 then I have 5 - 7 4 - 7 and so on I need to square all these terms I'm going to get some value here I need to add all that value and find the average so this is how I'm going to calculate my standard deviation all right so let me quickly um uh show you a code guys so this is how we will be working uh this uh in our python so here I have just imported a basic Library called numpy it is called as numerical python I've just given an alas called NP the values that you just saw I've just taken it as a array into a variable called as X all I have to do here is I will say x dot mean and I'm going to get the mean value and X do STD where I'm going to get my standard deviation now if I want my variance then I will have to multiply or Square this value so this is my variance so when you when it comes to python coding is pretty simple although we just have to understand what this mean and standard deviation does all right guys so this is the exact data that we had there all right so here um this was our standard deviation 2.9 and here also you got the standard deviation of 2.9 so the next thing we're going to talk about is Information Gain so this is some common terms that we will be talking about Information Gain Purity impurity so we have a formula for that the formula like I told you right we are going to use a lot of probability in that right so we're going to use a probability so the formula to calculate entropy is pi log Pi what is this Pi log Pi I'm going to find out let's say I have two sets of values I have some circles and I have some let's say St are first these are two different sets of data I need to find out the probability of this set only the circles I need to find out the probability of only the Stars so I need to do probability of say the dot multiplied by log of Pi this is just a formula guys you just have to know the calculation once you get into the Jupiter notebook or your code it's going to be with one line of code and I need to find out P of star probability of star multiplied by P of Star right when you multiply when you do a log value you will get a negative value so to negate this whole thing we are going to put a minus sign this becomes our entropy formula so what is this entropy formula what is this Information Gain just let's see with a example now the question is I have a data in that data let's try to uh concentrate on one and you should be able to understand the others so we have something called Outlook in that Outlook we have three values one is sunny overcast rainy windy has two choices false and true humidity has three Val two values high and normal and similarly the last column the question is how do I have to split my data why do you have to split your data we need to split so you're able to group the data because we are talking about a huge data set right so now we have to do something called as Information Gain calculate something called Information Gain what is that Outlook how many data points do you have in uh let's talk about windy guys this is a small example how many data points do you have in windy so I have 1 2 3 4 5 6 7 8 9 10 11 12 13 14 so I have a total of 14 data points eight data points went to the false side and six data points came to my true side right so what is the formula for Wy so I have eight data points here and four data points here so here you have only two values right one is called called is the yes the other one is called no so we need to find out the probability of yes what is probability of yes it is going to be 6 by 8 what is probability of no it is 4 by sorry 2 by 8 so we are going to apply this onto our formula remember that formula where I said Pi log Pi that is the formula you're going to apply it for the yes separately and the no separately so here you see 6 by 8 log 6 by 8 plus 2X 8 log 2x 8 this is your yes class and this is your no class and what is this set of value this is for these values here here you have 3 by yes is your 3x3 3x 6 and no is also your 3x 6 so we've applied it here what is this 8 by4 this is called weighted average I have a total number of eight data points here I have a total number of six data points here out of 14 so I don't want to blindly just take the probability value I will take the weighted average which is nothing but eight false values divided by total of 14 multiplied by this entropy that we just calculated plus 6 by4 multiplied by this value so I've got a value of 0.048 this is only for windy we need to calculate the same thing for Outlook as well so once you calculate you get a value of 0.247 here if you notice you have three sets of values so you need to do calculation thce so you have calculation one calculation two and calculation three the next is humidity so you have to do the calculation twice here and finally you're going to do the calculation for temperature and you have a value here so now you're going to compare all these values and we're going to see where where do we have the gain so the gain is24 0481 02 where is the gain highest for Outlook variable we have a very high gain so I'm going to say Outlook is going to be my first feature or I'm going to split my data based on Outlook because if I do that split I'm going to get a good gain in my data what is gain there is some information that I'm capturing to be able to split my data correctly that is called as Information Gain okay so inferential statistics I told you I will talk about the central limit theorem so what central limit theorem does is I'm going to take a population and I'm going to build multiple samples I have a sample one I have a sample two and I have a sample three so sample one sample two sample three here I'm going to get some set of values right some random values went into sample one some random values went into sample two some random values went into sample three you may end up with salaries or or you may end up with some values that are called outliers remember the example where I said salary was 10 and 20 but one person got a salary of 100 you may get this kind of values but what we are saying is for all this if you take the average I'm going to take the average or you can also call it mean I'm going to calculate my mean value here and if I plot the mean of let's say S1 the mean of S2 the mean of S3 mean of S4 mean of S5 and so on my Curve will be be normally distributed if I plot it for only sample one maybe the data is not following this normal distribution so I will take my different samples and if I plot my data then that means I'm going to say my data will be normally distributed this is called as Central limit theorem all right so I'm saying if I take the average of all these or the mean of all these values then those mean values for each of my sample will follow normal distribution so what is this inferential statistics guys there are two things one is called Point estimation and interval estimation so Point estimation means I am saying the average age of people will be 73 right how often can you give this exact value this is exactly a value right how often can you give this value not always probably not always or maybe we can never give this value so I'm going to give this particular uh exact point and I'm saying this is my sample mean I'm saying my sample mean is 73 so population average is also going to be 73 right we don't really use this point estimation because there is a very high chance that your population average need not be 23 then we come to this concept of interval estimation for Interval estimation although you're saying the average let's say the average age of a person is 73 I'm saying say the average age I think is going to be roughly between 69 and say 76 I'm not giving you exact value I'm going to give a range of value so the average age of the person is going to be between 69 to 76 and how confident are you so that is called as confidence interval this is not a point estimation this is interval I'm giving you a range of values all right guys so here this is what we are talking about Point estimation is an exact value you calculate your sample mean and you say that could be your population mean that may not be the case then comes this concept of interval estimation so here I gave you an example of 73 here I'm saying instead of saying 73 I'm going to give you a range the age is somewhere between 69 to let's say 76 this 69 is called your lower confidence limit 76 is called your upper confidence limit all right so there is something called as confidence interval so what is this confidence interval I'm not giving you a point I'm giving you a interval and how confident are you how confident are we to give that estimate for example I am saying this is your normal distribution curve this range this area the Shaded area is actually 90% all right so totally it is 100% so I'm going to give a range here I'm going to give a value for this and this and I'm going to say I am 90% confident that the age of uh the average age of a person in India is between 69 to 76 when I give 60 73 it is very difficult for me to give that point estimate so I give a range estimate all right so how confident am I saying that uh people are going this is the average age I am 90% confident right so this is my 90% confidence value and what will be the remaining value guys what will be this remaining value so this is my curve this is totally 100 and I'm saying this range is 90 that means these ranges put together should be 10% right so if both this put together is 10% then this area is going to be 5% and this is going to be my 5% right so we're going to use these to give a proper range this 90% confidence by default we actually have it as 95% confidence interval right so this is how we actually Define but if we are very particular we are we can always change this conference interval level okay guys so let's conclude our session thank you guys