Learn Hypothesis Testing | DataHour | Analytics Vidhya

Key Takeaways

foreign from different parts of the world welcome to hypothesis testing one of the most important topics in statistics world my name is Rohit Krishna and I work for JP Morgan as a data and scientist I have total of over five years of experience in data science field um I've worked on machine learning models the data analysis part the data science visualizations and stuff uh today we'll be dealing with hypothesis testing in general and uh I'll start by sharing my screen and then we will take forward let me start uh by discussing an example uh usually suppose that you are you are working for a manufacturing drug manufacturing company uh where you are a part of a research team just come up with a new drug right uh the theory is done you're you're done with the manufacturing but you want to test how this drug is uh affecting or what is the effect of this drug on different kinds of people in that scenario you cannot really go to each and every person uh that you uh that the drug is being tested on rather you collect certain samples from the entire population and then try to understand the behavior of that sample and then you attribute that behavior to the general population right so hypothesis testing is very important when you're dealing with samples uh you try to understand Its Behavior and then you try to try to map that behavior to the entire population right what can you expect from this session is that the general intuition behind hypothesis like what is it what is its role and importance in the statistics or research field uh followed by will deal with certain uh bit of mathematics on it uh because we will be solving certain examples how a particular hypothesis test is being conceptualized uh how can you come to a conclusion on your data and then uh we'll discuss about one of the most confusing topics I would say uh p-value and significance values and confidence intervals in uh in statistics subjects and Then followed by that we'll discuss uh various types of hypothesis that are available and its applications and uh one of the basic machine learning model called linear equation now I understand that uh we have uh we are we have people who are new to data science new to statistics and some of them might have already know uh what hypothesis listening is all about uh the session is named hypothesis testing A to Z but uh by that I mean we will try to conceptualize we'll try to dig deeper into the intuition behind the uh the the test rather than touching upon all the tests that are available in industry so we'll take Z test as an example uh we'll try to solve certain problems but once at the end of the session I believe that you would be able to uh you you'll be able to understand the terminologies of a hypothesis test the process how we do it and you can attribute or you can map those Concepts back to any hypothesis test at hand right so when you look at the formal definition of hypothesis testing right uh when you look at Wikipedia this is the definition that you find a method of statistical inference used to decide whether the data hand sufficiently support a particular hypothesis what is it what does this mean uh as I said suppose you are looking at election results right before election results you always you often find this service that uh news channels or certain other part organizations do uh it's very difficult to go and uh ask the opinion of each and every person in state or in the regional in the country right rather what they do is they collect samples they pick few people from different backgrounds and then uh base their analysis based the results on their opinion okay uh so statistics if you look at home if you look at whole statistics world it is divided into two parts one is descriptive statistics and so English differential statistics descriptive statistics deal with uh certain certain numbers that describe the group saying you have you are having 50 students of a class right uh you are trying to identify what is the average age of the students in the class or what is the average Mark scored by the students in the class or the what the maximum Max scored by a student in the class right so all these are kind of describing your data at hand uh and the the results are only attributed to that particular data right so trying to describe the data with certain numbers that is part of statistic that is part of descriptive statistics but inferential statistics is where you collected example let's take an example of the same example where you have 50 students in the group uh you try to understand their behavior and you base that understanding of the characteristics of that group and infer something about the population at large say you're trying to you have picked your five schools in a city uh you're trying to understand their behavior and based on the results of those five School samples uh you might be coming to a conclusion and deciding that this particular behavior is common across all the schools uh in a city or in a region in a country okay so that's the difference describe the statistics dealing with only the description describing the group at hand but in inferences Stylistics you try to infer about the general population uh right I see a question okay hello I I'll try to speak louder Right Moving On say we have a as I said population is very large and you try to pick samples right uh again coming back to the same election results example if you want your analysis to be uh correct what you try to do is you identify samples with different uh represent with good representations across different groups say you have uh you cannot go to a go to only rural areas and get their opinion and base your analysis basic results that this is going to happen in the election result no you uh if you want a accurate result you'll be needing uh survey results from different ethnic backgrounds different age groups different ethnicities different uh occupations right so the idea is that when you sample your data from the population that sample should be a good representation across all the subgroups in the population right if you look at this population diagram of the diagram that is in the slide I'm taking if if the color is the different characteristics like different uh attribute of uh regarding the population you see different colors being picked so that is basically representing your population well and this is really important because your entire analysis is based on the sample that you collect and if the sample itself is not a good representation you cannot base your results and attribute it to the population with good confidence okay uh now why is this important why is hypothesis important and what is its role you take any research paper uh now uh whatever experiment they have con they conduct take any field being beat manufacturing be it uh clinical types of medication uh make that discovery uh or machine learning model hypothesis right uh take anything at the end of the experiments you see you see you find a section where uh there's a statistical evidence that with certain degree of confidence we are basing our results and this is what is going to happen right so this became a common conclusive framework I would say uh for any researcher for any statistication uh to take the research forward and to uh and also for general public to uh to build the confidence on the exponent right with this with this knowledge at hand uh what we'll try to do is will take certain exam uh we will take an example of a particular data distribution and try to identify uh the descriptive stats about that the main the standard deviation about that and what will be our building block for uh for going to a hypothesis test okay and this would be this would have certain maths uh I would solve certain problems and if you are comfortable if you are interested you can have your panel paper at hand and then we can solve it together uh I see a question what if the sample was we collected was biased or incorrectly collected yes uh as I said T3 uh your entire analysis based on Sample so it is our responsibility to take sample with good representation for subgroups if not then uh the analysis that you do are the results that you control cannot be attributed to the general population it should be confined to the sample that you have collected or the or uh or the subgroups of from the sample that you have collected okay right uh let's take this a simple distribution where you have data 10 12 14 15 17 18 24 you have eight data points and you're trying to uh and let's identify what is the average and standard deviation of this group okay so you know that average uh probably [Music] so to calculate mean or average of the sample what we do is we add all these numbers and divide by the number of numbers divided by the number of data right so it should be 10 plus 12 plus till 24 by we have 8 right so the mean here would be 16. okay now this is the average value that this data represents now what this another term for standard deviation what is standard deviation is uh if you look at the mathematical formula it will be somewhere somewhere like this x minus X bar whole Square by n so now X bar here is the mean which is 16 in our case intuitively what standard deviation gives us is how deviated your samples are or the data points are from the uh from the mean value okay let's assume let's say that in this case our mean was somewhere here 16. okay let's calculate the standard deviation for the sample first and then we'll see what does that ingitility tell us so this would be 10 minus 16 whole square plus 12 minus 16 whole Square so what I'm doing is I'm taking each sample each data point sorry and then seeing how different it is from the mean divided by the total values but this would give me a value which is close to 4.03 Let's assume this is 4. okay what standard division tells us it tells us that from the mean value of the data how on average how deviated are the other data points okay so if you see here if I take each example 15 value is one number away from my new value 14 is two numbers away the difference between 12 and 16 is 4 this is 6. similarly on the other side uh from the mean value 17 is one number away from 16 it will be 2 2 again and 16 24 the difference is 8. okay so it it this actually gives us how spread how how is your data spread uh across the sample or are your data okay suppose if my deviation was set to instead of 4 my mean value being 16 I would say that most of my data points will be concentrated on between 14 and 18 okay this would be my General Distribution and values below 14 and 18 would be of less probability okay I can see I can say that most of my uh distribution data distributions line in this region so standard deviation gives us the average deviation how my data is split that uh okay now when you talk about normal distribution as it as I've taken examples of say students the average age of students or uh the election results any general behavior that you talk about it is uh it is for it it should follow a normal distribution okay what normal distribution is that majority of your data samples would lie in the middle half of the region the middle region and as and when as in when you go to the extreme end the the occurrence the probability of these values occurring would be less okay you can see most of the data has been concentrated here now we will base we will use this distribution as our base to uh to start our hypothesis analysis and there is something called uh CLT which tells us that if your data samples are greater than 30. right if a data samples is numbers if the number of data points in a sample is greater than 30 the distribution is assumed to follow standard normal distribution okay so once we take this as a criteria we can start our math and then pick one example we will give me one example and do our math so let's start with an example uh and we'll build the intuition behind the hypothesis How We Do It we have a medicine that is being manufactured by a company and each pin is supposed to have 14 milligrams milligrams of the active ingredient okay so uh we are manufacturing the pill which is trying to try which is trying to work on certain illness and each spin is composed of different uh ingredients right now one of them would be very active very important uh part of the drug say that's the the composition of that drug would be 14 milligrams okay now the researcher or the uh the drug manufacturer is aiming at this composition he wants he or she wants the active ingredient to be of 14 milligrams out of this uh drug okay but practically speaking uh when you when you manufacture a drug uh there are a lot of things that are factor in uh one is you new manufacturer we manufacture the drug not just in one Mission there might be multiple manufacturing production lines so are all the machines same producing exactly for 14 milligrams of acting ingredient is there any difference is it 14.01 is it 13.95 milligrams uh in the pill and if there's a difference if there's if there's something away from footing is it acceptable or not and we need to be very confident about this because uh of course when you when this drug is being being given to a particular patient if if the active ingredient is missing either it would cause side effects or it would not work on the illness right so to back up our analysis or our intuition behind this using statistics we can use hypothesis testing right now in the previous example that we have seen in this example we know that or most of the distribution will lie uh in the middle half of the region right let's take 14 milligrams as a mean value like this is what we are aiming for now there is a there is a very good possibility that our our manufacturing line is producing drugs which is probably 14.01 milligrams or 13.95 13. 15 right so you you know that even though the manufacturing production line is bit errorless you know that most of the distribution would lie here right intuitive and the probability that your manufacturing product line is producing drugs with 18 milligrams or 10 milligrams is less right uh with certain degree of confidence now taking this taking this region right there must be an acceptable threshold that you are okay with say uh so the drug manufacturer is saying that okay my my drug need not have exactly 14 milligrams but even if it is 14.95 I'm okay with it okay or even if it is 13.60 I'm okay with it but if it is 20 milligrams out of the pill that is of concern because this is not the general behavior of the behavior that the uh the manufacturer is aiming for okay so to test this he cannot uh so say the manufacturing Industries has produced a 10K uh say 10K pills now we cannot pick each sample separately and test what is the weight of what is the active ingredients weight in each sample rather what we do we take say 100 samples from these 10K samples from from this 10K population and then try to identify the behavior across this hundred samples all right now let's say for the hundred samples all right so let's say for the 100 samples my mean is here my distribution somewhat lines this is 18 as an example this is 10. okay me being a drug manufacturer I know uh the effect of drug I would say that if my drug is consisting of at least 16 milligrams or 12 milligrams I'm fine with it but if it crosses this region if it comes into this region then I would really worry about the drug okay similarly I want it to be at least 12 grams but if it is less than 12 grams that is of concern for me now when you have 100 samples at hand what you do is you find the average value of the the active ingredient so say that is somewhere around 15.2 milligrams right your your actual aim is to get 40 milligrams but after manufacturing your drug is you have taken 100 samples and the average value is 15.2 milligrams which is somewhere here okay now since this 15.2 milligrams is less than your threshold value less than your acceptable I mean uh within the acceptable region you are still okay to go ahead with your manufacturing value there's no problem but what if after you collected the samples instead of 15.2 your average value has turned out to be 17.6 milligrams which is somewhere here okay so this is a general this is a behavior that you are not expecting and you want to reject your hypothesis that my active ingredient is 40 milligrams of the production line no that's not so you have you have yeah you have some concern and you need to go back and revisit your manufacturing linear whatever composition You're Building okay so moving on if if you look at the red line right this is called the significance value in statistical terms okay and if your samples average value or that or whatever methods that you come that you get from the uh from the samples if it is within the acceptable range if it is within the acceptable range you accept your null hypothesis that my drug is okay to be delivered to a general public otherwise you reject a null hypothesis that 40 milligrams is not uh not what I see and I need to work on it okay so the intuition behind hypothesis testing is you take a population parameter you collect samples from the population and then you identify you see the difference uh if the difference between your assumed value which is 14 in our case and uh the sample estimate which was 17.6 in the previous example right if the difference is large and and if the difference is more than what your threshold is then you reject your hypothesis about the population and you revisit your analysis or experiment if the difference is small as in the previous case which is if it is 15.2 which is within the acceptable threshold you accept your null hypothesis and you go ahead with a my [Music] so there's a question initially when we claim something as null hypothesis is that analysis based on population or sample null hypothesis is always based on population DT uh that's what it is like the general Behavior Uh you say that children prefer chocolates more than adults right that that's a general Behavior you know about the population so always based on null hypothesis based on population and you collect samples do some math do some analysis statistical analysis and conclude uh the behavior to the population okay how to decide null hyperts and Alternate hypothesis I think uh that that's that's answered anyways um what is the percentage of sample from sampling from total population is there any specific values uh no I would say uh there's no percentage it depends on the side it depends on the size of the professions as you sample but it's always advisable that you collect samples uh with more number of data points the uh the larger data set you have uh the data would be approximating the normal distribution and then you can apply this test with good confidence so uh if you look at the books right you would say that the samples should be greater than or equal to 30 data points uh before doing any hypothesis test if the samples are less than 30 there are still other tests that you can apply but uh it's better you have more than 30 database uh yeah so hypothesis testing I hope the intuition is clear uh use you you check how extreme the value is from your sample from the hypothesis value and if the difference is large you reject your hypothesis if your result if the difference is within the acceptable range you accept it okay so formal definition about null hypothesis it's a statement about a population parameter which is believed to be true and Alternate hypothesis is the statement that directly contradicts the null hypothesis uh I think this is explained already now in the previous example what would be our null hypothesis it would be that my average value uh average uh composition of active ingredient in the drug is 14 milligrams whereas my alternate hypothesis would be uh the average value is not equal to 14 milligrams it can either be less than 14 it can be greater than 14 but it's definitely not 14. okay now when you talk about the distribution here as I said there is a certain threshold that your within which you can accept your null hypothesis right but this regions right where if you if your difference is falling in this region these regions are called rejection regions and the values the value here these values are called critical values okay so once a difference is falling within this critical region we reject your null hypothesis let's take the example previously we had numbers of say 10 12 14 15 17 18 18 and 24. okay uh we uh as I discussed in the introduction right we would be dealing with z-test uh as an example and then we'll build the entire hypothesis on that now what is actually z-score z-score is nothing but the number of standard deviations away from the mean say in previously our mean was 16 okay so each number here can be represented in terms of a z-score in the in terms of a standard deviation like how many standard deviations is 17 are we from 16 how many standard deviations is 18 away from 16 okay so what what are we doing essentially we are trying to uh convert range one range of values into a standard range okay into a into a z-score say first 16 if you if you apply this formula right where X is your actual value mu is the mean of the data distribution and sigma is the standard deviation if you remember standard deviation for the sample was 4 from the previous example and mu was 16 right now this this series of numbers like 10 12 14 15 can be represented with Z scores okay 16 times 16 if you substitute 16 here 16 minus 16 by 4 would be 0. so the Z value for mean would always be 0. okay similarly if you substitute 17 17 minus 16 by 4 it will be 1 by 4 which is 0.25 18 would be 0.5 20 would be 1 24 would be 2. similarly you can calculate for uh these values will be coming into the distribution right what did we do essentially we have mapped or maybe converted this data into a more standardized bit so for this example we have got range of values between 10 12 around in two digit numbers right say we have another example where we have uh where we have values ranging from thousands to blacks right even then uh to standardized there's something called Z table where we find for probability distributions so to standardize our analysis we will first convert those our data points into a standard scale and then use these values to backer analysis the second questions I'll try to answer them is there any way that the sample taken is biased or outlier values and the actual values will fall in the allowable range and the action values range yeah if I understood the question correctly uh you definitely need to you definitely need to see if your sample is consistent or hours and there's a way to identify those outlets and statistics which is uh multiple ways but if you look at the Box part distributions uh you can find out ways right but it's our responsibility to take sample that is a good represent of the population as I already told you next question can hypothesis testing be used to validate uh the variables you are using to build a machine learning model yes you can uh when you're trying to build a hypothesis or the uh or the general intuition behind any regression model right so I I think I'll be dealing with this in the last slide in this lecture let's hold on till then where did you just get the value of Z so Z is being calculated here so z z is a representation of it's a mapping of your original values into a standard value right so x minus mu by Sigma is this now suppose my my data point is 10. and my mean value is 16 as I said so I'm substituting 10 here mean new is 16 standard deviation is 4 so you get your you get your Z value to be minus 6 by 4 which is minus 1.5 right so that's how I get my Z values so another question if conducting seven of population is already impossible then how can we say some value has population mean so that's a general assumption uh so it depends on the kind of problem statement that you're working with if it's a drug manufacturing then you you know what your drug should have and then you uh based on your medical knowledge right you know that this is the acceptable or this is the composition that you're aiming for so that becomes a population mean or the popular hypothesized value the invert we mean always that's an example but uh there's always a high processed value from the population from your general understanding okay what g-score tells us mean why we calculate okay so Z score tells us how how deviated your data is uh or how deviated your particular value from the meanness say my I'm telling you that okay my I have a class of 50 students um and the average score scored by 50 students is 75 out of 100. okay this is the average score which is just a single number trying to represent the entire class but I want to get more information about uh more information about this uh the class students right how their score values I know that uh out of 100 90 is the maximum score uh 35 is the minimum value score by the student okay now this gives me an intuition about say my data is ranging from 35 to 90 with 75 being my average value but I want to know the standard the deviation in the data on average how much is the uh how much is each student away from the mean of 75 so that's what G tells us in terms of the standard deviation so if the standard deviation for this case is say 16 16 marks I would say that uh most of my data points are 16 marks away from the 75 value so that's what Z tells us go back and answer the question okay so coming back to this point once we have the data distribution hand like in this case 10 12 14 15 a standard scale which is [Music] minus one two plus one okay all right once we calculate the z-score how does it how does this score help us in identifying or concluding a hypothesis say in the previous example that we have dealt with 14 milligrams of other mean value okay uh we are mapping it after this after converting these values to Z score the mean of the Z uh the mean of the 14 milligrams would be zero right somewhere around here [Music] yeah so this is my mean now I want my drugs to be concentrated mostly in this area which is the middle half of the region which is which is within the acceptable threshold as I discussed what we do is once we collect samples right we we uh we come up with a z statistic and once we come up with the Z statistic we'll see whether that particular value is falling suppose this is the critical region once we once we calculate the Z statistic if it is within this region we accept the null hypothesis if it is falling in this region we reject the null hypothesis it's it's not actually accepting a hypothesis but we are failing to reject neural hypothesis we never accept the hypothesis but we fail to reject it okay now I'll deal with the Z so let's take in detail with an example in the later in the next slide but as I said you identify critical regions and then you try to try to conclude whether you can accept or reject the normal hypothesis now based on the kind of problem statement that you're dealing with or what you're trying to answer right uh you either accept your um I mean the rejection region would fall either on one side of the distribution or it could be on the both sides let me give you an example say uh you have there are incoming admissions for the University right and uh you're checking the average course code by the students uh in the college uh in the entrance exam you say that on average I would only accept students our minds you know universities consisting of students whose average score would be 110 in the entrance examination okay that's a that's a general bill or hypothesis now you've taken sample from the population from the students and then saw that uh the sample suggested that your average score is 120 rather than 110. in this case your actual value from the sample is greater than your null hypothesis value okay so that's when you take the right side the right end of the distribution as the rejection region so your null hypothesis is that states that your average value is 110 but your actual data is saying that it is 120 which is greater than your average value so you look at this part of the region and see if this 120 is calling within the acceptable region or if it is falling above this region if it is above this region then you reject a null hypothesis similarly say take the same example you want students which who have average score of 110 but your uh but your uh but you're actually it is suggesting that you are getting students which are of uh having 90 as the average score right which is very far away from your uh from your uh from from your front score that you desire in your University right so it is very away from that then you probably look at your admission and then see uh why are these students coming up okay uh taking previous example for two-tailed test where the mean value is this but uh your hypo your hypothesis value is this one but your actual data is suggesting that this is not equal to 14 it can either be greater than 14 or less than 14. so here you are interested in both the sides of the data distribution so this becomes 2D since you are dealing with only one tail here these two are called one tail test and the bottom region is called two tilters okay I see certain questions in the comments is there any statistical technique for selecting the sample from the population there are different sampling techniques uh so you have uh you have random sampling you have you have systematic sampling uh probably four or five kinds of sampling and uh it it depends it depends on how what is the kind of problem statement that you're dealing with sometimes random sampling is designed sometimes you go with systematic sampling so it's all in the hand of the researcher which kind of something does z-score distribution normalizes the data yes it does uh because just because you're scaling your data within the standard deviation right uh it definitely normalizes your data can the data points which are way too far from zero be treated as outliers you you cannot completely say that uh some mixture because um it depends on how your data values are distributed say your most of the difference between minus 20 to plus 20 and you're finding a value which is 40 okay now that become that becomes an extreme data point but what if a distribution has a values from minus 40 to plus 40 and you're finding a value which is 45 right uh so 45 is not as extreme uh as in the previous case right so in the second case it might not be an outlier but there are statistical techniques uh to identify whether a particular data point is considered out later or not is it used by statisticians only or data professionals also it is used by data professionals it is used by researchers in the medical field in the manufacturing field uh so so this this deals with any data right I mean beat any industry uh once you have data points you are trying to understand whether this data point is the the representation test of this data point is uh being done properly the analysis of the experimental results on the sample is according to your belief or your desire or not so it can be used by anybody so Ria asks the question there is a possibility that the data value in data set is biased or values are outliers but cases Q's Q ins case either right let's create but there sir has considered normal distribution okay yes so when you're trying to uh good question Riya let me come to that suppose your data your your data is not always normal uh in reality right let me erase everything only suppose you find a data discipline like this or like this wherein uh your lower values of are of higher probability and your larger values are of less probability okay now this is a population of this is a sample set Central limit theorem there's something called Central limit theorem which says that if you collect good number of samples from the population the the average metric or the the sample statistic that you're dealing with right if if you are dealing with more than 30 samples this will approximate a normal distribution because of this theorem we are assuming our analysis we are basing our Z test based on the normal distribution so if you have consider considerable number of samples any skip distribution would be converted into uh normal distribution once you collect different samples and then take average of the samples and plot the distribution of the samples okay what is advantage of a two-tailed test was a one-tailed test uh there's nothing it's it's not about Advantage it's the question that you're dealing with when you are trying to answer a question uh say your hypothesis of a certain value my mean value should be zero if your hype if if your question at interest is whether to identify whether the data is actually 0 or not equal to zero then you go for two tables because the value can be either less than or greater than zero but when you're interested to find whether the data is greater than zero or less than zero then you go with one table test so it's it's basically the question at hand uh it's not about the advantage uh what is p-value I'll come to that Sunday we will discuss it uh in the next slide okay let me continue with the session I'll come back to these questions again right so let's let's come back a bit uh keep mathematics aside and then see the procedure of hypothesis testing like right from conceptualizing your null hypothesis how can you conclude your uh analysis right first step is that you determine a hypothesis and Alternate hypothesis uh in the previous example again uh we have used mu equals to 14. uh this is my null hypothesis where my alternate hypothesis mu is not equal to okay [Music] you verify the necessary conditions that are satisfied and chooser a significance level now what does this mean in the previous slide uh yeah this one so I'm talking about this region as a critical value right this value is called the probability the area after this is called significance value the probability that or the proportion of the of this area compared to the entire distribution right now once you once you get this critical value uh we need to have a threshold at hand which tells us that you need to have Threshold at hand which tells us that if the critic if the value that we are getting is above this then uh you reject or you accept right now this is called significance value okay so how do you see how do you choose your significance value it is completely on the research on the researcher's hand but industry accepted like most of the people that you see is that you uh you see 0.05 as the significance value or uh five percent as a significance value we'll talk about this with an example in the next slide I'll come to that now once you come up with this significance value of the threshold uh to accept the rejectional hypothesis you you identify the critical regions like if this is the uh these are the critical values if my data point is falling in this region I reject the null hypothesis so this this becomes my critical reasons okay after that you compute your test statistic uh just like your z-score which is x minus B by Sigma as an example once you get this value right like you have a z value and you see if Z value is falling below the critical region or above the critical region in this case if yes then you make a decision whether to reject or accept your null hypothesis and then apply this math so this till this point is it's just numbers right now intuitively I uh interpret this decision to the context of a problem if your drug manufacturing should you uh should you uh go ahead with your manufacturing process because you find the results accurate or according to the behavior or should you revisit your uh composition or the manufacturing procedure or not right you interpret that decision in the context so that's generally a six stage uh process uh right from conceptualizing your hypothesis to concluding your result foreign and then uh hopefully things are clear by this okay College a has an average SAT score of 1500 okay from a random sample of 125 freshman psychology students we find the average stats code ticket 1450 with a standard deviation of 100 if you take these two statements right my null hypothesis here is that average score is 15. 1500. foreign [Music] as part of the experiment you have identified 125 students okay and you are seeing that their average is 1450. now this is my population mean and this is my sample I denote it by X barcel and this by new okay and you know that the standard deviation is 100 for the sample what does this mean on average so this this becomes the representative of the 125 students right about 14 minutes average but when you look at the spread of the data here so the score by this 125 students you know that uh I the standard deviation is either one uh you can see the values from 1550 to 1350 right this is a standard deviation the values will definitely range above 2015 uh sorry below 1050 and above 1550 but in general you find data to be distributed like this most of your data like this okay so that's that's given in the data now what what are we trying to find here we want to know if these freshman psychology students are representative of the overall population in terms of SAS score what are the high qualities okay so the the the problem at hand is that I have taken 125 students into consideration I have taken the sample sample me but is it representing the actual hypothesized mean value about according to my general understanding if yes then we accept the hypothesis that yes my College college a is having students with a SAT score of 1500 if not then this hypothesis value about about the college a is wrong so average score is not 1500 in general but uh it is something else and you need to recalculate that because okay now let's let's solve this problem uh step by step and completely enter hypothesis test okay so my mu again is 1500 the number of samples is 125 125 students my sample average is 1450 and my standard deviation or the sample deviation is 100. okay we know that z-score is X by x minus mu by Sigma by root n okay so this is the formula what we do now is we substitute this x is 1450 minus 1500 by 100 by root of 125 which gives me value of minus 50 by 100 by 5 close to uh this would be around 5.5 meters or no to 2500 by 100 will be 25 right that's my z-score sorry right that's my Z score now what do we do here there's something called Z table which tells us that what is the probability of finding probability of this proportion of the data once a critical value is given in this case let's choose the let's choose that our extreme region right my acceptable threshold is Alpha 0.05 okay which means that if I find my average value to be in this distribution on uh on the extreme five percent end of the data either on the lower side or on the upper side if my Z statistic is falling in this side I reject my null hypothesis okay if you look at the Z table these values would be 1.96 and this would be minus 1.96 okay 0 being your mean value you have one yeah you have one year minus one so this becomes my critical value okay and if my Z statistic is coming to be above this region then I reject if it is greater than my Z static is greater than 1.96 or if it is less than minus 1.96 I reject the null hypothesis saying that my hypothesis mean is not 1500 but something else okay so I'm not here with the explanation on one tail versus student tests okay uh I I'll deal with that with with two examples this is one example which is having one tail uh two tail test I'll deal with one tail test in the next example [Music] all right so the Z the z-score that that we have calculated here comes out to be um minus 5.59 just do the math here uh it comes out to be minus 5.99 which is again uh stop one second so yeah so the z-score is 25 here so my 25 value is way above than 1.96 which is at the extreme end so my average value is not 1500 so what is my conclusion I conclude that the average size score is not 59 right so this becomes two tail test because we are looking at both ends either the average score can be less than 1500 or greater than 1500 right let's take another example here now this is an example of one tail test a farmer is trying out a planting technique that he hopes will increase the yield of his pea plants right over the last five years the average number of pots on one of his P plans was 145 so his General assumption is that for each P plan right the average number of pots that he finds is 145 with a standard deviation of 100 okay so in general it can be either 45 parts per plant or it can be 245 with an average value of 145. okay now this year after trying his new planting technique he takes a random sample of 144 of his plants so what is my sample number it is 144 data points and he found that the average number of parts to be 147. Okay so he wonders whether or not this is statistical significant increase right here we are dealing with increase uh we are trying to answer the question whether whether the data had the sample is suggesting an increasingly pot or uh it is the same it's not talking about the decrease in the number of bonds so this becomes our one tail test previously it was what that you are trying to look whether the average score was 1500 you're not trying to see whether it is less than 1500 or if it is only less than or if it is only greater than we are okay with both sides right so depending on the question you select one tail or two table okay now in this case uh water is hypothesis and the test statistic whether uh can he rely that uh since he has found based on 144 samples he has found that instead of uh based on 144 samples right uh instead of 145 Parts he has on average found 147 Parts but is this 144th number of samples sufficient uh to conclude that whatever new planting technique that he has come up with will work on every every plant uh if he applies this technique right so the the if you look at the idea I mean if you look at the integration behind it you're essentially trying to see whether the average value or the the the effect of this particular technique is by chance or is it really producing some of it okay let's solve this as well so coming to the Z value uh X bar minus mu by Sigma by root n right so this turns out to be 140 uh 147 which is the sample mean minus 145 by 100 by 44. okay now if you calculate the z-score this would come something uh this would be around 0.24 so only sorry yeah so this would be around zero point two four okay now keep this keep this aside till now what we have done is have substituted our values in this formula now let's choose a significant value as I said once we get this value we need a threshold to identify whether this is falling within the range or above the range right so again coming to the normal distribution which is this is our mean taking five percent as the significance value which means I'm looking at the five percent of the extreme end right uh in this case we are dealing with the increase in the data right so selecting the right reason this becomes my five percent of the entire proportion and if you look at the Z table this value is assumed to be one point six four five okay the value here 1.645 now what is the value that you got 0.24 right so 0.24 Falls within this region below 1.645 so your Z value is less than your critical value clear which means we are we are failing to reject our hypothesis what is a null hypothesis here or null hypothesis that the plant technique even though you apply the planting technique the average value will be 145 right uh and your statistic suggests that you don't have any significant value significant increase in your uh in your average value after this and since it's falling within the this region you are you are failing to reject anal hypothesis which means that your planning technique is not actually working on your plans it is just by chance that you found the average to be 147 but it is not statistically significant right but what if your z-score has come out to be two in this case the Z value would be somewhere here the difference between your hypothesis value and your data suggested value is kind of very large than the acceptable threshold then you say that your plant detected that the the farm the planet technical formula has applied is actually making a difference on the plants right so what do we conclude by this analysis we say that whatever planting technique is there it's it's not actually working on plants that the effect is not as huge as expected and your null hypothesis failed to be rejected okay um how did we get a critical value of 1.65 all right so the critical value here right uh hello okay uh the critical value in this case comes from a data from a z table or any any statistic test right there are Z table uh there are tables for Z for Z test there is a z table for T Test there is a t table for Chi Square test you find uh Casper table as well in that case what you do is you go to the table and you identify so since 0.05 is our significance value you look at the probability you look at the z-score value for this probability probability value right there you get 1.645 okay uh I can probably show you just let me grab this video hope you can see this a table so um all right so based on the probability value right since 0.05 is taken as the extent right uh subtracting one from that if it if you're looking at 0.95 right uh you take a corresponding probability value and then you look at the row and call corresponding role columns in the Z table and that's when you get your Z statistic so for the probability value of 0.9945 resistoric Z statistic is 2.54 okay 2.5 first and then the second decision would be four similarly for the probability that we have considered here uh which is 0.05 RZ statistic would be from uh would be coming down to 1.645 from the table okay so after dealing with the two examples right we know we came to know that how do we conceptualize uh a hypothesis uh how do we uh come up with a significance value significance values are understanding uh it can be 0.05 can be 0.01 can be 0.1 it depends on the researcher and then we compute the statistic from the sampling distribution and then set and then check uh how different is the sample from your hypothesis value and if it is very different you reject a null hypothesis and if it is within the acceptable region you fail to reject a null hypothesis okay now p-value let's come to this this is again uh if you look at the formal definition assuming the null hypothesis is true so whatever my basic assumption is there right like my drug manufacturing example where the mean of acting gradient is 14 milligrams assuming that that is true what is the probability of getting a statistic number which is very far from 14. okay so that is called P value and and based on that P value and taking significance value you can either reject your uh field reject on a hypothesis I know it's too much to take but let me deal with this with an example uh consider the same probability distribution like this okay I have my new here me my uh Z statistic right uh the critical value is this you just say 1.645 in the previous example how did we get 1.645 it is com it is just dependent on the the significance value that you are selecting it's not dependent on the sample data distribution so for five six five percent uh for the five percent significance value my my z-score is 1.645 which is here okay now uh as in the previous example right as in the previous example uh I have my data points say we have collected 30 data points we have got the Z statistic to be somewhere around here from the sample data right my my data is somewhere here which it says 0.9 my Z value okay now what is p-value p-value is a probability of finding a finding a number greater than what you got so the entire region which is marked by white right white shade is the p-value this is the probability value that given my null hypothesis is true uh given my null hypothesis to the probability of finding Z statistic on the data distribution greater than 0.9 which is this okay now if you see my critical region is this my I have my critical value of this uh of the data distribution to be uh this region I know that my P value is I know that my P value is greater than the critical region or the significance value so my p y my P value here is 0.9 say uh or 0.09 intuitively correct uh and my significance value is 0.05 okay now if my P value is greater than my significance value you fail to reject the null hypothesis because it is in the acceptable region responding in the acceptability right uh a similar example let's take the same scenario where you have new your critical your critical value is 1.645 and your Z statistic came out to be 1.8 okay so 1.8 value will be somewhere around this okay and your P value for this 1.0 statistic say would be 0.02 the probability right in this region this region okay If You observe the difference between significance value and the p-value it's important to understand significance value is your significance value gives the probability where the region is above your critical value or below a critical value if it is total test but P value comes from your data distribution from your sample after getting a sample you calculate your Z statistic and look at the probability of extremence from that right and see the probability that you find which is 0.02 in this case if it is less than the significance value now in this case it is forming at the extreme end and as we said if it is falling above the significance value or the greater than the critical region uh we reject the null hypothesis so in this case we reject it okay so instead of calculating the entire z-score going to the Z table and uh coming to a conclusion we can simply use p-value and there are statistical packages available both in R and python uh where you just input your data and you get your uh your statistic values and output along with the p-value right so just based on the p-value if it is less than your significance value usually we select 0.05 if it is less than the significant value the reject General hypothesis uh and if it is greater than the significance value you accept the value purposes right a simpler way of analyzing a hypothesis result rather than going and calculating uh then that statistic and going to Z table and coming up us all right since we are running up on the time I'll just touch upon this concept for you let's say still now what we have done we have we know that we have assumed something from the population okay that this uh this is a general behavior from the population and uh we have used sample from this population and we have studied the characteristics of this sample and then we came to a conclusion about the population right so this was the General uh intuition behind behind the discussion till now now confidence intervals work in a different way what we do is we first collect sample okay and within within with certain degree of confidence say 95 percent confidence or 90 confidence they say that uh whatever value that you have founded whatever statistic that we found from the sample uh will lie within uh we'll we'll lie with a certain range right this comes this range comes from the sample right this range comes from the sample and we attribute this to a population clear let me let me talk about this with an example let's say that a sample distribution uh we have a we have a new value right uh we have say 30 samples we calculated the sample mean from this say it has come somewhere between 12 right the sample means 12. now what I would now try to do is since my mean value is 12 I want to know I want to uh state with certain degree of confidence say 90 of the confidence that since my sample mean is 12 I would say that with 90 conference that my population mean also would be lying in this range now this range represents 90 of the area under the Curve okay so I'll be rejecting the extreme five percent five percent here I'll take the middle 90 percent I'll see the range here and I say that uh with 90 confidence I'll be uh my Sam with 90 confidence the population mean would be ranging between uh these ranges with the mean value with the mean value probably close to 12. okay so in this example as it is mentioned here with 95 confidence interval uh the population mean is between 72.85 and 107.815 okay now how does this 95 come up it's again our uh it's again under our uh uh discretion which conference we need to take if it is 99 what would we do we go to a rather more extreme ends if it is 80 our region of Interest would be decreasing right so as our confidence decreases our range also decreases right that's what confidence intervals gives us it gives us an idea about the population parameter uh by studying the sample statistic now as we're dealing with the hypothesis right like there are there are four cases that you can uh broadly uh look into one is that null hypothesis can be true or it is actually false uh again taking the example of 14 milligrams right my null hypers are 14 milligrams if it is believed to be true we get under these conditions but if my assumption is wrong now my null hypothesis itself is false which is actually the 14 milligrams is not correct but it is somewhere around 12 milligrams or 16 milligrams right so my null hypothesis response so under these two different conditions we we have two possibilities either we fail to reject the null hypoxes or we reject the null hypothesis okay now this gives us four cases okay now when the hypothesis is true and you fail to reject the hypothesis which means uh my critical value my critical values are here your hypothesis true and your the sample statistic is falling in this region right in this region so which is a correct decision so you are good with this decision okay when your hypothesis is true you should be failing to reject another hypothesis similarly when your hypothesis Falls you have to reject on a hypothesis right like suppose if my uh if my ACT if actually the uh active ingredient composition is not 14 grams milligrams but it is something else and after conducting your experiment using statistics you assume you even your statistics told you that uh the null hypothesis is false then you again came to a correct decision but the problem is these two cases right when your hypothesis is true but you're trying to but you're rejecting hypothesis uh then that is called type 1 error which is Alpha and this Alpha is nothing but the significance value that we're talking about okay so the probability this probability right this is called Alpha this is type 1 L which means your null hypothesis is actually true but your sample is giving you a number which is very extreme that even though your hypothesis is true you are rejecting your hypothesis right you're you're actually rejecting a hypothesis when you should not so that gives you type 1 error the second case when your hypothesis Falls but you fails to result say uh actually my my main is not 14 uh practically speaking my main essay somewhere around 12 but after conducting experiment you're getting your sample data between 13 and it is within the acceptable region so even though you have to reject your null hypothesis you are failing to do that so that's when the type 2 error app is okay so you need to be capable with these two types of errors and then uh take your significance value in such a way that you have you you convey your results with greater confidence okay all right so that's with z-test I guess uh right from conceptualizing your hypothesis defining your null hypothesis and Alternate hypothesis coming up with a test statistic and seeing if it is falling in the extreme regions of your probability distribution if it is at the estimates you reject your null hypothesis if not you accept it right now that's Z test for you uh but there are different kinds of tests uh available uh namely t-test calculator which is which are again useful in different scenarios uh briefly speaking uh z-test is used when your uh when your sample data has more than 30 30 data points right and when you are when you know about your population distribution like I talked about the standard deviation right when you know the standard deviation of a population and when you're uh when or when your data points are more than 30 you can use z-test but these will come in handy where your data points are less than 30 or when your population standard deviation is okay uh I'll not get into specifics since you're lacking in time but uh based on the data right like first you need to understand like what is your sample size what is uh what parameter what information you know about the sample and the population and uh that's when that's how you can select which test to apply okay so Z test and t-test uh f-test these are all numerical a test based on which which work on numbers but Chi Square tests work on categories okay so I'll deal with simple example in the next life of chi Square test uh and then I think uh we'll take questions if there are any so till now we were looking at one what what one particular uh data type uh one particular drug sample or one particular collection results or the average score scored by students right so which are numbers now uh imagine a scenario where uh you're trying to conduct a survey and you you believe that the female population is more inclined towards Democratic party rather than a main problem male population right now this is your hypothesis uh which you want to check right so in this case my mother hypothesis will be there is no effect of gender on the selection of the party whereas my alternate hypothesis would be female uh female voters would prefer Democratic party more rather than Republican party okay so that's where that's how we come up with the original hypothesis right foreign how do we come up with the conclusion we what Chi Square does is it analyzes the pattern between these numbers like uh with these four numbers that you do that you see here right it checks whether the distribution in this is is there a pattern which is above an acceptable threshold above above the expected value uh and if yes then then there's a different relationship between gender and the party that the voters sell okay if not then we say that the the voting to the vote to democratic Republican party is independent of the gender okay so that's how Chi Square test helps us it helps us to come get to conclusions with the categories the categories at hand so chi-squitous can be used in two different scenarios one is to understand the goodness of a fit so say you have uh uh you have only one category right Democratic party and you have gender at hand okay so fifty percent of the male candidates male voters are uh uh voting Democratic party sixty percent of the male cat voters are voting to the Democratic party right so easy is acceptable so this is the observed value right and there is something called expected value that you usually expect like 50 uh in this case it would be half of the population would be closer to democratic party and half of the population of mail would be choosing Democratic party right now we will see how different is the observed values from the expected value using this formula using these two formulas and we come up with the statistic called chi-square I'm not going into the mathematics right uh and if this chi-square statistic is greater than the critical value right if this is greater than the critical value then we come to a conclusion that the effect uh that uh the the fitness the goodness of the fit is not as as expected okay similarly when you look at the test of Independence yeah this this example comes in handy where you have two different categories and your your base assumption would be your null hypothesis would be gender is independent of voting okay independent of party selection where is your where uh your alternate hypothesis gender is dependent for your party under selection is dependent upon the gender so in this case you're trying to test the independence of these two categories gender and the party right so Chi Square test helps in these two scenarios all right um so this is the last slide one of the questions was regarding how we how was hypothesis formulation useful in machine learning model building right uh so so those those are interested those are not linear regression what we try to identify is suppose say that we are trying to uh so this is the weight of different persons this is the data that I have okay uh over seven samples here and this becomes my input say I'm trying to identify the height of the person based on their weight okay now X becomes my independent variable here uh talking in terms of uh regression and Y becomes my dependent Revenue okay so what am I trying to do here I am trying to predict the height when I only have weight at hand okay so if I plot weight see this is 140 to 250 and this is my x-axis and I plot the height on Y axis so this is somewhere on 50 to say 80. the data is something like this okay you find a linear Trend here uh even though the the points are a bit scattered you find a linear Trend between uh height and weight so there's a linear relationship that you can say which is found between weight and height as weight increases you know that your height increases in general there might be certain Outlets uh certain uh certain points which do not follow but in general you find a linear Trend here okay now without applying any mathematical technique or any regression model or any machine learning model if I if I ask you to say append it height a simple thing that I can do is take an average of the height in the data and then uh then say that this is my credited value so irrespective of the weight I say that my height will be on average say 70 this is the average so irrespective weight for even for 148 predict 70 even for 270 okay so that is without any math any regression model I'm just using an average of the weight from the uh average of the height from the sample that I have okay so if if I take that as example right like my Y is 17. if you remember the straight line equation this Alpha plus beta x or y equal to MX plus C that we have studied in our schools right where X is the value of uh x axis x uh the value on the x axis C is the y-intercept right and M is the slope okay coming back to the coming back to the previous assumption if I predict if I take y value to be 70 what I'm trying to do is this is equal to 70 plus 0 of some x value okay so in this case my M my slope is 0 or beta is 0. and my C value is 70. okay so without applying any regression model my regression equation uh without applying any complicated math my regression equation came out to be y equal to 70 which is this horizontal line right suppose taking a trend into consideration okay my regression model uh there are certain optimization techniques that are playing in hand and you come up with certain equation which will try to generalize the uh behavior of predicting the height with respectively and suppose this is the uh line that has come up the model has come up with which is say y equal to 70 plus some 0.1 into x value okay this is my equation y equal to 70 plus 0.1 into X okay in this case your beta value is 0.1 and in this case your beta value is 0. okay when your beta value is 0 as in this case you are saying that there is no relationship absolutely no relationship between your x uh X variable your independent variable and your dependent variable okay so my null hypothesis in this case would be my slope is 0 which means there is no relationship okay whereas my alternate hypothesis is that my slope is not zero which is in the second case in this case my slope is not zero there is some 0.1 slope uh which suggests that my the values of Y are definitely dependent upon the X which means as X is increasing the Y is also increasing right correct in this case no as X increasing the Y value is remaining constant even though you take X to B 140 lbs or uh 212 lbs my value is not my y value is not changing right so there is no relationship between Y and X but in this case your X is increasing your Y is also increasing so there is definitely a relationship and how can you quantify that relationship in this equation the value of slope or with the value of beta so that's how I'm conceptualizing my hypothesis okay now what you do after coming up with regression model say your regression model is somewhere like this your equation is like this uh y equal to uh some uh 0.1 x plus 70 okay now how can this hypothesis test help us it can tell us whether the slope that is in hand right is it is it actually representing the data or is it actually predicting a value correctly uh or should uh or do we expect the slope value to be a bit higher so uh as you as you know if this is a trend you can have a regression line like this or you can have a prediction line like this okay now whatever slope your data is showing is it statistically significant enough to base your uh conclusion that y value is dependent on X or is it not that statistically significant okay so that's how you can use hypothesis testing even integration models or even in your machine learning models uh to to uh to come up conclusion to come up on conclusions after evaluating your models the coming up which are modeled equation okay so uh summing up what uh we have done we have seen what is population what is sample uh how do we sample data how why is it important to collect samples which is a good representative of all the subcategories in the population and monthly collected sample uh we come up with a significance value of the acceptable threshold I would say uh above which if your data if your sample is suggesting your data is above that acceptable threshold your reject the null hypothesis uh and if it is not you accept General hypothesis and there are different techniques like Z test T Test F test questions depending upon the data data at hand depending upon the kind of question that you want to solve uh you would select a test and then go about it so so that's from my side uh if you have any questions I think there are [Music] a number of comments I'll try to answer these questions and uh if if you're interested you can stay back how to decide I think I think I've answered most of the questions because they are posted previously um so why not why why not 99 is one of the questions so uh it's up to your choice Ashish so as I said with some researchers uh discretion which significance values you have to choose no but understand this thing so if yes if you're if you're selecting 95 percent as the significance value on the five percent significance value and your acceptable range is 95 percent uh compared to your 99 percent okay if you're selecting 99 you would only be left with the top one percent or bottom one percent or uh or if it is two-tailed test uh bottom 0.5 and top 0.5 percent right as we uh as the rejection regions so you're confining your uh what is your analysis to a very restrictive Zone okay whereas 95 percent gives you a better uh a a better a better spread of the data where uh whereas your your rejection regions are also of good quantification not just point five percent or one percent but you have five percent of the values which are following in the rejection region right and it's again uh dependent on the data at hand and the question that you're trying to solve if it is drug Discovery right if there's an application like drug Discovery you you cannot uh you cannot do a mistake right so you you can probably select 99 of your 99 as your uh significance value or the one personal significance value because you want to be very confident about your statistic right if it is not that important if you are trying to do something about the election results or if you're trying to do something on the manufacturing defect site right then 95 percent is feasible so it's it's Upon Our interest thanks a lot for joining uh have a great rest of the day [Music]