Beginner Friendly Data Science Course With Python 2026 | Data Science Training | Simplilearn

Key Takeaways

[music] Hey everyone, welcome to our data science with Python full course. Ever wondered how data can transform into powerful decisions? Picture turning a mountain of raw numbers into insightful that drive the future. This data science with Python course is your gateway to making that happen. So whether you're just starting out or looking to level up your skills, we have got you covered. We will kick off things by building a solid foundation in statistics, then dive into the world of Python, your go-to tool for handling data. And along the way, you'll get hands-on experience with large language models, the cuttingedge tech that revolutionizing data analysis. And by the end of this course, you'll not only grasp the [clears throat] theory of statistics, but also know how to apply to solve real world challenges. Plus, we will prepare you for those tricky data science interview questions so you can walk into your next interview with confidence and impress your future employers. So, let's get started. >> Deep learning. Deep learning was first introduced in the 1940s. Deep learning did not develop suddenly. It developed slowly and steadily over seven decades. Many thesis and discoveries were made on deep learning from the 1940s to 2000. Thanks to companies like Facebook and Google, the term deep learning has gained popularity and may give the perception that it is a relatively new concept. Deep learning can be considered as a type of machine learning and artificial intelligence or AI that imitates how humans gain certain types of knowledge. Deep learning includes statistics and predictive modeling. Deep learning makes processes quicker and simpler which is advantageous to data scientists to gather, analyze and interpret massive amounts of data. Having the fundamentals discussed, let's move into the different types of deep learning. Neural networks are the main component of deep learning. But neural networks comprise three main types which contain artificial neural networks or ANN, convolution neural networks or CNN and recurrent neural networks or RNN. Artificial neural networks are inspired biologically by the animal brain. Convolutional neural networks surpass other neural networks when given inputs such as images, voice or audio. It analyzes images by processing data. Recurrent neural networks uses sequential data or series of data. Convolutional neural networks and recurrent neural networks are used in natural language processes, speech recognition, image recognition, and many more. Machine learning. The evolution of ML started with the mathematical modeling of neural networks that served as the basis for the invention of machine learning. In 1943, neuroscientist Warren McCullik and logician Walter Pittz attempted to quantitatively map out how humans make decisions and carry out thinking processes. Therefore, the term machine learning is not new. Machine learning is a branch of artificial intelligence and computer science that uses data and algorithms to imitate how humans learn, gradually increasing the systems accuracy. There are three types of machine learning which include supervised learning. What is supervised learning? Well, here machines are trained using label data. Machines predict output based on this data. Now coming to unsupervised learning. Models are not supervised using a training data set. It is comparable to the learning process that occurs in the human brain while learning something new. And the third type of machine learning is reinforcement learning. Here the agent learns from feedback. It learns to behave in a given environment based on actions and the result of the action. This feature can be observed in robotics. Now coming to the evolution of AI, the potential of artificial intelligence wasn't explored until the 1950s. Although the idea has been known for centuries, the term artificial intelligence has been around for a decade. Still, it wasn't until British polymath Alan Turing posed the question of why machines couldn't use knowledge like humans do to solve problems and make decisions. We can define artificial intelligence as a technique of turning a computer-based robot to work and act like humans. Now, let's have a glance at the types of artificial intelligence. Weak AI performs only specific tasks like Apple's Siri, Google Assistant, and Amazon's Alexa. You might have used all of these technologies, but the types I am mentioning after this are under experiment. General AI can also be addressed as artificial general intelligence. It is equivalent to human intelligence. Hence, an AGI system is capable of carrying out any task that a human can. Strong AI aspires to build machines that are indistinguishable from the human mind. Both general and strong AI are hypothetical right now. Rigorous research is going on on this matter. There are many branches of artificial intelligence which include machine learning, deep learning, natural language processing, robotics, expert systems, fuzzy logic. Therefore, the correct answer for which is not a branch of artificial intelligence is option A, data analysis. Now that we have covered deep learning, machine learning and artificial intelligence, the final topic is data science. Concepts like deep learning, machine learning and artificial intelligence can be considered a subset of data science. Let us cover the evolution of data science. The phrase data science was coined in the early 1960s to characterize a new profession that would enable the comprehension and analysis of the massive volumes of data being gathered at the time. Since its beginnings, data science has expanded to incorporate ideas and methods from other fields, including artificial intelligence, machine learning, deep learning, and so forth. Data science can be defined as the domain of study that handles vast volumes of data using modern tools and techniques to find unseen patterns, derive meaningful information, and make business decisions. Therefore, data science comprises machine learning, artificial intelligence, and deep learning. There are a lot of areas where data science can be used. One of the very common one is fraud detection or fraud prevention. There are a lot of fraudulent activities or transactions primarily on the internet. It's very easy to commit fraud and therefore we can use data science to either prevent or detect fraud. There are certain algorithms, machine learning algorithms that can be used like for example some outlier techniques, clustering techniques that can be used to detect fraud and prevent fraud as well. So who is a data scientist rather? It is actually a very generic role that defines somebody who is working with data is known as a data scientist. But there can be very specific activities and the roles can be actually much more specific. What exactly a person does within the area of data science can be much more specific. But broadly anybody working in the area of data science is known as a data scientist. So what does a data scientist do? These are some of the activities. Data acquisition, data preparation, data mining, data modeling and then model maintenance. We will talk about each of these in a great detail but at a very high level the first step obviously is to get the raw data which is known as data acquisition. It can be all kinds of format and it could be multiple sources but obviously that raw data cannot be used as it is for performing data mining activities or data modeling activities. So the data has to be planned and prepared for using in the data models or in the data mining activity. So that is the data preparation. Then we actually do the data mining which can also include some exploratory activities. And then if we have to do stuff like machine learning then you need to build a machine learning model and test the model get insights out of it. And then if um the model is fine you deploy it and then you need to maintain the model because over a period of time it is possible that you need to tweak the model because of change in the process or change in the data and so on. So that all comes under the model maintenance. So let's take deeper look at each of these activities. Let's start with data acquisition. So the stage of data acquisition basically the data scientist will collect raw data from all possible sources. So this could be typically an RDBMS which is a relational database or it can also be a non RDBMS or could be flat files or unstructured data and so on. So we need to bring all that data from different sources if required. We need to do some kind of homogeneous formatting so that it all fits into in looks at least format from a format perspective it looks homogeneous. So that may be requiring some kind of transformation. Very often this is loaded into what is known as data warehouse. So this can also be sometimes referred to as ETL or extract transform and load. So a data warehouse is like a common place where data from different sources is brought together so that people can perform data science activities like reporting or data mining or statistical analysis and so on. So data from various sources is put in a centralized place which is known as a data warehouse. So that is also known as ETL and in order to do this there can be data scientists can take help of some ETL tools. There are some existing tools that a data scientist can take help of like for example data stage or talent or informatica. These are pretty good tools for performing these ETL activities and getting the data. The next stage now that you have the raw data into a data warehouse, you still probably are not in a position to straight away use this data for performing the data mining activities. So that is where data preparation comes into play and there are multiple reasons for that. One of them could be the data is dirty. There are some missing values and so on and so forth. So a lot of time is actually spent in this particular stage. So a data scientist spends a lot of time almost 60 to 70% of the time in this part of the project or the process which is data preparation. So there are again within this there can be multiple sub activities starting from let's say data cleaning you will probably have missing values the data there is some columns the values are missing or the values are incorrect uh there are null values and so on and so forth. So that is basically the data cleaning part of it. Then you need to perform certain transformations like for example normalizing the data and so on. Right? or you could probably have to modify a categorical values into numerical values and so on and so forth. So these are transformational activities then we may have to handle outliers. So the data could be such that there are a few values which are way beyond the normal behavior of the data for whatever reason either people have keyed in wrong values or for some reason some of the values are completely out of range. So those are known as outliers. So there are certain ways of handling these outliers and detecting and handling these outliers. So this is a part of what is known as exploratory analysis. So you quickly explore the data to find out are there. So and you can use visual tools like plots and identify what are the outliers and see how we can get rid of the outliers and so on. Then the next part could be data integrity. Data integrity is to validate for example if there are some primary keys that all the primary keys are populated there are some foreign keys then at least most of the foreign keys should be populated and otherwise when we are trying to query the data you may get wrong values and so on. So that is the data integrity part of it and then we have what is known as data reduction. Sometimes we may have duplicate values. We may have columns that may be duplicated because they are coming from different sources. The same values are there and so on. So a lot of this can be done using what is known as data reduction and thereby you can reduce the size of the data drastically because very often this could be written in data which can be removed and so on. So let's take a look at what are the various techniques that are used for data cleaning. So we need to ensure that the data is valid and it is consistent and uniform and accurate. So these are the various parameters that we need to ensure as a part of the data cleaning process. Now what are the techniques that that are used for data cleaning or uh so we will see what each of these are in this particular case and uh so what is the data set that we have? We have data about a bank and its customer details. So let's take an example and see how we go about cleaning the data. And in this particular example, we're assuming we are using Python. So let's assume we loaded this data which is the raw file CSV. This is how the customer data looks like and um we will see for example we take a closer look at the geography column. We will see that there are quite a few blank spaces. So how do we go about when we have some blank spaces or if it is a string value then we put a empty string here or we just use a space or empty string if they are numerical values then we need to come up with a strategy uh for example we put the mean value so wherever it is missing we find the mean for that particular column. So in this case, let's assume we have credit score and we see that quite a few of these values are missing. So what do we do here? We find the mean for this column for all the existing values and we found that the mean is equal to 638.6. So we kind of write a piece of code to replace wherever there are blank values. Nan is basically like null and uh we just go ahead and say fill it with the mean value. So this is a piece of code we are writing to fill it. So all the blanks or all the null values get replaced with the mean value. Now one of the reasons for doing this is that very often if you have some such situation many of your statistical functions may not even work. So that's the reason you need to fill up these values or either get rid of these records or fill up these values with something meaningful. So this is one mechanism which is basically using a mean. There are a few others. As we move forward we can see what are the other ways. For example, we can also say that any missing value in a particular row, if even one column the value is missing, you just drop that particular row or delete all rows where even a single column has missing values. So that is one way of dealing. Now the problem here can be that if a lot of data has let's say one or two columns missing and uh we drop many such rows then overall you may lose out on let's say 60% of the data has some value or the other missing 60% of the rows then it may not be a good idea to delete all the rows like in that manner because then you're losing pretty much 60% of your data therefore your analysis won't be accurate but if it is only 5 or 10% then this will work. Another way is only to drop values where or rather drop rows where all the columns are empty which makes sense because that means that record is of really no use because it has no information in it. So there can be some situations like that. So we can provide a condition saying that drop the records where all the columns are blank or not applicable. We can also specify some kind of a threshold. Let's say you have 10 or 20 columns in a row. you can specify that maybe five columns are blank or null then you drop that record. So again we need to take care that such a condition such a situation the amount of data that has been removed or excluded is not large. If it is like maybe 5% maximum 10% then it's okay but by doing this if you're losing out on a large chunk of data then it may not be a good idea. You need to come up with something better. What else we need to do next is so the data preparation part is done. So now we get into the data mining part. So what exactly we do in data mining? Primarily we come up with ways to take meaningful decisions. So data mining will give us insights into the data what is existing there and then we can do additional stuff like maybe machine learning and so on to get perform advanced analytics and so on. So the one of the first steps we do is what is known as data discovery and uh which is basically like exploratory analysis. So we can use tools like Tableau for doing some of this. So let's just take a quick look at how we go about that. So Tableau is excellent data mining or actually more of a reporting or a BI tool and you can download a trial version of Tableau at tableau.com or there is also Tableau public which is free and you can actually use and play around. However, if you want to use it for enterprise purpose then is a commercial software. So you need to purchase license and you can then run some of the data mining activities. Say your data source your data is in some Excel sheet. So you can select the source as Microsoft Excel or any other format and the data will be brought into the Tableau environment and then it will show you what is known as dimensions and uh measures. So dimensions are all the descriptive columns. So and Tableau is intelligent enough to actually identify these dimensions and measures. So measures are the numerical values. So as you can see here uh customer ID, gender, geography these are all dimensions non-numerical values whereas age, balance, credit score and so on are numeric values. So they come under measures. So you've got your data into Tableau and then you want to let's say build a small model and you want to let's say solve a particular problem. So what is the problem statement? All right, let's say we want to analyze why customers are leaving the bank which is known as uh exit and we want to analyze and see if what are some of the factors for exiting the bank and we want to let's assume consider these uh three of them like let's say gender, credit card and geography these as a criteria and analyze if these are in any way impacting or have some bearing on the customer exiting or the customer exit behavior. Okay. So let's um use Tableau and very quickly we will be able to find out how these uh parameters are affecting. All right. So let's see. So this is our customer data. So from our Excel sheet we have data set about let's say 10,000 rows and we want to find out what is the criteria. Let's start with gender. Let's say we want to first use gender as a criteria. So Tableau really offers an easy drag and drop kind of a mechanism. So that makes it really really easy to perform this kind of analysis. So what we need to do is exited says whether the customer has exited or not. So it has a value of zero and one and then of course you have gender and so on. So we will take these two and simply drag and drop. Okay. So exited and then we will put gender. And if we drag and drop into the analysis side of of Tableau. All right. So here what we are doing is we are showing male female as the two different columns here and zero for people who did not exit and one for people who exited and that is colorcoded. So the blue color means people who did not exit and uh this yellow color means people who did exit. All right. So now if we pull the data here create like bar graphs this is how it would look. Uh so what is yellow? Let's go back. So yellow is uh who exited and uh for the male only 16.45% have exited and we can also draw a reference line that will help us or even provide aliases. So these are a lot of fancy stuff that is um provided by Tableau. You can create aliases and so that it looks good rather than basic labels and you can also add a reference line. So you add a reference line something like this. From here we can make out that on an average female customers exit more than the male customers. Right? So that is what we are seeing here on an average. So we have analyzed based on gender. We do see that there is some difference in the male and female behavior. Now let's take the next criteria which is the credit card. So let's see if having a credit card has any impact on the customer exit behavior. So just like before we drag and drop the credit card has credit card column if we drag and drop here and then we will see that there is pretty much no difference between people having credit card and not having credit card. 20.81% of people who have no credit card have exited and similarly 20.18% of people who have credit card have also exited. So the credit card is not having much of an impact. That's what this piece of analysis shows. Last we will basically go and check how the geography is impacting. So once again we can drag and drop geography column onto this side. And uh if we see here there are geographies like I think there are about three geographies like France, Germany and uh Spain and um we see that there is some kind of a impact with the geography as well. Okay. So what we derive from this is that the credit card is really we can ignore the credit card variable or feature from our analysis because that doesn't have any impact but gender and geography we can keep and do further analysis. Okay. All right. So what are some of the advantages of data mining? Bit more detailed analysis can help us in predicting the future trends and it also helps in identifying customer behavior patterns. Okay. So you can take informed decisions because the data is telling you or providing you with some insights and then you take a decision based on that. If there is any fraudulent activity, data mining will help in quickly identifying such a fraud as well and of course it will also help us in identifying the right algorithm for performing more advanced data mining activities like machine learning and so on. All right. So the next activity now that we have the data we have prepared the data and perform some data mining activity the next step is model building. Let's take a look at model building. So what is model building? If we want to perform a more detailed data mining activity like maybe perform some machine learning then you need to build a model. And how do you build a model? First thing is you need to select which algorithm you want to use to solve uh the problem on hand and also what kind of data that is available and so on and so forth. So you need to make a a choice of the algorithm and based on that you go ahead and create a model train the model and so on. Now machine learning is kind of at a very high level classified into supervised and unsupervised. So if we want to predict a continuous value could be a price or a temperature or or a height or a length or things like that. So those are continuous values and if you want to find some of those then you use techniques like regression, linear regression, simple linear regression, multiple linear regression and so on. So these are the algorithms. On the other hand, there will be situations or there may be situations where you need to perform unsupervised learning. In case of unsupervised learning, you don't have any historical labeled data so to learn from. So that is when you use unsupervised learning. And uh some of the algorithms in unsupervised learning are clustering. K means clustering is the most common algorithm used in unsupervised learning. And similarly in supervised learning if you want to perform some activity on categorical values like for example it is not measured but it is counted like you want to classify whether this image is a cat or a dog whether you want to classify whether this customer will buy the product or not or you want to classify whether this email is spam or not spam. So these are examples of categorical values and uh these are examples of classification. Then you have algorithms like logistic regression, K nearest neighbor or KN&N and support vector machine. So these are some of the algorithms that are used in this case. And similarly in case of unsupervised learning if you need to perform on categorical values you have some algorithms like association analysis and hidden marco model. Okay. So in order to understand this better, let's take uh an example and uh take you through the whole process and then we will also see how the code can be written to perform this. Now let's take our example here where we want to perform a supervised learning which is basically we want to do a multilinear regression which means there are multiple independent variables and then we want to perform a linear regression to predict certain value. So in this particular example we have world happiness data. So this is a data about the happiness quotient of people from various countries and we are trying to predict and see whether our how our model will perform. So what is the question that we need to ask? First of all how to describe the data and then can we make a predictive model to calculate the happiness score. Right? So based on this we can then decide on what algorithm to use and what model to use and so on. So variables that are available or used in this model. This is a list of variables that are available. There is a happiness rank. I'll load the data and or I'll show you the data in a little bit so it becomes clear what are these. So there is what is known as happiness rank. Happiness score which is happiness score is more like a absolute value whereas rank is what is the ranking and then which country we are talking about and within that country which region and what kind of economy and whether the family which family and health details and freedom trust generosity and so on and so forth. So there are multiple variables that are available to us and uh the specific details probably are not required and there can be um in another example the variables can be completely different. So we don't have to go into the details of what exactly these variables are but it's just enough to understand that we have a bunch of these variables and now we need to use either all or some of these variables and then which we also sometimes refer to as features and then we need to build our model and train our model. All right. So let's assume we will use Python in order to perform this analysis or perform this machine learning activity and I will actually show you in our lab in in a little bit this whole thing we will run the live code but quickly I will run you through the slides and then we will go into the lab. So what are we doing here? First thing we need to do is import a bunch of libraries in Python which are required to perform our analysis. Most of these are for manipulating the data, the preparing the data and then scikitlearn or skarn is the library which we will use actually for this particular machine learning activity which is linear regression. So we have numpy, we have pandas and so on and so forth. All these libraries are imported and then we load our data and the data is in the form of a CSV file and there are different files for each year. So we have data for 2015, 16 and 17. And uh so we will load this data and then combine them, concatenate them to prepare a single data frame. And uh here we are making an assumption that you are familiar with Python. So it becomes easier if you are familiar with Python programming language or at least some programming language so that you can at least understand by looking at the code. So we are reading the file each of these files for each year and this is basically we are creating a a list of all the names of the columns we will be using later on you will see in the code. So we have loaded 2015 then 2016 and then also 2017. So we have created um three data frames and then we concatenate all these three data frames. This is what we are doing here. Then we identify which of these columns are required. Which for example some of the categorical values do we really need? We probably don't. Then we drop those columns so that we don't unnecessarily use all the columns and make the computation complicated. We can then create some plots using plotly library and it has some powerful features including creation or creation of maps and so on. just to understand the pattern the happiness quotient or how the happiness is across all the countries. So it's a nice visualization we can see each of these countries how they are in terms of their happiness score. This is the legend here. So the lighter colored countries have lower ranking and so these are the lower ranking ones and these are higher ranking which means that the ones with these dark colors are the happiest ones. So as you can see here, Australia and maybe this side uh US and so on are the happiest ones. Okay. The other thing that we need to do is the correlation between the happiness score and happiness rank. We can find a correlation using a scatter plot and we find that yes they are kind of inversely proportion which is obvious. So if the score is high, happiness score is high then they are ranked number one. For example, highest is scored as number one. So that's the idea be behind this. So the happiness score given here and the happiness rank is actually given here. So they are inversely proportional because the higher the score the the absolute value of the rank will be lower. Right? Number one has the highest value of the score and so on. So they are inversely correlated but there is a strong what this graph shows is that there is a strong correlation between happiness rank and happiness score. And then we do some more plots to visualize this. we determined that probably rank and score are pretty much conveying the same message. So we don't need both of them. So we will kind of drop one of them and uh that is what we are doing here. So we drop the happiness rank and similarly. So this is one example of how we can remove some columns which are not adding value. So we will see in the code as well how that works. Moving on, this is a correlation between pretty much each of the columns with the other columns. So this is a correlation you can plot using plot function and uh we will see here that for example happiness score and happiness score are correlated strongest correlation right because every variable will be highly correlated to itself. So that's the reason so the darker the color is the higher the correlation and as so the and correlation in numerical terms goes from 0 to one. So one is the highest value and it can only be between 0 and one. Correlation between two variables can be only have a value between 0 and one. So the numerical value can go from 0 to one and one here is dark color and zero is kind of dark but it is blue color. From red it goes down the dark blue color indicates pretty much no correlation. So the from this heat map we see that happiness and economy and family are probably also health probably are the most correlated and then it keeps decreasing after freedom kind of keeps decreasing and coming to pretty much uh zero. All right. So that is a correlation graph and then we can probably use this to find out which are the columns that need to be dropped which do not have very high correlation and uh we take only those columns that we will need. So this is the code for dropping some of the columns. Once we have prepared the data when we have the required columns then we use scikitlearn to actually split the data. First of all, this is a normal machine learning process. You need to split the data into training and test data set. In this case, we are splitting into 80/20. So 80 is the training data set and 20 is the test data set. So that's what we are doing here. So we use train test split method or function. So you have all your training data in X_ra, the labels in Y train. Similarly, x test has the test data the inputs whereas the labels are in y test. So that's how and this value whether it is 8020 or 50/50 that is all individual preference. So in our case we are using 8020. All right. And uh then the next is to create a linear regression instance. So this is what we are doing. We're creating an instance of linear regression and then we train the model using the fit function and uh we are passing x and y which is the x value and the label data regular input and the label data label information. Then we do the test we run the or we perform the evaluation on the test data set. So this is what we are doing with the test data set and then we will evaluate how accurate the model is and using the scikit land functionality itself. We can also see what are the various parameters and what are the various coefficients because in linear regression you will get like a equation of like a straight line y is equal to beta 0 plus beta 1 x1 plus beta 2 x2 those beta 1 beta 2 beta 3 are known as the coefficients and beta 0 is the intercept. After the training you can actually get these information of the model what is the intercept value what are the coefficients and so on by using these functions. So let's take quickly go into the lab and take a look at our code. Okay. So this is my lab. This is my Jupyter notebook where the code I have the actual code and I will take you through this code to run this linear regression on the world happiness data. So we will import a bunch of libraries numpy, pandas, plot, plotly and so on also. Yeah, scikitle learn that's also very important. So that's the first step. Then I will import my data and uh the data is in three parts. There are three files one for each year 2015, 2016 and 2017. And it is a CSV file. So I've imported my data. Let's take a look at the data quickly. Glance at data. So this is how it looks. So we have the country, region, happiness rank and then happiness score. There are some standard errors and then what is the per capita family and so on. So and then we will keep going. We will create a list of all these column names we will be using later. So for now just we I will run this code. No need of major explanation at this point. We know that some of these columns probably are not required. So you can use this drop functionality to remove some of the columns which we don't need like for example region and standard error will not be contributing to our model. So we will basically drop those values out here. So we use the drop and then we created a vector with these names column names that's what we are passing here. Instead of giving the names of the columns here we can pass a vector. So that's what we are doing. So this will drop from our data frame. It will remove region and standard error these two columns. Then the next step we will read the data for 2016 and also 2017 and then we will concatenate this data. So let's do that. So we have now data frame called happiness which is a concatenation of both all the three files. Let's take a quick look at the data now. So most of the unwanted columns have been removed and you have all the data in one place for all the three years. And this is how the data looks. And if you want to take a a look at the summary of the columns, you can say describe and uh you will get this information. For example, for each of the columns, what is the count? What's what is the mean value? Standard deviation, especially the numeric values, okay, not the categorical values. So this is a quick way to see how the data is and uh initial little bit of exploratory analysis can be done here. So what is the maximum value? What's the minimum value and so on for each of the columns. All right. So then we go ahead and create some visualizations using plotly. So let us go and build a plot. So if we see here now this is the relation correlation between happiness rank and happiness score. This is what we have seen in the slides as well. We can see that there is a tight correlation between them. Only thing is it is inverse correlation but otherwise they are very tightly correlated which also says that they both probably provide the same information. So there is no not much of value add. So we'll go ahead and drop the happiness rank as well from our columns. So that's what we're doing here. And now we can do the creation of the correlation heat map. Let us plot the correlation heat map to see how each of these columns is correlated to the others and we as we have seen in the slides. This is how it looks. So happiness score is very highly correlated. So this is the legend we have seen in the slide as well. So blue color indicates pretty much zero or very low correlation. Deep red color indicates very high correlation. And the value correlation is a numeric value and the value goes from 0 to one. If the two items or two features or columns are highly correlated then there will be as close to one as possible and two columns that are not at all correlated will be as close to zero as possible. So that's how it is. For example here happiness score and happiness score every column or every feature will be highly correlated to itself. So it is like between them there will be correlation value will be one. So that's why we see deep red color. But then others are for example with higher values are economy and then health and then maybe family and freedom. So these are generosity and trust are not very highly correlated to happiness score. So that is uh one quick exploratory analysis we can do and uh therefore we can drop the country and the happiness rank because they also again don't have any major impact on the analysis on our analysis. So now we have prepared our data. There was no need to clean the data because the data was clean. But if there were some missing values and so on as we have discussed in the slides, we would have had to perform some of the data cleaning activities as well. But in this case, the data was clean. All we needed to do was just the preparation part. So we removed some unwanted columns and we did some exploratory data analysis. Now we are ready to perform the machine learning activity. So we use scikitlearn for doing the machine learning. Scikitlearn is Python library that is available for performing our uh machine learning. Once again we will import some of these libraries like pandas and numpy and also scikitlearn. First step we will do is split the data in 2080 format. So you have all the test data which is 20% of the data is test data and 80% is your training data. So this test size indicates how much of it is in the what is the size of the test data. The remaining which is here we are saying 02 therefore that means training is8. So training data is 80%. All right. So we have executed that split the data and now we create an instance of the linear regression model. So lm is our linear regression model and we pass x and y the training data set and call the function fit so that the model gets trained. So now once that is done training is done, training is completed and now what we have to do is we need to predict the values for the test data. So the next step is using so you see here fit will basically run the training method. Predict will actually predict the values. So we are passing the input values which is the independent variables and we are asking for the values of the dependent variable which is which we are capturing in y prime and we use the predict method here lm.predict. So this will give us all the predicted y values and remember we already have y test has the actual values which are the labels so that we can use these two to compare and find out how much of it is error. So that's what we are doing here. We are trying to find the difference between the predicted value and the actual value. Y test is the actual value for the test data and Y predict is the predicted value. We just found out the predicted value. So we will run that and we can do a quick check as to how the data looks how is the difference. So in some cases it is positive, some cases it is negative but in most of the cases I think the difference is very small. This is exponential to the power of 0 - 04 and so on. So looks like our model has performed reasonably well. We can now check some of the parameters of our model like the intercept and the coefficients. So that's what we are doing here. So these are the coefficients of the various parameters that we or the coefficients of the various independent variables. Okay. So these are the values. Then we can quickly go ahead and list them down as well against the corresponding independent variables. So the coefficients against the corresponding independent variables. So 1.0051 051 is the coefficient for economy. N9983 is for family, coefficient for family and health and so on and so forth. Right? So that's what this is showing. Now we can use the functionality readily available functionality of scikitlearn and then plot that to find some of the parameters which determine the accuracy of this model like for example what is the mean square error and so on. So that's what we are doing here. So let's just go ahead and run this. So you can see here that the root mean square error is pretty low which is a good sign and uh which is a one of the measures of u how well our model is performing. We can do one more quick plot to just see how the actual values and the predicted values are looking. And once again you can see that as we have seen from the root mean square error root mean square error is very very low. That means that the actual values and the predicted values are pretty much matching up almost matching up. And this plot also shows the same. So this line is going through the predicted values and the actual values and the difference is very very low. So again this is actual data. This is one example where the the accuracy is high and the predicted values are pretty much matching with the actual values. But in real life you may find that these values are slightly more scattered and you may get the error value can be relatively on the higher side. The root mean square error. Okay. So this was a good and quick example of uh the code to perform data science activity or a machine learning or data mining activity. In this case we did what is known as linear regression. So let's go back to our slides and see what else is there. So we saw this these are the coefficients of each of the features in our code and uh we have seen the root mean square error as well and uh with we can take a few hundred countries certain values and actually predict to see if how the model is performing and I think we have done this as well and in this case as we have seen pretty much the predicted values and the actual values are pretty much matching which means our model is almost 100% accurate as I mentioned real life it may not be the case but in this particular case we have got a pretty good model which is very good also subsequently we can assume that this is how the equation in linear regression the model is nothing but an equation like y is equal to beta 0 plus beta 1 x1 plus beta 2 x2 plus beta 3 x3 and so on. So this is what we are showing here. So this is our intercept which is beta 0 and then we have beta 1 into economy value, beta 2 into the family value, beta 3 into health value and so on. So that is what is shown here. Okay. So I think the next step once we have the results from the data mining or machine learning activity, the next step is to communicate these results to the appropriate stakeholders. So that is what we will see here now. So how do we communicate? Usually you take these results and then either prepare a presentation or put it in a document and then show them these actionable results orable insights and uh you need to find out who are your target audience and uh put all the results in context and uh maybe if there was a problem statement you need to put this results in the context of the problem statement. what was our initial goal that we wanted to achieve. So that we need to communicate here based on you remember we started off with what is the question and what is the data and so on and then what is the answer. So we we need to put the results and then what is the methodology that we have used all that has to be put and clearly communicated in business terms so that the people understand very well from a business perspective. So once the model building is done, once the results are published and communicated, the last part is maintenance of this model. Now very often what can happen is the model may have to be subsequently updated or modified because of multiple reasons. Either the the data has changed, the way the data comes has changed or the process has changed or for whatever reason the accuracy may keep changing. Once you have trained the model the for example we got a very high accuracy but then over a period of time there can be various factors which can cause that. So from time to time we need to check whether the model is performing well or not. The accuracy needs to be tested once in a while and if required you may have to rebuild or retrain the model. So you do the assessment, you you see if it needs any tweaks or changes and then if it is required you need to probably retrain the model with the latest data that you have and then you deploy it. You build the model, train it and then you deploy it. So that is like the maintenance cycle that you may have to take the model. Data analyst versus data engineer versus data scientist. Which one to choose? This is one of the most popular questions asked by learners looking for a career in data and analytics. I'm sure you too would have come across these job roles in the ever growing data science landscape. Though they all deal with data, these jobs are not the same. There are significant differences between what a data analyst, data engineer, and a data scientist does. We will look at these job roles and the differences in detail. First, let's look at some data analytics and data science trends. The analytics and data science market is thriving. Data analytics, data engineering, and data science are the key trends in today's exhilarating market. As per statist.com, the global big data analytics market revenue will grow at a caggr of 30% with revenue reaching over 68 billion US by 2025. According to Technavio, the enterprise data management market is expected to increase by 64.08 billion US by 2025 as per markets and markets.com. The big data market size is projected to grow from 162.6 billion US in 2021 to $273.4 billion US in 2026. Now, another report from research drive says that the data science platform market is estimated to reach 224.3 billion US by 2026. So, with so much data available and companies making huge investments to drive business insights, the job opportunities for data analysts, data engineers, and data scientists are going to increase in 2022 and over the coming years. Now let's learn the major differences between data analyst versus data engineer versus data scientist. So who are they? A data analyst analyzes and interprets vast volumes of data in order to extract meaningful information out of it. They find solutions to a business problem and make critical business decisions. The insights provided by data analysts are important to companies that want to understand the needs of their end customers. But talking about who a data engineer is, a data engineer on the other hand builds infrastructure and scalable pipelines to manage the flow of data and prepare it for analysis. So basically they optimize the systems that enable data analysts and data scientists to perform their job efficiently. Data scientists are professionals who analyze and visualize existing data and use algorithms to build predictive models for making future decisions. They also engage with business leaders to understand their needs and present complex findings. With that, let's look at the primary roles and responsibilities of these three job roles. Data analysts are responsible to collect, clean, store and process data. They discover hidden patterns from data by performing exploratory data analysis and visualize data by creating charts and graphs. Acquiring data from primary and secondary sources is one of their key tasks. They build reports and dashboards and also maintain databases. Now talking about the roles and responsibilities of a data engineer. A data engineer performs data acquisition. The design, build and test data as well to develop and maintain data architecture. Data engineers are tasked with testing, integrating, managing and optimizing data from a variety of sources. So they integrate data into existing data pipelines, prepare data for modeling and perform various ETL operations. Now talking about the roles and responsibilities of a data scientist. So data scientists develop machine learning models to identify trends in data for making decisions. They develop hypothesis and use their knowledge of statistics, data visualization and machine learning to forecast the future for the business. Data scientists visualize data and use storytelling techniques and also write programs to automate data collection and processing. Now move on to the skills possessed by data analysts, data engineers, and data scientists. To become a data analyst, you need to have good hands-on experience with writing SQL queries. You should have excellent Microsoft Excel skills for analyzing data. Data analysts are also good at programming and they need to know how to visualize data, solve business problems, and possess domain knowledge. Data engineers should have a solid understanding of SQL, MongoDB and programming. They need to have a good command of data architecture, scripting, data warehousing and ETL. Data engineers are also good at Hadoop based analytics. Now talking about the skills for a data scientist. So a data scientist should have experience with programming in Python and R. They should have a very good understanding of mathematics and statistics as well. Data scientists need to possess analytical thinking and data visualization skills as well. Machine learning, deep learning, and decision-m are other critical skills every data scientist should have. Now we look at the salaries of a data scientist, a data analyst as well as a data engineer. So a data analyst in the United States earns over $70,000 peranom while in India a data analyst can earn nearly 7 lak 25,000 rupees peranom. A data engineer in the United States can earn over $112,500 per year and in India you can earn over 9 lakh rupees peranom. Talking about the salary of a data scientist, a data scientist in the United States earns over $117,000 peranom and in India, a data scientist can earn over 11 lakh rupees peranom. Coming to the final section of this video, we'll look at the top companies hiring for data analysts, data engineers and data scientists. So we have the first company as Google. Then we have Tesla. Next we have the e-commerce giant Amazon, the internet giant Facebook or the social media giant Facebook. We have the tech giant Oracle. We also have Verizon and Airbnb. So these are some of the top companies that hire for the three roles. If you are now let's talk about the life cycle of a data science project. Okay. The first step is the concept study. In this step, it involves understanding the business problem, asking questions, get a good understanding of the business model, meet up with all the stakeholders, understand what kind of data is available and all that is a part of the first step. So here are a few examples. We want to see what are the various specifications and then what is the end goal? What is the budget? Is there an example of this kind of a problem that has been maybe solved earlier? So all this is a part of the concept study and another example could be a very specific one to predict the price of a 1.35 karat diamond and there may be relevant information inputs that are available and we want to predict the price. The next step in this process is data preparation. data gathering and data preparation also known as data munching or sometimes it is also known as data manipulation. So what happens here is the raw data that is available may not be usable in its current format for various reasons. So that is why in this step a data scientist would explore the data. He will take a look at some sample data. Maybe pick there are millions of records. Pick a few thousand records and see how the data is looking. Are there any gaps? Is the structure appropriate to be fed into the system. Are there [clears throat] some columns which are probably not adding value? May not be required for the analysis. Very often these are like names of the customers. They will probably not add any value or much value from an analysis perspective. the structure of the data. Maybe the data is coming from multiple data sources and the structures may not be matching. What are the other problems? There may be gaps in the data. So the data all the columns all the cells are not filled. If you're talking about structured data, there are several blank records or blank columns. So if you use that data directly, you'll get errors or you'll get inaccurate results. So how do you either get rid of that data or how do you fill this gaps with something meaningful? So all that is a part of data munching or data manipulation. So these are some additional subtopics within that. So data integration is one of them. If there are any conflicts in the data there may be data may be redundant. Yeah data res redundancy is another issue. There may be you have let's say data coming from two different systems and both of them have customer table for example or customer information. So when you merge them there is a duplication issue. So how do we resolve that? So that is one data transformation. As I said there will be situations where data is coming from multiple sources and then when we merge them together they may not be matching. So we need to do some transformations to make sure everything is similar. We may have to do some data reduction. If the data size is too big, you may have to come up with ways to reduce it meaningfully without losing information. Then data cleaning. So there will be either wrong values or you null values or there are missing values. So how do you handle all of that? A few examples of very specific stuff. So there are missing values. How do you handle missing values or null values? Here in this particular slide we are seeing three types of issues. One is missing value then you have null value. You see the difference between the two right? So in the missing value there is nothing blank. Null value [clears throat] it says null. Now the system cannot handle if there are null values. Similarly there is improper data. So it's supposed to be numeric value but there is a string or a non-numeric value. So how do we clean and prepare the data so that our system can work flawlessly. So there are multiple ways and and there is no one common way of doing this. It can vary from project to project. It can vary from what exactly is the problem you're trying to solve. It can vary from data scientist to data scientist, organization to organization. So these are like some standard practices people come up with and and [music] of course there will be a lot of trial and error. Somebody would have tried out something and it worked and it'll continue to use that mechanism. So that's how we need to take care of data cleaning. Now what are the various ways of doing you know if if values are missing how do you take care of that? Now if the data is too large and um only a few records have some missing values then it is okay to just get rid of those entire rows for example. So if you have a million records and out of which 100 records don't have full data. So there are some missing values in about 100 records. So it's absolutely fine because it's a small percentage of the data. So you can get rid of the entire records which have missing values. But that's not a very common situation. Very often you will have multiple or at least you know large number of data set. For example out of million records you may have 50,000 records which are like having missing values. Now that's a significant amount. You cannot get rid of all those records. Your analysis will be inaccurate. So how do you handle such situations? So there are again multiple ways of doing it. One is you can probably if a particular values are missing in a particular column you can probably take the mean value for that particular column and fill all the missing values with the mean value so that first of all you don't get errors because of missing values and second you don't get results that are way off because these values are completely different from what is there. So that is one way. Then a few other could be either taking the median value or depending on what kind of data we are talking. So something meaningful we will have put in there. If we are doing some machine learning activity then obviously as a part of data preparation you need to split the data into training and test data set. The reason being if you try to test with a data set which the system has already seen as a part of training then it will tend to give reasonably accurate results because it has already seen that data and that is not a good measure of the accuracy of the system. So typically you take the entire data set the input data set and split it into two parts and again the ratio can vary from person to person individual preferences. Some people like to split it into 50/50. Some people like it as 63.33 and 33.3. This is basically 2/3 and 1/3. And some people do it as 80/20. 80 for training and 20 for testing. So you split the data, perform the training with the 80% and then use the remaining 20% for testing. All right. So that is one more data preparation activity that needs to be done before you start analyzing or applying the data or putting the data through the model. Then the next step is model planning. Now this models can be statistical models. This could be machine learning models. So you need to decide what kind of models you're going to use. Again it depends on what is the problem you're trying to solve. So if it is a regression problem, you need to think of a regression algorithm and come up with a regression model. So it could be linear regression. Or if you're talking about classification, then you need to pick up an appropriate classification algorithm like logistic regression or decision tree or SVM and then you need to train that particular model. So that is the model building or model planning process and the cleaned up data has to be fed into the model. And apart from cleaning you may also have to in order to determine what kind of model you will use you have to perform some exploratory data analysis to understand the relationship between the various variables and u see if the data is appropriate and so on. Right? So that is the additional preparatory step that needs to be done. So little bit of details about exploratory data analysis. So what exactly is exploratory data analysis? It's basically to as the name suggests you're just exploring you just receive the data and you're trying to explore and uh find out what are the data types and what is the is the data clean in in each of the columns what is the maximum minimum value. So for example there are out ofthe-box functionality available in tools like R. So if you just ask for a summary of the table, it will tell you for each column it will give some details as to what is the mean value, what is the maximum value and so on and so forth. So this exercise or this exploratory analysis is to get an understanding of your data and then you can take steps to during this process you find there are a lot of missing values you need to take steps to fix those. You will also get an idea about what kind of model to be used and so on and so forth. What are the various techniques used for exploratory data analysis? Typically these would be visualization techniques like you use histograms. Uh you can use box plots, you can use scatter plots. So these are very quick ways of identifying the patterns or a few of the trends of the data and so on. And then once your data is ready, you you've decided on the model, what kind of model, what kind of algorithm you're going to use. If you're trying to do machine learning, you need to pass your 80% the training data or rather you use that training data to train your model. And the training process itself is iterative. So the training process you may have to perform multiple times and once the training is done and you feel it is giving good accuracy then you move on to test. So you take the remaining 20% of the data. Remember we split the data into training and test. So the test data is now used to check the accuracy or how well our model is performing and if if there are further issues let's say and model is still during testing if the accuracy is not good then you may want to retrain your model or use a different model. So this whole thing again can be iterative but if the test process is passed or if the model passes the test then it can go into production and it will be deployed. All right. So what are the various tools that we use for model planning? R is an excellent tool in a lot of ways. Whether you're doing regular statistical analysis or machine learning or any of these activities or in along with our studio provides a very powerful environment to do data analysis including visualization. It has a very good integrated visualization or plot [clears throat] mechanism which can be used for doing exploratory data analysis and then later on to do analysis detail analysis and machine learning and so on and so forth. Then of course you can write Python programs. Python offers a rich library for performing data analysis and machine learning and so on. MATLAB is a very popular tool as well especially during education. So this is a very easy to learn tool. So MATLAB is another uh tool that can be used. And then last but not least SAS. SAS is again very powerful. It is a preparatory tool and it has all the components that are required to perform very good statistical analysis or perform data science. So those are the various tools that would be required for or that that can be used for model building. And uh so the next step is model building. So we have done the planning part. We said okay what is the algorithm we going to use? What kind of model we going to use? Now we need to actually train this model or build the model rather so that it can then be deployed. So what are the various uh ways or what are the various types of model building activities. So it could be let's say in this particular example that we have taken you want to find out the price of 1.35 karat diamond. So this is let's say a linear regression problem. You have data for various carats of diamond and you use that information you pass it through a linear regression model or you create a linear regression model which can then predict your price for 1.35 carat. So this is one example of model building and then little bit details of how linear regression works. So linear regression is basically coming up with a relation between an independent variable and a dependent variable. So it is pretty much like coming up with equation of a a straight line which is the best fit for the given data. So like for example here y is equal to mx + c. So y is the dependent variable and x is the independent variable. We need to determine the values of m and c for our given data. So that is what the training process of uh this model does. At the end of the training process, you have a certain value of m and c and um that is used for predicting the values of any new data that comes. All right. So the way it works is we use the training and the test data set to train the model and then validate whether the model is working fine or not using test data and uh if it is working fine then it is taken to the next level which is put in production. If not the model has to be retrained. If the accuracy is not good enough then the model is retrained maybe with more data or you come up with a newer model or algorithm and then repeat that process. So it is an iterative process. Once the training is completed training and test then this model is deployed and we can use this particular model to determine what is the price of 1.35 karat diamond. Remember that was our problem statement. So now that we have the best fit for this given data, we have the price of 1.35 karat diamond which is 10,000. So this is one example of how this whole process works. Now how do we build the model? There are multiple ways. You can use Python for example and use libraries like pandas or numpy to build the model and implement it. This will be available as a separate tutorial, a separate video in this playlist. So stay tuned for that. Moving on, once we have the results, the next step is to communicate this results to the appropriate stakeholders. So which is basically taking this results and preparing like a presentation or a dashboard and communicating these results to the concerned people. So finishing or getting the results of the analysis is not the last step. But you need to as a data scientist take this results and present it to the team that has given you this problem in the first place and explain your findings explain the findings of this exercise and recommend maybe what steps they need to take in order to overcome this problem or solve this problem. So that is the pretty much once that is accepted and the last step is to operationalize. So if everything is fine your data scientists presentations are accepted then they put it into practice and thereby they will be able to improve or solve the problem that they stated in step one. Okay. So quick summary of the life cycle. You have a concept study which is basically understanding the problem asking the right questions and trying to see if there is uh enough data to solve this problem and then even maybe gather the data. Then data preparation the raw data needs to be manipulated. You need to do data munching so that you have the data in a certain proper format to be used by the model or our analytics system. And then you need to do the model planning. What kind of a model? what algorithm you will use for a given problem and then the model building. So the exact execution of that model happens in step four and you implement and execute that model and uh put the data through the analysis in this step and then you get the results. These results are then communicated packaged and presented and communicated to the stakeholders and once that is accepted that is operationalized. So that is the final step. Let's begin this lesson by defining the term statistics. Statistics is a mathematical science pertaining to the collection, presentation, analysis, and interpretation of data. It's widely used to understand the complex problems of the real world and simplify them to make well-informed decisions. Several statistical principles, functions, and algorithms can be used to analyze primary data, build a statistical model, and predict the outcomes. An analysis of any situation can be done in two ways. Statistical analysis or a non-statistical analysis. Statistical analysis is the science of collecting, exploring, and presenting large amounts of data to identify the patterns and trends. Statistical analysis is also called quantitative analysis. Non-statistical analysis provides generic information and includes text, sound, still images, and moving images. Non-statistical analysis is also called qualitative analysis. Although both forms of analysis provide results, statistical analysis gives more insight and a clearer picture, a feature that makes it vital for businesses. There are two major categories of statistics, descriptive statistics and inferial statistics. Descriptive statistics helps organize data and focuses on the main characteristics of the data. It provides a summary of the data numerically or graphically. Numerical measures such as average, mode, standard deviation or SD and correlation are used to describe the features of a data set. Suppose you want to study the height of students in a classroom. In the descriptive statistics, you would record the height of every person in the classroom and then find out the maximum height, minimum height, and average height of the population. Inferial statistics generalizes the larger data set and applies probability theory to draw a conclusion. It allows you to infer population parameters based on the sample statistics and to model relationships within the data. Modeling allows you to develop mathematical equations which describe the inter relationships between two or more variables. Consider the same example of calculating the height of students in the classroom. In inferial statistics, you would categorize height as tall, medium, and small, and then take only a small sample from the population to study the height of students in the classroom. The field of statistics touches our lives in many ways. From the daily routines in our homes to the business of making the greatest cities run, the effect of statistics are everywhere. There are various statistical terms that one should be aware of while dealing with statistics. population, sample variable, quantitative variable, qualitative variable, discrete variable, continuous variable. A population is the group from which data is to be collected. A sample is a subset of a population. A variable is a feature that is characteristic of any member of the population differing in quality or quantity from another member. A variable differing in quantity is called a quantitative variable. For example, the weight of a person, number of people in a car. A variable differing in quality is called a qualitative variable or attribute. For example, color, the degree of damage of a car in an accident. A discrete variable is one which no value can be assumed between the two given values. For example, the number of children in a family. A continuous variable is one in which any value can be assumed between the two given values. For example, the time taken for a 100 meter run. Typically, there are four types of statistical measures used to describe the data. They are measures of frequency, measures of central tendency, measures of spread, measures of position. Let's learn each in detail. Frequency of the data indicates the number of times a particular data value occurs in the given data set. The measures of frequency are number and percentage. Central tendency indicates whether the data values tend to accumulate in the middle of the distribution or toward the end. The measures of central tendency are mean, median, and mode. Spread describes how similar or varied the set of observed values are for a particular variable. The measures of spread are standard deviation, variance and quartiles. The measure of spread are also called measures of dispersion. Position identifies the exact location of a particular data value in the given data set. The measures of position are percentiles, quartortiles and standard scores. Statistical analysis system or SAS provides a list of procedures to perform descriptive statistics. They are as follows. Proc print, proc contents, proc means, proc frequency, proc univariant, proc gart, proc box plot, proc g-plot, proc print, it prints all the variables in a SAS data set. Proc contents, it describes the structure of a data set. Proc means it provides data summarization tools to compute descriptive statistics for variables across all observations and within the groups of observations. Proc frequency it produces oneway to inway frequency and crosstabulation tables. Frequencies can also be an output of a SAS data set. PROC univariat. It goes beyond what proc means does and is useful in conducting some basic statistical analyses and includes highresolution graphical features. PROC G- chart. The G-chart procedure produces six types of charts. Block charts, horizontal vertical bar charts, pie doughut charts, and star charts. These charts graphically represent the value of a statistic calculated for one or more variables in an input SAS data set. The variables can be either numeric or character. Proc box plot. The box plot procedure creates sidebyside box and whisker plots of measurements organized in groups. A box and whisker plot displays the mean, quartiles, and minimum and maximum observations for a group. Proc GPlot G-plot procedure creates two-dimensional graphs including simple scatter plots, overlay plots in which multiple sets of data points are displayed on one set of axis, plots against the second vertical axis, bubble plots, and logarithmic plots. In this demo, you'll learn how to use descriptive statistics to analyze the mean from the electronic data set. Let's import the electronic data set into [snorts and clears throat] the SAS console. In the left plane, rightclick the electronic.xlsx data set and click import data. The code to import the data generates automatically. Copy the code and paste it in the new window. The proc means procedure is used to analyze the mean of the imported data set. The keyword data identifies the input data set. In this demo, the input data set is electronic. The output obtained is shown on the screen. Note that the number of observations, mean, standard deviation, and maximum and minimum values of the electronic data set are obtained. This concludes the demo on how to use descriptive statistics to analyze the mean from the electronic data set. So far you have learned about descriptive statistics. Let's now learn about inferial statistics. Hypothesis testing is an inferial statistical technique to determine whether there is enough evidence in a data sample to infer that a certain condition holds true for the entire population. To understand the characteristics of the general population, we take a random sample and analyze the properties of the sample. We then test whether or not the identified conclusions correctly represent the population as a whole. The population of hypothesis testing is to choose between two competing hypotheses about the value of a population parameter. For example, one hypothesis might claim that the wages of men and women are equal, while the other might claim that women make more than men. Hypothesis testing is formulated in terms of two hypotheses. Null hypothesis, which is referred to as Hnull. Alternative hypothesis, which is referred to as H1. The null hypothesis is assumed to be true unless there is strong evidence to the contrary. the alternative hypothesis is assumed to be true when the null hypothesis is proven false. Let's understand the null hypothesis and alternative hypothesis using a general example. Null hypothesis attempts to show that no variation exists between variables and alternative hypothesis is any hypothesis other than the null. For example, say a pharmaceutical company has introduced a medicine in the market for a particular disease and people have been using it for a considerable period of time and it's generally considered safe. If the medicine is proved to be safe, then it is referred to as null hypothesis. To reject null hypothesis, we should prove that the medicine is unsafe. If the null hypothesis is rejected, then the alternative hypothesis is used. Before you perform any statistical tests with variables, it's significant to recognize the nature of the variables involved. Based on the nature of the variables, it's classified into four types. They are categorical or nominal variables, ordinal variables, interval variables, and ratio variables. Nominal variables are ones which have two or more categories, and it's impossible to order the values. Examples of nominal variables include gender and blood group. Ordinal variables have values ordered logically. However, the relative distance between two data values is not clear. Examples of ordinal variables include considering the size of a coffee cup, large, medium, and small, and considering the ratings of a product, bad, good, and best. Interval variables are similar to ordinal variables except that the values are measured in a way where their differences are meaningful. With an interval scale, equal differences between scale values do have equal quantitative meaning. For this reason, an interval scale provides more quantitative information than the ordinal scale. The interval scale does not have a true zero point. A true zero point means that a value of zero on the scale represents zero quantity of the construct being assessed. Examples of interval variables include the Fahrenheit scale used to measure temperature and distance between two compartments in a train. Ratio scales are similar to interval scales in that equal differences between scale values have equal quantitative meaning. However, ratio scales also have a true zero point which give them an additional property. For example, the system of inches used with a common ruler is an example of a ratio scale. There is a true zero point because 0 in does in fact indicate a complete absence of length. In this demo, you'll learn how to perform the hypothesis testing using SAS. In this example, let's check against the length of certain observations from a random sample. The keyword data identifies the input data set. The input statement is used to declare the aging variable and cards to read data into SAS. Let's perform a t test to check the null hypothesis. Let's assume that the null hypothesis to be that the mean days to deliver a product is 6 days. So null hypothesis equals 6. Alpha value is the probability of making an error which is 5% standard and hence alpha equals 0.05. The variable statement names the variable to be used in the analysis. The output is shown on the screen. Note that the p value is greater than the alpha value which is 0.05. Therefore, we fail to reject the null hypothesis. This concludes the demo on how to perform the hypothesis testing using SAS. Let's now learn about hypothesis testing procedures. There are two types of hypothesis testing procedures. They are parametric tests and non-parametric tests. In statistical inference or hypothesis testing, the traditional tests such as test test and ANOVA are called parametric tests. They depend on the specification of a probability distribution except for a set of free parameters. In simple words, you can say that if the population information is known completely by its parameter, then it is called a parametric test. If the population or parameter information is not known and you are still required to test the hypothesis of the population, then it's called a non-parametric test. Non-parametric tests do not require any strict distributional assumptions. There are various parametric tests. They are as follows. TA test, ANOVA, chai squared, linear regression. Let's understand them in detail. Test. A t test determines if two sets of data are significantly different from each other. The test test is used in the following situations. To test if the mean is significantly different than a hypothesized value. To test if the mean for two independent groups is significantly different. To test if the mean for two dependent or paired groups is significantly different. For example, let's say you have to find out which region spends the highest amount of money on shopping. It's impractical to ask everyone in the different regions about their shopping expenditure. In this case, you can calculate the highest shopping expenditure by collecting sample observations from each region. With the help of the t test, you can check if the difference between the regions are significant or a statistical fluke. ANOVA ANOVA is a generalized version of the t test and used when the mean of the interval dependent variable is different to the categorical independent variable. When we want to check variance between two or more groups, we apply the ANOVA test. For example, let's look at the same example of the t test example. Now you want to check how much people in various regions spend every month on shopping. In this case, there are four groups, namely east, west, north, and south. With the help of the ANOVA test, you can check if the difference between the regions is significant or a statistical fluke. Chi square. Chiquare is a statistical test used to compare observed data with data you would expect to obtain according to a specific hypothesis. Let's understand the Chiquare test through an example. You have a data set of male shoppers and female shoppers. Let's say you need to assess whether the probability of females purchasing items of $500 or more is significantly different from the probability of males purchasing items of $500 or more. Linear regression. There are two types of linear regression. Simple linear regression and multiple linear regression. Simple linear regression is used when one wants to test how well a variable predicts another variable. Multiple linear regression allows one to test how well multiple variables or independent variables predict a variable of interest. When using multiple linear regression, we additionally assume the predictor variables are independent. For example, finding relationship between any two variables, say sales and profit is called simple linear regression. Finding relationship between any three variables, say sales, cost, telemarketing is called multiple linear regression. Some of the non-parametric tests are will coxin rank sum test and crus Wallace H test. Will coxin rank sum test. The Woxin signed rank test is a nonparametric statistical hypothesis test used to compare two related samples or matched samples to assess whether or not their population mean ranks differ. In woxin rank sum test, you can test the null hypothesis on the basis of the ranks of the observations. Cruscoll Wallace H test. Cruscoll Wallace H test is a rank-based non-parametric test used to compare independent samples of equal or different sample sizes. In this test, you can test the null hypothesis on the basis of the ranks of the independent samples. The advantages of parametric tests are as follows. Provide information about the population in terms of parameters and confidence intervals. easier to use in modeling, analyzing, and for describing data with central tendencies and data transformations. Express the relationship between two or more variables. Don't need to convert data into rank order to test. The disadvantages of parametric tests are as follows. Only support normally distributed data. Only applicable on variables, not attributes. Let's now list the advantages and disadvantages of non-parametric tests. The advantages of non-parametric tests are as follows. Simple and easy to understand. Do not involve population parameters and sampling theory. Make fewer assumptions. Provide results similar to parametric procedures. The disadvantages of non-parametric tests are as follows. Not as efficient as parametric tests, difficult to perform operations on large samples manually. We'll discuss the types of distribution in statistics. But before we move ahead, let's have a brief introduction on what is probability distribution. A probability distribution is a list of all of the possible outcomes of a random variable along with the corresponding probability values. And it is used in many fields, but we rarely do explain what they are. So in this video we'll discuss the three main types of probability distribution that is normal, binomial and poison distribution. So let's move ahead. So what is normal distribution? Normal distribution is a continuous probability density that has a probability density function which gives us a symmetrical bell curve. Now data can be distributed or spread out in different ways but there are many cases where the data tends to be around a central value with no bias to the left or right which means that it doesn't show any particular spikes towards the left or the right and it gets close to a normal distribution. Half of the data will fall on the left of the mean and the other half will fall on the right. Now let's take a look at a graph which shows the height distribution in a class. As you can see, the average height is in the middle and the data to the left of the average height represents the short people and the data to the right of it represents the taller people. The y-axis shows us the likelihood of any of these heights occurring. The average height has the most distribution or it has the most number of cases in the class. And as the height decreases or increases the number of people who have that height also decreases. This kind of a distribution is called a normal distribution where the average or the mean is always the highest point and any other point after that or before that is significantly lower. The resulting data gives us a bell curve. And as you can see there is no abrupt bias or spike in the data anywhere except for the average height. So this kind of a curve is called a bell curve and it's usually seen in a normal distribution. The reason we call this a normal distribution is because the data is normally distributed with the average being the highest and all the other data points having a lower likelihood. Now we came across two terms which are associated with normal distribution. Continuous probability density and probability density function. What is continuous probability density? Continuous probability density is a probability distribution where the random variable X can take any given value. Because there are infinite values that X could assume. The probability of X taking on any specific value zero. For example, let's say you have a continuous probability density for men's height. What is the probability that a man will have the exact height of 70 in? It is impossible to find this out because the probability of one man measuring exactly 70 in is very low. It is more probable that he will measure around 70.1 in or maybe 69.97 in. And it doesn't stop there. The fact is that it's impossible to exactly measure any variable that's on a continuous scale. And because of this, it's impossible to figure out the probability of one exact measurement which is occurring in a continuous probability density. Next, we have the probability density function. It's nothing but a function or an expression which is used to define the range of values that a continuous random variable can take. An example of this would be to godge the risk and reward of a stock. A probability density function is a statistical measure which is used to gge the likelihood of a discrete value. A discrete variable can be measured exactly while a continuous variable can have infinite values. However, for both continuous as well as discrete variables, we can define a function which gives us the range of values within which these variables will fall. And that function is known as the probability density function. Now let's take a look at standard deviation. What is standard deviation? Standard deviation is used to measure how the values in your data differ from one another or how spread out your data is. A standard deviation is a statistic that measures the dispersion of a data set relative to its mean. The standard deviation is calculated as the square root of variance by determining each data point's deviation relative to the mean. If the data points are further from the mean, that means that there's a higher deviation within the data set and then the data is set to be more spread out. This leads to a higher standard deviation too. Let's take an example of income in rural and urban areas. In rural areas, let's say such as farming areas, the income doesn't differ that much. more or less everyone earns the same. Because of this, our bell curve has a very low standard deviation and it has a very narrow peak. However, in urban areas, the wealth distribution is very uneven. Some people can have very high incomes and can be earning a lot while other people can have very low incomes. The furthermore the data distribution between these two income points is going to be more spread out because there are lot more people living there who work in various fields and who have various incomes. Because of this, our standard deviation is more spread out and our bell curve will also have a wider peak. Now, how can we find the standard deviation? Standard deviation is obtained by subtracting each data value from the mean and finding the squared average of these values. Let's look at how we can do this with the help of an example. These values correspond to the height of various dogs. We can find the mean by finding the average of all these values which is nothing but adding all the values and dividing it by the total number of values. The mean that we get is 394. This means that the average height of a dog is 394 mm. To find the standard deviation, first we need to subtract the height from the mean. This will tell us how far from the mean our data points actually are. Next, we will square up all of these differences and add them up and again divide it by the total number of values that we have. This is called the variance. The variance that we get in this case is 21704. Finally, when we find the square root of this value, we will get the standard deviation. The standard deviation here is 147. The standard deviation will tell us how our data points differ from the average. And it gives us a basic value suggesting how spread out our data is from the very middle or from the mean. So when we plot these values, this value 147 will mean that a curve will have a width of 147 points around the mean. Now what is the standard normal distribution? The standard normal distribution is a type of normal distribution that has a mean of zero and a standard deviation of one. This means that the normal distribution has its center at zero and it has intervals which increase by one. All normal distributions like the standard normal distribution are unimodel and symmetrically distributed with a bell-shaped curve. However, a normal distribution can take on any value as its mean and standard deviation. In the standard normal distribution, however, the mean and standard deviation are always fixed. When you standardize a normal distribution, the mean becomes zero and the standard deviation becomes one. This allows you to easily calculate the probability of certain values occurring in your distribution or to compare data sets with different mean and standard deviations. The curve shows a standard normal distribution. As you can see again the data is centered at zero. This does not mean that the data necessarily starts at zero. This means that after standardizing this point is where our mean will lie. In a standard normal distribution the standard deviation is one. So all the data points will increase or decrease in steps of one. Let's better understand a standard normal distribution with the help of an example. Again as you can see the data is centered around zero which is nothing but the mean. Let's again consider the weights of students in class 8. The average weight here is around 50 kgs and the data increases and decreases in steps of five. The data over here in this curve is evenly distributed along these steps. This is what a standard normal distribution will look like. We already know that the mean of our data is 50. And because the data is increasing and decreasing in equal steps, we can just standardize it and take it to mean that the data is increasing and decreasing in steps of one. This is what a standard normal distribution looks like. And when you have a data which looks like this, you can always standardize it and convert it into a standard normal distribution. Now standard normal distribution has a couple of properties which makes calculation comparatively easy. The first one is that 68% of the values fall within the first standard deviation. Which means that 68% of all data values on this curve will fall between the range of minus1 to 1 or the first interval ranging from minus1 to 1. The second property is that 95% of the rest of the values are within the second standard deviation or from the second negative point to the second positive point. And finally 99.7% of the values fall within the third standard deviation or from the third negative point to the third positive point. This makes calculations on standard normal distribution fairly easy. You can compare scores on different distributions with different means and standard deviations. You can normalize scores for statistical decision making using standard normal distribution. You can find the probability of observations in a distribution which fall above or below a given value. And finally, you can find the probability that a mean significantly differs from a population mean. Now let's take a look at zcore. So what is a zcore? A zcore is used to tell us how far from the mean a data point actually is. It is calculated using the mean and standard deviation. So it can be said that the zed score is how many standard deviations below the mean our data is. Basically by using the zed score we can get an approximate location of where our data point lies on the graph with regards to the mean. Now the zed score is given by subtracting the data point from the mean and dividing it by standard deviation. This can also be written as X minus mu divided by sigma. Now any normal distribution can be standardized by converting its values into zed scores. The zed score will tell you how many standard deviation from the mean each values lie. While data points are referred to as X in a normal distribution, they are called zed or zed scores in the zed distribution. A zed score is a standard score that will tell you how many standard deviations away from the mean an individual point will lie. A positive zed score will mean that your x value is greater than the mean and a negative zed score will mean that your x value is less than the mean. A zed score of zero will mean that your x value is equal to the mean. And again to standardize a value from a normal distribution all we have to do is convert it to a zed score by subtracting the mean from our individual value and dividing it by the standard deviation. Now let's see how we can find the zed score from data points with the help of a solved example. Let's do a case study. In this case study we'll be taking the summary of daily travel time of a person who's commuting to and from work. All these values are in minutes and using these values we have to calculate the mean, the standard deviation and the zed score. These values are as shown. As we can see there are 13 values in total. Let's start by finding the mean. The mean is the average and it can be gotten by adding all of these values and dividing it by the total number of values. This gives us a value of 38.6. The mean tells us the average of all our data points, which means on an average, he travels for 38.6 minutes to reach work. Next, let's subtract the individual values from our mean and calculate the variance and standard deviation. The values on the left give us the values that we get after subtracting it from the mean. And the variance can be calculated by squaring all of these values, adding up all of the squared values, and dividing it by the total number of values. At the end of the day, we get a variance of 140. To calculate the standard deviation, all we have to do is take a square root of the variance, which gives us a value of 11.8. Now, the mean signifies the average of our values, and we already know this. It gives us the average time which is taken to travel. But the standard deviation will tell us the average value of how much our data points differ from the mean. It tells us the deviation within our own data and it tells us how far away on an average a point is from the mean. Now the value that we get is 11.8 which means that on an average a single data point is around 11.8 data points away from the mean. Now let's calculate the zed score. The zed score is given by subtracting individual data points from the mean and dividing it by the standard deviation. We know that we have a standard deviation of 11.8 and a mean of 38.6. Using these values, we can calculate the zed scores for individual x values. Now we know that a negative zed score means that our x value is lower than our mean. But what does the number 1.06 mean? This means that the zed score for 26 is 1.06 standard deviations away from the mean. The negative symbol here means that our x value is less than the mean. And by how less? 1.06 times the standard deviation. Now we know that the negative value of a zed score means that our x value is less than our mean. But what does the number 1.06 mean? This means that the zed score is 1.06 times the standard deviation less than the mean. The same thing can be said for the zed score of 33. It is 0.47 times the standard deviation less than the mean. The zcore of 65 is 2.23 times the standard deviation more than the mean. That means it has to be added to the mean. The reason that we know it's more than the mean is because this has a positive value. So this means that using zed scores, we can know where our data points fall relative to other points on the graph. The zed score will tell us how far away from the mean a point is in steps of our standard deviation. basics and terminology. The first one is outcome. Whenever we do an experiment like flipping a coin or rolling a dice, we get an outcome. For example, if we flip a coin, we get an outcome of heads or tails. And if you roll a dieice, we get an outcome of 1 2 3 4 5 or six. Random experiment. A random experiment is any well- definfined procedure that produces an observable outcome that could not be perfectly predicted in advance. A random experiment must be well defined to eliminate any vagueness or surprise. It must produce a definite observable outcome so that you know what happened after the random experiment is run. Random events. Consider a simple example. Let us say that we toss a coin up in the air. What can happen when it gets back? It either gives a head or a tail. These two are known as outcome. And the occurrence of an outcome is an event. Thus, the event is the outcome of some phenomenon. The last one is sample space. A sample space is a collection or a set of possible outcomes of a random experiment. The sample space is represented using the symbol S. The subset of all possible outcomes of an experiment is called events. And a sample space may contain a number of outcomes that depends on the experiment. If it contains a finite number of outcomes, then it is known as a discrete or finite sample spaces. Now let's discuss what is random variable. A random variable is a numerical description of the outcome of a statistical experiment. A random variable that may assume only a finite number of values is set to be discrete. One that may assume any value in some interval on the real number line is set to be continuous. Let's see an example. Let X be a random variable defined as a sum of numbers when two dices are rolled. X can assume the values 2 3 4 5 6 7 8 9 10 11 and 12. Notice there's no one here because the sum on the two dice can never be one. Now that we know the basics, let's move on to binomial distribution. The binomial distribution is used when there are exactly two mutually exclusive outcomes of a trial. These outcomes are appropriately labeled success and failure. The binomial distribution is used to obtain the probability of observing X successes in n number of trials with the probability of success on a single trial denoted by P. The binomial distribution assumes that P is fixed for all the trials. Here's a real life example of a binomial distribution. Suppose you purchase a lottery ticket. Then either you are going to win the lottery or not. In other words, the outcome will be either success or failure that can be proved through banimal distribution. There are four important conditions that needs to be fulfilled for an experiment to be a binomial experiment. The first one is there should be a fixed number of end trials carried out. The outcome of a given trial is only two that is either a success or a failure. The probability of success remains constant from trial to trial. It does not changes from one trial to another. And the trials are independent. The outcome of a trial is not affected by the outcome of any other trial. To calculate the binomial coefficient, we use the formula which is ncr into p ^ r into 1 - p to the power n minus r where r is the number of success in n number of trials and p is the probability of success. 1 - p denotes the probability of a failure. Now let's use this formula to solve an example. Suppose a dice is tossed three times. What is the probability of no five turning up 1 five and 3 fives turning up? To calculate the no five turning up here r is equal to 0 and n is equal to 3. Substituting the value in the formula we have 3 0 into 1x 6 ^0 into 5x 6 ^ 3 where 1x 6 is the probability of success and 5x 6 is the probability of failure. Calculating this equation we'll get the value to be 0.5787. In a similar manner to calculate the probability of 1 15 turning up we'll replace r with one and n will be three. So px1 will be equal to 3 c1 into 1x 6 ^ 1 into 5x 6 ^2 which will come out to be 0.347. And for 35 turning up we substitute r equal to 3 and the formula will remain the same and we'll get the value to be 0.0046. Now that we are done with the concepts of binomal probability distribution, here's a problem for you to solve. Post your answers in the comment section and let us know. A poison distribution is a probability distribution used in statistics to show how many times an event is likely to happen over a given period of time. To put it another way, it's a count distribution. Poison distribution are frequently used to comprehend independent event at a constant rate over a given interval of time. The poison distribution was developed by French mathematician Simon Dennis Poison in 1837. A poison distribution is used in cases where the chances of any individual event being a success is very small. The number of defective pencils per box of a 6,000 pencil. the number of plane crashes in India in one year or the number of printing mistakes in each page of a book. All of these example can have use of poison distribution. The poison distribution can be used to calculate how likely it is that something will happen X number of times. A random variable X has a poison distribution with parameter lambda. And the formula for that is e to the power minus lambda into lambda to the power x divided by x factorial where x can be the number of times the event is happening. The value of e is taken as 2.7182. Let's discuss some application of poison distribution. If you want to calculate the number of deaths per day or week due to rare disease in a hospital, you can use a poison distribution. In a similar manner, the count of bacteria perceive >> are an aspiring data scientist who's looking out for online training and certification in data science from the best universities and industry experts then search no more simply learns post-graduate program in data science from Caltech University in collaboration with IBM should be the right choice. For more details on this program, please use the link in the description box below. Hello everyone, welcome to another session by simply learn. Today we are going to discuss the base theorem, an important subtopic that comes under probability theory. We'll start this video by talking about probability and conditional probability. After that we'll move on to the base theorem and understand its formula and a real life example where the base theorem can be used. So let's get started. What is probability? Probability is a branch of mathematics concerning numerical descriptions of how likely an event is to occur or how likely it is that a proposition is true. The probability of an event is a number between 0 and one. Well, roughly speaking, zero indicates the impossibility of the event and one indicates certaintity. The higher the probability of an event, the more likely is that the event will occur. Let's look at an example. A simple example is the tossing of a fair unbiased coin. Since the coin is fair, the outcome that is heads and the tails are both equally probable. The probability of heads equals the probability of the tails. And since no other outcomes are possible, the probability of either heads or tails can be set to be 1 by 2, which is also 50%. The probability of an event can be calculated by number of ways it can happen divided by the total number of outcomes. Now that we know about the probability, let's see if you can answer this question. What is the probability of drawing a jack and a queen consecutively from a deck of 52 cards without replacement? Here are your options. Post your answers in the comment section and let us know. Now let's move on to conditional probability. Let A and B be the two events associated with a random experiment. Then the probability of A's occurrence under the condition that B has already occurred and probability of B is not equal to zero is called the conditional probability. It is denoted by P A/B. Thus we can say that P A/B is equal to P A intersection B divided by P of B where P A/B is the probability of occurrence of A given that B has already occurred and PB is the probability of occurrence of B. To know more about conditional probability, you can check our previous video which is specifically on conditional probability. Now let's move on to base theorem. The base theorem is a mathematical formula for calculating condition probability in probability and statistics. In other words, it is used to figure out how likely an event is associated on its proximity to another. B law or base rule are the other names of this theorem. The formula for the base theorem can be written in a variety of ways. The most common version is P A/ B is equal to P of B / A into P of A divided by P of B. Where P A/B is the conditional probability of event A occurring given that B is true and P A and P of B are the probabilities of A and B occurring independently of one another. Let's solve a problem using the base theorem to understand it better. There is a cricket match tomorrow and in recent years it has rained only 5 days each year. Unfortunately the meteorologist has predicted the rain for tomorrow. Now when it rains the mologist correctly forecast rain 90% of the time and when it doesn't rain he incorrectly forecast rain 10% of the time. Let's calculate what is the probability that it will rain on the match day. So the two sample spaces here are the events that it rains and it does not rain. Additionally, a third event is also there that mologist predicts the rain. So the notation for these events appear below. Event A1 is equal to it rains on the match day. Event A2 that it does not rain on the match day and event B is the meteorologist predicting the rain. Now in terms of probability, we know the following. Probability of A1 is 5 by 365 that it rains 5 days in a year which will come out to be 0.0136. P A2 is 360 by 365 that is no days for 360 days in an year which will come out to be 986. PB/ A1 is.9. This signifies when it rains the meteorologist predicts the rain 90% of the time. In a similar manner, P B by A2 is 0.1 that it does not rain. The meteorologist predicts the rain 10% of the time. Combining all this, we can calculate P A1/B that is the probability it will rain on the given match day given a forecast of rain by mologist. The answer can be determined using the base theorem as shown below. So here's the formula of the base theorem and putting all the values that we have calculated in the previous slide. The probability that it will rain on the match day given a forecast of the rain by meteorologist will come out to be 0.111 which will be equal to 11.11%. So there's an 11% chance that it will rain on the match day given that the meteorologist has predicted the rain. I hope this example is clear to you. >> It's a weekend [music] and John decided to watch the latest movie recommended by Netflix at his friend's place. Before [music] heading out, he asked Siri about the weather and realized it would rain. So, he decided to take his Tesla for the long journey and switched to autopilot on the highway. After coming home from the eventful day, he started wondering how technology [music] has made his life easy. He did some research on the internet and found out that Netflix, Siri, [music] and Tesla are all using AI. So, what is AI? AI or artificial intelligence is nothing but making computers [music] based machines think and act like humans. Artificial intelligence is not a new term. John McCarthy, a computer scientist, coined the [music] term artificial intelligence back in 1956. But it took time to evolve as it demanded [music] heavy computing power. Artificial intelligence is not confined to just movie recommendations and virtual assistants. Broadly classifying, [music] there are three types of AI. Artificial narrow intelligence, also called weak AI, [music] is the stage where machines can perform a specific task. Netflix, Siri, chatbots, [music] spatial recommendation systems are all examples of artificial narrow intelligence. Next up, we have artificial general intelligence, referred to as an intelligent agent's capacity to comprehend or pick up any intellectual skill that a human can't. We are halfway [music] into successfully implementing this space. IBM's Watson supercomputer and GPT3 fall under this category. And lastly, artificial super intelligence. [music] It is the stage where machines surpass human intelligence. You might have [music] seen this in movies and imagined how the world would be if machines occupy it. Fascinated by this, John did more research and [music] found out that machine learning, deep learning, and natural language processing are all connected with artificial intelligence. Machine learning, a subset of AI, is the process of automating and enhancing how [music] computers learn from their experiences without human help. Machine learning can be used in email spam detection, medical diagnosis, [music] etc. Deep learning can be considered a subset of machine learning. It is a field that is based on learning and improving [music] on its own by examining computer algorithms. While machine learning uses simpler concepts, deep learning works with artificial neural networks which [music] are designed to imitate the human brain. This technology can be applied in face recognition, [music] speech recognition, and many more applications. Natural language processing, popularly known as NLP, can be defined [music] as the ability of machines to learn human language and translate it. Chatbots fall under this category. Artificial intelligence is advancing in every [music] crucial field like healthcare, education, robotics, banking, e-commerce, and the [music] list goes on. Like in healthcare, AI is used to identify diseases, helping health care service providers [music] and their patients make better treatment and lifestyle decisions. Coming to the education sector, AI is helping teachers automate grading, organizing, and facilitating parent guardian conversations. [music] In robotics, AI powered robots employ real-time updates to detect obstructions in their path and instantaneously [music] design their routes. Artificial intelligence provides advanced data analytics that is transforming banking by reducing fraud [music] and enhancing compliance. With this growing demand for AI, more and more industries are looking for AI engineers who can help them develop intelligent systems and offer them lucrative salaries [music] going north of $120,000. The future of AI looks promising [music] with the AI market expected to reach $190 billion by 2025. >> We know [music] humans learn from their past experiences and machines follow instructions given by humans. But what if humans can train the machines to learn from their past data and do what humans can do and much faster? Well, [music] that's called machine learning. But it's a lot more than just learning. It's also about understanding and reasoning. So today we will learn about the basics of machine learning. So that's Paul. He loves listening to new songs. He either likes them or dislikes them. Paul decides this on the basis of the song's tempo, [music] genre, intensity, and the gender of voice. For simplicity, let's just use tempo and intensity for now. So here tempo is on the x-axis ranging from relaxed to fast whereas intensity is on the y-axis ranging from light to soaring. We see that Paul likes the song with fast tempo and soaring intensity while he dislikes the song with relaxed tempo and light [music] intensity. So now we know Paul's choices. Let's say Paul listens to a new song. Let's name it as song [music] A. Song A has fast tempo and a soaring intensity. So it lies somewhere here. Looking at the data, can you guess whether Paul will like the song or not? Correct. So Paul likes this song. By looking at Paul's past choices, we were able to classify the unknown song very easily. Right? Let's [music] say now Paul listens to a new song. Let's label it as song B. So song B lies somewhere here with medium tempo and medium intensity. Neither relaxed nor fast, neither light nor soaring. Now, can you guess whether Paul likes it or not? Not able to guess whether Paul will like it or dislike it. Are the choices unclear? Correct. We [music] could easily classify song A. But when the choice became complicated as in the case of song B, yes, [music] and that's where machine learning comes in. Let's see how. In the same example for song B, if we draw a circle around the song B, we see that there are four votes for like whereas [music] one vote for dislike. If we go for the majority votes, we can say that Paul will definitely like the song. That's all. This was a basic machine learning algorithm also. It's [music] called K nearest neighbors. So this is just a small example in one of the many machine learning algorithms. Quite easy, right? Believe me, it is. But what happens when the choices become complicated as in the case of song B? That's when machine learning comes in. [music] It learns the data, builds the prediction model and when the new data point comes in, it can easily predict for it. More the data, better the model, higher will be the accuracy. There are many ways in which the machine learns. It could be either supervised learning, unsupervised learning or reinforcement [music] learning. Let's first quickly understand supervised learning. Suppose your friend gives you 1 million coins of [music] three different currencies. Say 1 rupee, 1 and 1 dirham. Each coin has different weights. For example, a coin of 1 rupee weighs 3 g. 1 euro weighs 7 [music] g and 1 dirham weighs 4 g. Your model will predict the currency of the coin. Here your weight becomes [music] the feature of coins while currency becomes the label. When you feed this data to the machine learning model, it learns which feature is associated with which label. For example, [music] it will learn that if a coin is of 3 g, it will be a 1 rupee coin. Let's give a new coin to the machine. On the basis of the weight of the new coin, your model will predict the currency. Hence, supervised learning uses labeled data to train [music] the model. Here, the machine knew the features of the object and also the labels associated with those features. [music] On this note, let's move to unsupervised learning and see the difference. Suppose you have cricket data set of various players [music] with their respective scores and wickets taken. When we feed this data set to the machine, the machine identifies the pattern of player performance. So, it plots this data with the respective wickets [music] on the x-axis while runs on the y-axis. While looking at the data, you'll clearly see that there are [music] two clusters. The one cluster are the players who scored high runs and took less wickets [music] while the other cluster is of the players who scored less runs but took many wickets. So here we interpret these two clusters as batsmen and bowlers. The important point to note here is that there were no labels of batsmen and bowlers. [music] Hence the learning with unlabeled data is unsupervised learning. So we saw supervised learning where the data was labeled and the unsupervised learning where the data was unlabeled. And then there is reinforcement learning which is a reward-based learning or we can say that it works on the principle of feedback. Here let's say you provide the system with an image of a dog and ask it to identify it. The system identifies it as a cat. So you give a negative feedback to the machine saying that it's a dog's image. The machine will learn from the feedback and finally if it comes across any other image of a dog, it'll be able [music] to classify it correctly. That is reinforcement learning. To generalize machine learning model, let's see a flowchart. >> [music] >> Input is given to a machine learning model which then gives the output according to the algorithm applied. If it's right, we take the output as our final result. Else we provide feedback to the training model and ask it [music] to predict until it learns. I hope you've understood supervised and unsupervised learning. So let's have a quick quiz. You have to determine whether the given scenarios uses supervised or unsupervised learning. Simple, [music] right? Scenario one. Facebook recognizes your friend in a picture from an album [music] of tagged photographs. Scenario two, Netflix recommends new movies based on someone's past movie choices. [music] Scenario three, analyzing bank data for suspicious transactions and flagging the fraud transactions. [music] Think wisely and comment below your answers. Moving on, don't you sometimes wonder how is machine learning possible in [music] today's era? Well, that's because today we have humongous data available. Everybody's online either making a transaction or just surfing the internet and [music] that's generating a huge amount of data every minute and that data my friend is the key to analysis. Also, the memory handling capabilities of computers have largely increased which helps them to process such huge amount of data at hand [music] without any delay. And yes, computers now have great computational powers. So there are a lot of applications of machine learning out there. To name a few, machine learning is used in healthcare where diagnostics are predicted for doctor's review. The sentiment analysis that the tech giants are doing on social media is another interesting application of machine learning. Fraud detection in the finance sector and also to predict customer churn in the e-commerce [music] sector. While booking a cab, you must have encountered search pricing often where it says the fair of your trip has been updated. [music] Continue booking. Yes, please. I'm getting late for office. Well, that's an interesting machine learning model which is used by global taxi giant Uber and [music] others where they have differential pricing in real time based on demand, the number of cars available, bad weather, rush hour, etc. So they use the search pricing model to ensure that those who need a cab can get one. Also, it uses predictive modeling to predict where the demand will be high with a [music] goal that drivers can take care of the demand and search pricing can be minimized. Great. Hey Siri, can you remind me to book a cab at 6 p.m. today? >> Okay, I'll remind you. >> Thanks. >> No problem. >> If you are an aspiring data scientist who's looking out for online training and certification in data science from the best universities and industry experts, then search no more. Simply learns post-graduate program in data science from Caltech University in collaboration with IBM should be the right choice. For more details on this program, please use the link in the description box below. Let's dive in a little deeper and see how machine learning works. Let's say you provide a system with the input data that carries the photos of various kinds of fruits. Now you want the system to figure out what are the different fruits and group them accordingly. So what the system does? It analyzes the input data. Then it tries to find patterns. Patterns like shapes, size and color. Based on these patterns, the system will try to predict the different types of fruit and segregate them. Finally, it keeps track of all such decisions it took in the process to make sure it's learning. the next time you ask the same system to predict and segregate the different types of fruits, it won't have to go through the entire process again. That's how machine learning works. Now let's look into the types of machine learning. Machine learning is primarily of three types. First one is supervised machine learning. As the name suggests, you have to supervise your machine learning while you train it to work on its own. It requires labeled training data. Next up is unsupervised learning wherein there will be training data but it won't be labeled. Finally, there's reinforcement learning wherein the system learns on its own. Let's talk about all these types in detail. Let's try to understand how supervised learning works. Look at the pictures very very carefully. The monitor depicts the model or the system that we are going to train. This is how the training is done. We provide a data set that contains pictures of a kind of a fruit. say an apple. Then we provide another data set which lets the model know that these pictures were that of a fruit called apple. This ends the training phase. Now what we will do is we provide a new set of data which only contains pictures of apple. Now here comes the fun part. The system can actually tell you what fruit it is and it will remember this and apply this knowledge in future as well. That's how supervised learning works. You are training the model to do a certain kind of an operation on its own. This kind of a model is generally used into filtering spam mails from your email accounts as well. Yes, surprise, aren't you? So, let's move on to unsupervised learning. Now, let's say we have a data set which is cluttered. In this case, we have a collection of pictures of different fruits. We feed this data to the model and the model analyzes the data to figure out patterns in it. In the end, it categorizes the photos into three types. as you can see in the image based on their similarities. So you provide the data to the system and let the system do the rest of the work. Simple, isn't it? This kind of a model is used by Flipkart to figure out the products that are well suited for you. Honestly speaking, this is my favorite type of machine learning out of all the three. And this type has been widely shown in most of the sci-fi movies lately. Let's find out how it works. Imagine a newborn baby. You put a burning candle in front of the baby, the baby does not know that if it touches the flame, its fingers might get burned. So, it does that anyway and gets hurt. The next time you put that candle in front of the baby, it will remember what happened the last time and would not repeat what it did. That's exactly how reinforcement learning works. We provide the machine with a data set wherein we ask it to identify a particular kind of a fruit, [clears throat] in this case an apple. So what it does as a response, it tells us that it's a mango. But as we all know, it's a completely wrong answer. So as a feedback, we tell the system that it's wrong. It's not a mango, it's an apple. What it does, it learns from the feedback and keeps that in mind. When the next time when we ask a same question, it gives us the right answer. It is able to tell us that it's actually an apple. That is a reinforced response. So that's how reinforcement learning works. It learns from his mistakes and experiences. This model is used in games like Prince of Persia or Assassin's Creed or FIFA where in the level of difficulty increases as you get better with the games. Just to make it more clear for you, let's look at a comparison between supervised and unsupervised learning. Firstly, the data involved in case of supervised learning is labeled. As we mentioned in the examples previously, we provide the system with a photo of an apple and let the system know that this is actually an apple. That is called label data. So the system learns from the label data and makes future predictions. Now unsupervised learning does not require any kind of label data because its work is to look for patterns in the input data and organize it. The next point is that you get a feedback in case of supervised learning. That is once you get the output the system tends to remember that and uses it for the next operation. That does not happen for unsupervised learning. And the last point is that supervised learning is mostly used to predict data whereas unsupervised learning is used to find out hidden patterns or structures in data. I think this would have made a lot of things clear for you regarding supervised and unsupervised learning. Now let's talk about a question that everyone needs to answer before building a machine learning model. What kind of a machine learning solution should we use? Yes, you should be very careful with selecting the right kind of solution for your model because if you don't, you might end up losing a lot of time, energy, and processing cost. I won't be naming the actual solutions because you guys aren't familiar with them yet. So we will be looking at it based on supervised, unsupervised, and reinforcement learning. So let's look into the factors that might help us select the right kind of machine learning solution. First factor is the problem statement describes the kind of model you will be building or as the name suggests it tells you what the problem is. For example, let's say the problem is to predict the future stock market prices. So for anyone who is new to machine learning would have trouble figuring out the right solution. But with time and practice you will understand that for a problem statement like this solution based on supervised learning would work the best for obvious reasons. Then comes the size, quality and nature of the data. If the data is cluttered, you go for unsupervised. If the data is very large and categorical, we normally go for supervised learning solutions. Finally, we choose the solution based on their complexity. As for the problem statement wherein we predict the stock market prices, it can also be solved by using reinforcement learning. But that would be very very difficult and timeconuming unlike supervised learning. Algorithms are not types of machine learning. In the most simplest language, they are methods of solving a particular problem. So the first kind of method is classification which falls under supervised learning. Classification is used when the output you are looking for is a yes or a no or in the form A or B or true or false. Like if a shopkeeper wants to predict if a particular customer will come back to his shop or not, he will use a classification algorithm. The algorithms that fall under classification are decision tree, knife base, random forest, logistic regression and KNN. The next kind is regression. This kind of a method is used when the predicted data is numerical in nature. Like if the shopkeeper wants to predict the price of a product based on its demand, it would go for regression. The last method is clustering. Clustering is a kind of unsupervised learning. Again, it is used when the data needs to be organized. Most of the recommendation system used by Flipkart, Amazon, etc. make use of clustering. Another major application of it is in search engines. The search engines study your old search history to figure out your preferences and provide you the best search results. One of the algorithms that fall under clustering is K means. Now that we know the various algorithms, let's look into four key algorithms that are used widely. We will understand them with very simple examples. The four algorithms that we will try to understand are K nearest neighbor, linear regression, decision tree and KN base. Let's start with our first machine learning solution. K nearest neighbor. K nearest neighbor is again a kind of a classification algorithm. As you can see on the screen, the similar data points form clusters. the blue one, the red one and the green one. There are three different clusters. Now, if we get a new and unknown data point, it disclassified based on the cluster closest to it or the most similar to it. K in KN&N is the number of nearest neighboring data points we wish to compare the unknown data with. Let's make it clear with an example. Let's say we have three clusters in a cost to durability graph. First [snorts] cluster is of footballs. The second one is of tennis balls and the third one is of basketballs. From the graph we can say that the cost of footballs is high and the durability is less. The cost of tennis balls is very less but the durability is high and the cost of basketballs is as high as the durability. Now let's say we have an unknown data point. We have a black spot which can be one kind of the balls but we don't know what kind it is. So what we'll do we'll try to classify this using KNN. So if we take K is equal to 5, we draw a circle keeping the unknown data point in the center and we make sure that we have five balls inside that circle. In this case we have a football, a basketball and three tennis balls. Now since we have the highest number of tennis balls inside the circle, the classified ball would be a tennis ball. So that's how K nearest neighbor classification is done. Linear regression is again a type of supervised learning algorithm. This algorithm is used to establish linear relationship between variables, one of which would be dependent and the other one would be independent. Like if we want to predict the weight of a person based on his height, weight would be the dependent variable and height would be independent. Let's have a look at it through an example. Let's say we have a graph here showing a relationship between height and weight of a person. Let's put the y-axis as age and the x-axis as weight. So the green dots are the various data points. These green dots are the data points and D is the mean squared error. That is the perpendicular distances from the line to the data points are the error values. This error tells us how much the predicted values vary from the original value. Let's ignore this blue line for a while. So let's say if this is our regression line, you can see the distance from all the data points from this line is very high. So if we take this line as a regression line, the error in the prediction will be too high. So in this case the model will not be able to give us a good prediction. Let's say we draw another regression line here like this. Even in this case you can see that the perpendicular distance of the data points from the line is very high. So the error value will still come as high as the last one. So this model will also not be able to give us a good prediction. So what to do? So finally we draw a line which is this blue line. So here we can see that the distance of the data points from the line is very less relative to the other two lines we drew. So the value of D for this line will be very less. So in this case if we take any value on the x-axis the corresponding value on the y-axis will be our prediction. And given the fact that the d is very low our prediction should be good also. This is how regression works. We draw a line a regression line that is in such a way that the value of D is the least eventually giving us good predictions. This algorithm that is decision tree is a kind of an algorithm you can very strongly relate to. It uses a kind of a branching method to realize the problem and make decisions based on the conditions. Let's take this graph as an example. Imagine yourself sitting at home getting bored. You feel like going for a swim. What you do is you check if it's sunny outside. So that's your first condition. If the answer to that condition is yes, you go for a swim. If it's not sunny, then the next question you would ask yourself is if it's raining outside. So that's condition number two. If it's actually raining, you cancel the plan and stay indoors. If it's not raining, then you would probably go outside and have a walk. So that's the final node. That's how decision tree algorithm works. You probably use this every day. It realizes a problem and then takes the decisions based on the answers to every conditions. Nightb algorithm is mostly used in cases where a prediction needs to be done on a very large data set. It makes use of conditional probability. Conditional probability is the probability of an event say A happening given that another event B has already happened. This algorithm is most commonly used in filtering spam mails in your email account. Let's say you receive a mail. The model goes through your old spam mail records. Then it uses space theorem to predict if the present male is a spam mail or not. So P C of A is the probability of event C occurring when A has already occurred. P A of C is the probability of event A occurring when C has already occurred. and P C is the probability of event C occurring and PA is the probability of event A occurring. Let's try to understand night base with a better example. Night base can be used to determine on which days to play cricket based on the probabilities of a day being rainy, windy or sunny. The model tells us if a match is possible. If we consider all the weather conditions to be event A for us and the probability of a match being possible event C. So the model applies the probabilities of event A and C into the base theorem and predicts if a game of cricket is possible on a particular day or not. In this case, if the probability of C of A is more than 0.5, we can be able to play a game of cricket. If it's less than 0.5, we won't be able to do that. That's how NY's algorithm works. >> We're going to cover reinforcement learning today and what's in it for you. We'll start with why reinforcement learning. We'll look at what is reinforcement learning. We'll see what the different kinds of learning strategies are that are being used today in computer models under supervised versus unsupervised versus reinforcement. We'll cover important terms specific to reinforcement learning. We'll talk about Marov's decision process and we'll take a look at a reinforcement learning example. Well, we'll teach a tic-tac-toe how to play. Why reinforcement learning? Training a machine learning model requires a lot of data which might not always be available to us. Further, the data provided might not be reliable. Learning from a small subset of actions will not help expand the vast realm of solutions that may work for a particular problem. And you can see here we have the robot learning to walk. Um, very complicated setup when you're learning how to walk. And you'll start asking questions like, if I'm taking one step forward and left, what happens if I pick up a 50 lb object? How does that change how a robot would walk? These things are very difficult to program because there's no actual information on it until the it's actually tried out. Learning from a small subset of actions will not help expand the vast realm of solutions that may work for a particular problem. And we'll see here it learned how to walk. This is going to slow the growth that technology is capable of. Machines need to learn to perform actions by themselves and not just learn off humans. And you see the objective climb a mountain. Real interesting point here is that as human beings, we can go into a very unknown environment and we can adjust for it and kind of explore and play with it. Most of the models, the non-reinforcement models in computer u machine learning aren't able to do that very well. Uh there's a couple of them that can be used or integrated to see how it goes is what we're talking about with reinforcement learning. So what is reinforcement learning? Reinforcement learning is a subbranch of machine learning that trains a model to return an optimum solution for a problem by taking a sequence of decisions by itself. Consider a robot learning to go from one place to another. The robot is given a scenario must arrive at a solution by itself. The robot can take different paths to reach the destination. It will know the best path by the time taken on each path. It might even come up with a unique solution all by itself. And that's really important is we're looking for unique solutions. Uh we want the best solution, but you can't find it unless you try it. So we're looking at uh our different systems or different model. We have supervised versus unsupervised versus reinforcement learning. And with the supervised learning, that is probably the most controlled environment. uh we have a lot of different supervised learning models whether it's linear regression neural networks um there's all kinds of things in between decision trees the data provided is labeled data with output values specified and this is important because we talk about supervised learning you already know the answer for all this information you already know the picture has a motorcycle in it so you're supervised learning you already know that um the outcome for tomorrow for you know going back a week you're looking at stock you can already have like the graph of what the next day looks like. So you have an answer for it and you have labeled data which is used. You have an external supervision and solves problems by mapping labeled input to known output. So very controlled unsupervised learning and unsupervised learning is really interesting because it's now taking part in many other models. They start with an you can actually insert an unsupervised learning model um in almost either supervised or reinforcement learning as part of the system which is really cool. Uh data provided is unlabelled data. The outputs are not specified. Machine makes its own predictions used to solve association with clustering problems. Unlabeled data is used. No supervision. Solves problems by understanding patterns and discovering output. Uh so you can look at this and you can think um some of these things go with each other. They belong together. So it's looking for what connects in different ways. And there's a lot of different algorithms that look at this. Um when you start getting into those are some really cool images that come up of what unsupervised learning is. How it can pick out say uh the area of a donut. one model will see the area of the donut and the other one will divide it into three sections based on its location versus what's next to it. So there's a lot of stuff that goes in with unsupervised learning and then we're looking at reinforcement learning. Probably the biggest industry in today's market uh in machine learning or growing market. It's very it's very infant stage uh as far as how it works and what it's going to be capable of. The machine learns from its environment using rewards and errors used to solve rewardbased problems. No predefined data is used. No supervision. Follows trail and error problem solving approach. Uh so again we have a random at first you start with a random. I try this it works and this is my reward. Doesn't work very well maybe or maybe doesn't even get you where you're trying to get it to do. and you get your reward back and then it looks at that and says well let's try something else and it starts to play with these different things finding the best route. So let's take a look at important terms in today's reinforcement model and this has become pretty standardized over the last uh few years. So these are really good to know. We have the agent uh agent is the model that is being trained via reinforcement learning. So this is your actual u entity that has however you're doing it whether you're using a neural network or Q table or whatever combination thereof. This is the actual agent that you're using. This is the model and you have your environment. Uh the training situation that the model must optimize to is called its environment. Uh and you can see here I guess we have a robot who's trying to get chest full of gems or whatever. And that's the output. And then you have your action. This is all possible steps that can be taken by the model and it picks one action and you can see here it's picked three different uh routes to get to the chest of diamonds and gems. We have a state the current position condition returned by the model. And you could look at this uh if you're playing like a video game. This is the screen you're looking at. Uh so when you go back here uh the environment is the whole game board. So, if you're playing one of those Mobius games, you might have the whole game board going on. Uh, but then you have your current position. Where are you on that game board? What's around that? What's around you? Um, if you were talking about a robot, the environment might be moving around the yard, where it is in the yard, and what it can see, what input it has in that location. That would be the current position condition returned by the model. And then the reward uh to help the model move in the right direction. It is rewarded. Points are given to it to appraise some kind of action. So yeah, you did good or if uh didn't do as good trying to maximize the reward and have the best reward possible. And then policy. Policy determines how an agent will behave at any time. It acts as a mapping between action and present state. This is part of the model. What what what is your action that you're you're going to take? what's the policy you're using to have an output from your agent. One of the reasons they separate a policy as its own entity is that you usually have a prediction um of a different options and then the policy well how am I going to pick the best based on those predictions I'm going to guess at different options and we'll actually weigh those options in and find the best option we think will work. Uh, so it's a little tricky, but the policy thing is actually pretty cool how it works. Let's go ahead and take out look at a reinforcement learning example. And just in looking at this, we're going to take a look uh consider what a dog um that we want to train. Uh so the dog would be like the agent. So you have your your puppy or whatever. Uh and then your environment is going to be the whole house or whatever it is where you're training them. And then you have an action. We want to teach the dog to fetch. So action equals fetching. Uh and then we have a little biscuit. So we can get the dog to perform various actions by offering incentives such as a dog biscuit as a reward. The dog will follow a policy to maximize this reward and hence will follow every command and might even learn new actions like begging by itself. Uh so you have b you know so we start off with fetching. It goes oh I get a biscuit for that. It tries something else. and you get a handshake or begging or something like that and it goes, "Oh, this is also reward-based." And so it kind of explores things to find out what will bring it as biscuit. And that's very much like how reinforced model goes is it uh looks for different rewards. How do I find can I try different things and find a reward that works? The dog also will want to run around and play and explore it environment. Uh this quality of model is called exploration. So there's a little randomness going on in exploration and explores new parts of the house. Climbing on the sofa doesn't get a reward. In fact, it usually gets kicked off the sofa. So let's talk a little bit about Marov's decision process. Uh, Marov's decision process is a reinforcement learning policy used to map a current state to an action where the agent continuously interacts with the environment to produce new solutions and receive rewards. And you'll see here's all of our different uh uh vocabulary we just went over. We have a reward or state or agent or environment and our action. And so even though the environment kind of contains everything um that you you really when you're actually writing the program, your environment is going to put out a reward and state that goes into the agent. Uh the agent then looks at this uh state or it looks at the reward usually um first and it says okay I got rewarded for whatever I just did or I didn't get rewarded and then it looks at the state and then it comes [clears throat] back and if you remember from policy the policy comes in um and then we have a reward. The policy is that part that's connected at the bottom. And so it looks at that policy and it says, "Hey, what's a good action that will probably be similar to what I did?" Or um uh sometimes they're completely random, but what's a good action that's going to bring me a different reward? So, taking the time to just understand these different pieces as they go is pretty important in most of the models today. Um, and so a lot of them actually have templates based on this that you can pull in and start using. Um, pretty straightforward as far as once you start seeing how it works. Uh, you can see your environment send it says, "Hey, this is the agent did this. If you're a character in a game, this happened and it shoots out a reward in a state." The agent looks at the reward, looks at the new state, and then takes a little guess and says, "I'm going to try this action." And then that action goes back into the environment. it affects the environment. The environment then changes depending on what the action was and then it has a new state and a new reward that goes back to the agent. So in the diagram shown, we need to find the shortest path between node A and D. Each path has a reward associated with it and the path with a maximum reward is what we want to choose. The nodes A, B, C, Denote the nodes to travel from node uh A to B is an action. Reward is the cost of each path and policy is each path taken. And you can see here A can go uh to B or A can go to C right off the bat or it can go right to D. And if you explored all three of these uh you would find that A going to D was a zero reward. Um A going to C and D would generate a different reward. Or you could go AC B D. There's a lot of options here. Um and so when we start looking at this diagram, you start to realize that even though uh today's reinforced learning models do really good at um finding an answer, they end up trying almost all the different directions you see. And so they take up a lot of work uh or a lot of processing time for reinforcement learning. They're right now in their infant stage and they're really good at solving simple problems. And we'll take a look at one of those in just a minute in a tic-tac-toe game. Uh but you can see here uh once it's gone through these and it's explored, it's going to find the AC D is the best reward. It gets a full 30 points for it. So let's go ahead and take a look at a reinforcement learning demo. Uh in this demo, we're going to use reinforcement learning to make a tic-tac-toe game. You'll be playing this game against the machine learning model. And we'll go ahead and we're doing it in Python. So, let's go ahead and go through um I always uh not always actually have a lot of Python tools. Let's go through um Anaconda, which will open up a Jupyter notebook. Seems like a lot of steps, but it's worth it to keep all my stuff separate, and it's also has a nice display when you're in the Jupyter notebook for doing Python. So, here's our Anaconda Navigator. I open up the notebook, which is going to take me to a web page. And I've gone in here and created a new uh Python folder. In this case, I've already done it and enabled it. Change the name to tic-tac-toe. Uh, and then for this example, uh, we're going to go ahead and import a couple things. We're going to, um, import numpy as np. We'll go ahead and import pickle. Numpy, of course, is our number array. And then, uh, pickle is just a nice way sometimes for storing, uh, different information, uh, different states that we're going to go through on here. Uh, and so we're going to create a class called state. We're going to start with that. And there's a lot of uh lines of code to this uh class that we're going to put in here. Don't let that scare you too much. There's not as much here. Um it looks like there's going to be a lot here, but there really is just a lot of setup going on in the in our class state. And so we have up here, we're going to initialize it. Um we have our board. Um, it's a tic-tac-toe board, so we're only dealing with nine spots on the board. Uh, we have player one, player two, uh, is end. We're going to create a board hash. Uh, we'll look at that in just a minute. We're just going to store some information in there. Symbol of player equals one. Um, so there's a few things going on as far as the initialization. Uh, then something simple. We're just going to get the hash um of the board. We're going to get the information from the board on there, which is uh columns and rows. We want to know when a winner occurs. Uh so if you get three in a row, that's what this whole section here is for. Uh let me go ahead and scroll up a little bit. And you can get a copy of this code if you send a note over to SimplyLearn. We'll send you over um this particular file and you can play with it yourself and see how it's put together. I don't want to spend a huge amount of time on this uh because this is just some real general Python coding. Uh but you can see here we're just going through all the rows and you add them together and if it equals three, three in a row. Same thing with columns. Uh diagonal. So you got to check the diagonal. That's what all this stuff does here is it just goes through the different areas. Actually, let me go ahead and put There we go. Um, and then it comes down here and we do our sum and it says true uh minus three. Just says did somebody win or is it a tie? So, you got to add up all the numbers on there anyway just in case they're all filled up. And next, we also need to know available positions. Um, these are ones that don't no one's ever used before. This way, when you try something or the computer tries something, uh, it's not going to give it an illegal move. That's what the available positions is doing. Uh then we want to update our state. And so you have your position going in. We're just sending in the position that you just chose. And you'll see there's a little user interface we put in there. You p pick the row and column in there. And again, I mean, this is a lot of code. Uh so really it's kind of a thing you'd want to go through and play with a little bit and just read through it, get a copy of it. Uh great way to understand how this works. And here is a given reward. Um, so we're going to give a reward. Result equals self-winner. This is one of the hearts of what's going on here. Uh, is we have a result self.winner. So if there's a winner, then we have a result. If the result equals one, here's our feedback. Uh, if it doesn't equal one, then it gets a zero. So it only gets a reward in this particular case if it wins. And that's important to know because different uh systems of reinforced learning do rewarding a lot differently depending on what you're trying to do. This is a very simple example with a 3x3 board. Imagine if you're playing a video game. Uh certainly you only have so many actions, but your environment is huge. You have a lot going on in the environment and suddenly a reward system like this is going to be just um is going to have to change a little bit. it's going to have to have different rewards and different setup. And there's all kinds of advanced ways to do that as far as weighing you add weights to it. And so they can add the weights up depending on where the reward comes in. So it might be that you actually get a reward. In this case, you get the reward at the end of the game. And I'm spending just a little bit of time on this because this is an important thing to note. But there's different ways to add up those rewards. it might have like if you take a certain path um the first reward is going to be weighed a little bit less than the last reward because the last reward is actually winning the game or scoring or whatever it is. So this reward system gets really complicated on some of the more advanced uh setups. Um, in this case though, you can see right here that they give a a 0.1 and a 0.5 reward um just for getting a picking the right value and something that's actually valid instead of picking an invalid value. So, rewards again, that's like key. It's huge. How do you feed the rewards back in? Um, then we have a board reset. That's pretty straightforward. It just goes back and resets the board to the beginning because it's going to try out all these different things while it's learning. It's going to do it by trial and error. So, you have to keep resetting it. And then, of course, there's the play. We want to go ahead and play uh rounds equals 100. Depends on what you want to do on here. Um you can set this different. You obviously set that to higher level, but this is just going to go through and you'll see in here uh that we have player one and player two. This is this is the computer playing itself. Uh, one of the more powerful ways to learn to play a game or even learn something that isn't a game is to have two of these models that are basically trying to beat each other. And so they always they keep finding explore new things. This one works for this one. So this one tries new things, it beats this. We've seen this in um chess, I think, was a big one where they had the two players in chess with reinforcement learning. uh is one of the ways they train one of the top um computer chess playing algorithms. Uh so this is just what this is. It's going to choose an action. It's going to try something and the more it tries stuff um the more we're going to record the hash. We actually have a board hash where they self get the hash set up on here where it stores all the information. And then once you get to a win, one of them wins, it gets the reward. Uh then we go back and reset and try again. And then kind of the fun part we actually get down here is uh we're going to play with a human. So we'll get a chance to come in here and see what that looks like when you put your own information in. And then it just comes in here and does the same thing it did above. It gives it a reward for its things um or sees if it wins or ties. Um looks at available positions, all that kind of fun stuff. And then finally, we want to show the board. Uh so it's going to print the board out each time. Really um as an integration is not that exciting. What's exciting uh in here is one looking at this reward system. Whoops. Play one more up. The reward system is really the heart of this. How do you reward the different uh setup and the other one is when it's playing it's got to take an action. And so what it chooses for an action is also the heart of reinforcement learning. How do we choose that action? And those are really key to right now where reinforcement learning is uh in today's uh technology is uh figuring this out. How do we reward it and how do we guess the next best action? So we have our uh environment and you can see the environment is we're going to be or the state uh which is kind of like what's going on. We're going to return the state depending on what happens. And we want to go ahead and create our agent. Uh in this case, our player. So each one is let me go and grab that. And so we look at a class player. Um this is where a lot of the magic is really going on is what how is this player figuring out how to maneuver around the board? And then the board of course returns a state uh that it can look at and a reward. Uh so we want to take a look at this. We have a name uh self state. This is class player. And when you say class player, we're not talking about a human player. We're talking about u just a uh the computer players. And this is kind of interesting. So remember I told you depending on what you're doing, there's going to be a decay gamma. Um explore rate. Uh these are what I'm talking about is how do we train it? Um as you try different moves, it gets to the end. The first move is important, but it's not as as important as the last one. And so you could say that the last one has the heaviest weight. And then as you as you get there, the first one, let's see, the first move gives you a five reward, the second gives you a two reward, and the third one gives you a 10 reward because that's the final ending. You got it. The 10's going to count more than the first step. Uh, and here's our uh, we're going to, you know, get the board information coming in and then choose an action. This was the second part that I was talking about that was so important. Uh so once you have your training going on, we have to do a little randomness. And you can see right here is our NP random uh uniform. So it's picking out a random number. Take a random action. This is going to just pick which row and which column it is. Um and so choosing the action. This one you can see we're just doing random states. uh choice, length of positions, action position, and then it skips in there and takes a look at the board uh for P and positions. You it's actually storing the different boards each time you go through so it has a record of what it did so it can properly weigh the values. And this simply just appends a hash state. What's the last state? Pinned it to the uh u to our states on here. Here's our feedback reward. the reward comes in and it's going to take a look at this and say is it none uh what is the reward and here is that formula remember I was telling you about up here um that was important because it has decay gamma times the reward this is where as it goes through each step and this is really important this is this is kind of the heart of this of what I was talking about earlier uh you have step one and And this might have a a reward of two. You have step two. I probably should have done ABC. This has a step three. Uh step four. So on till you get to step in. And this might have a reward of 10. Uh so reward of 10. We're going to add that. But we're not adding uh let's say this one right here. Uh let's say this reward here right before 10 was um let's say it's also 10. That just makes the the uh math easy. So we had 10 and 10. Uh we had 10. This is 10 and 10 in whatever it is. But it's time it's 0.9. Uh so instead of putting a full 10 here, we only do nine. That's uh 0.9 time 10. And so this formula um as far as the decay times the reward minus the cell state value uh it basically adds in it says here's one or here's two. I'm sorry I should have done this ABC would have been easier. Uh so the first move goes in here and it puts two in here. Uh then we have our self uh setup on here. You can see how this gets pretty complicated in the math, but this is really the key is how do we train our states and we want the the final state, the win to get the most points. If you win, you get most points. U and the first step gets the least amount of points. So, you're really training this almost in reverse. You're training, you're training it from the last place where you have like it says, "Okay, this is now where I where need to sum up my rewards and I want to sum them up going in reverse and I want to find the answer in reverse." Kind of an interesting uh uh play on the mind when you're trying to figure this stuff out. And of course, we want to go ahead and reset the board down here uh and save the policy load policy. These are the different things that are going in between the agent and the state to figure out what's going on. Let's go ahead and load that up. And then finally, we want to go ahead and create a human player. And the human player is going to be a little different uh in that uh you choose an action row and column. Here's your action. Uh if action is if action in positions, meaning positions that are available, uh you return the action. If not, it just keeps asking you until you get an action that actually works. And then we're going to go ahead and append to the hash state, which uh we don't need to worry about because it returns the action up here. And feed forward. Uh again, this is because it's a human. Um at the end of the game, bat, propagate, and update state values. This part isn't being done because it's not programming uh the model. Uh the model is getting its own rewards. So, we've gone ahead and loaded this in here. Uh, so here's all our pieces. And the first thing we want to do is set up uh P1, player one, uh, P2, player two, and then we're going to send our players to our state. So, now it has P1, P2, and it's going to play, and it's going to play 50,000 rounds. Now, we can probably do a lot less than this, and it's not going to get the full results. In fact, you know what? Uh, let's go ahead and just do five. Uh, just to play with it because I want to show you something here. Oops. Somewhere in there I forgot to load something. There we go. I must have forgot to run this run. Oops, forgot a reference there for the board rows and columns 3x3. There is actually in the state it references that. We just tack it on on the end. It was supposed to be at the beginning. Uh, so now I've only set this up with um, see where are we going here? I've only set this up to train five times. And the reason I did that is we're going to uh, come in and actually play it. And then I'm going to change that and we can see how it differs on there. There we go. And it didn't make it through a run. And we're going to go ahead and save the policy. Uh, so now we have our player one and our player two policy. Uh, the way we set it up, it has two separate policies loaded up in there. And then we're going to come in here and we're going to do uh player one is going to be the computer experience rate zero. Load policy one. Human player human. And we're going to go ahead and play this. Now remember, I only went through it um uh just one round of training. In fact, minimal training. And so it puts an X there. And I'm going to go ahead and do row zero, column one. You can see this is very uh basic on here. And so I put in my zero. And then I'm going to go zero, block it, zero, zero. And you can see right here, it let me win. Uh just like that, I was able to win. Zero two. And woo, human wins. So I only trained it five times. We're going to run this again. And this time, uh, instead of five, let's do 5,000 or 50,000. I think that's what the guys in the back had. And this takes a while to train it. This is where reinforcement learning really falls apart. Look how simple this game is. We're talking about uh a 3x3 set of columns. And so for me to train it on this um I could do a Q table which would take which would go much quicker. Um you could build a quick Q table with almost all the different options on there and uh you would probably get a the same result much quicker. We're just using this as an example. So when we look at reinforcement learning, you need to be very careful what you apply it to. It sounds like a good deal until you do like a large neural network where you're doing um you set the neural network to a learning increment of one. So every time it goes through it learns and then you do your action. So you pick from the learning uh setup and you actually try actions on the learning setup until you get the what you think is going to be the best action. So you actually feed what you think is right back through the neural network. There's a whole layer there which is really fun to play with. and then it has an output. Well, think of all those processes. I mean, that is just a huge amount of work it's going to do. Uh, let's go ahead and skip ahead here. Give it a moment. It's going to take a a minute or two to go ahead and run. [sighs and gasps] Now, to train it, uh, we went ahead and let it run. And it took a while. This this took, um, I got a pretty powerful processor, and it took about five minutes plus to run it. and we'll go ahead and uh run our player setup on here. Oops, it brought in the last Whoops, it brought in the last round. So, give me just a moment to reddo the policy save. There we go. I forgot to save the policy back in there and then go ahead and run our player again. So, we we've saved the policy and then we want to go ahead and load the policy for P1 as a computer. And we can see the computer's gone in the bottom right corner. I'm going to go ahead and go uh one one which is the center and it's gone right up the top. And if you have ever played tic-tac-toe, you know the computer has me. Uh but we'll go ahead and play it out. Row zero, column two. There it is. And then it's gone here. And so I'm going to go ahead and go row 0 one two. No, zero one. There we go. And column zero. That's where I want it. Oh, and it says I Okay, you your action. There we go. Boom. Uh, so you can see here we've got a didn't catch the win on this. It said tie. Um, kind of funny that it didn't catch the win on there. But if we play this a bunch of times, you'll find that it's going to win more and more. The more we train it, the more the reinforcement happens. This lengthy training process uh is really the stopper on reinforcement learning. As this changes, reinforcement learning will be one of the more powerful uh packages evolving over the next decade or two. In fact, I would even go as far as to say it is the most important uh machine learning tool and artificial intelligence tool out there as it learns not only a simple tic-tac-toe board, but we start learning environments. And the environment would be like in language. If you're translating a language or something from one language to the other, so much of it is lost if you don't know the context. it's in what's the environments it's in. And so being able to attach environment and context and all those things together is going to require reinforcement learning to do. So again, if you want to get a copy of the tic-tac-toe board, it's kind of fun to play with. Uh run it, you can test it out, you can do u you know, test it for different uh uh values. You can switch from P1 computer uh where we loaded the policy one to load the policy two and just see how it varies. There's all kinds of things you can do on there. >> Supervised learning uses labeled data to train machine learning models. Labelled data means that the output is already known to you. The model just needs to map the inputs to the outputs. An example of supervised learning can be to train a machine that identifies the image of an animal. Below you can see we have a trained model that identifies the picture of a cat. Unsupervised learning uses unlabelled data to train machines. Unlabelled data means there is no fixed output variable. The model learns from the data, discovers patterns and features in the data and returns the output. Here is an example of an unsupervised learning technique that uses the images of vehicles to classify if it's a bus or a truck. So the model learns by identifying the parts of a vehicle such as the length and width of the vehicle, the front and rear end covers, roof hoods, the types of wheels used, etc. Based on these features, the model classifies if the vehicle is a bus or a truck. Reinforcement learning trains a machine to take suitable actions and maximize reward in a particular situation. It uses an agent and an environment to produce actions and rewards. The agent has a start and an end state, but there might be different parts for reaching the end state like a maze. In this learning technique, there is no predefined target variable. An example of reinforcement learning is to train a machine that can identify the shape of an object given a list of different objects such as square, triangle, rectangle or a circle. In the example shown, the model tries to predict the shape of the object which is a square. Here now let's look at the different machine learning algorithms that come under these learning techniques. Some of the commonly used supervised learning algorithms are linear regression, logistic regression, support vector machines, K nearest neighbors, decision tree, random forest and knive base. Examples of unsupervised learning algorithms are K means clustering, hierarchical clustering, DB scan, principal component analysis and others. Choosing the right algorithm depends on the type of problem you're trying to solve. Some of the important reinforcement learning algorithms are Q-learning, Monte Carlo, SARSA and deep Q network. Now let's look at the approach in which these machine learning techniques work. So supervised learning takes labeled inputs and maps it to known outputs which means you already know the target variable. Unsupervised learning finds patterns and understands the trends in the data to discover the output. So the model tries to label the data based on the features of the input data. While reinforcement learning follows trial and error method to get the desired solution. After accomplishing a task, the agent receives an award. An example could be to train a dog to catch the ball. If the dog learns to catch a ball, you give it a reward such as a biscuit. Now let's discuss the training process for each of these learning methods. So supervised learning methods need external supervision to train machine learning models and hence the name supervised. They need guidance and additional information to return the result. Unsupervised learning techniques do not need any supervision to train models. They learn on their own and predict the output. Similarly, reinforcement learning methods do not need any supervision to train machine learning models. And with that let's focus on the types of problems that can be solved using these three types of machine learning techniques. So supervised learning is generally used for classification and regression problems. We'll see the examples in the next slide. And unsupervised learning is used for clustering and association problems. While reinforcement learning is reward based. So for every task or for every step completed there will be a reward received by the agent. And if the task is not achieved correctly, there will be some penalty used. Now let's look at a few applications of supervised, unsupervised and reinforcement learning. As we saw earlier, supervised learning are used to solve classification and regression problems. For example, you can predict the weather for a particular day based on humidity, precipitation, wind speed, and pressure values. You can use supervised learning algorithms to forecast sales for the next month or the next quarter for different products. Similarly, you can use it for stock price analysis or identifying if a cancer cell is malignant or benign. Now talking about the applications of unsupervised learning, we have customer segmentation. So based on customer behavior, likes, dislikes and interests, you can segment and cluster similar customers into a group. Another example where unsupervised learning algorithms are used is customer churn analysis. Now let's see what applications we have in reinforcement learning. So reinforcement learning algorithms are widely used in the gaming industries to build games. It is also used to train robots to perform human tasks. >> Have you ever wondered how your favorite online store seems to know exactly what you are looking for? Every time you browse, add to cart or wish list and item, you are leaving clues about your style, favorite colors, brands, and even shopping times. Data scientists jump in, analyze these patterns and create a super personalized shopping experience. Suddenly, the store is showing you just the right pieces at just the right time, almost like it's reading your mind. That's data science, turning your clicks into a shopping spree crafted just for you. Hello everyone, welcome back to Simply Learns YouTube channel. If you're already a data science enthusiast or just got curious about this exciting field, you're in the right place. Today in this video, I'm diving into 10 essential steps to help you become the next in- demand data scientist and land that dream job. No more waiting. Let's dive right in and get you on the path to your future in data science. So, let's see the 10 essential steps to become the next data scientist in demand. Step number one is programming languages. Starting with Python is a beginner is a great move because it's simple, versatile, and widely used in data science. Python straightforward syntax makes it beginner friendly, helping you grasp programming basics quickly and dive into data science libraries like pandas, numpy and mattplotive with ease. Adding R to your skill set is valuable because it excels at statistical analysis and data visualization, two essential parts of data science. you can be comfortable with Python and R within a month or two. So moving on to the next step that is version control system. Learning a version control system like Git is essential because it allows you to track, manage and collaborate and code effectively. With Git, you can save different versions of your work, making it easy to backtrack if something goes wrong or to experiment without losing progress. This is especially useful when working with complex data science projects where you might try out different models of analysis techniques. One or two weeks for practice along with Python and R is good to get start. Now moving on to the third step that is data structures and algorithms. Learning data structures and algorithms is crucial for becoming a data scientist because they provide the foundation for efficient data handling and problem solving. Data structures like arrays, stacks, cues, and trees help you store and organize data in ways that make it easier and faster to access, process, and analyze. Algorithms, on the other hand, give you strategies to perform tasks like searching, sorting, and optimizing data operations, which are essential for handling large data sets. While many candidates struggle with the essay, mastering it gives you an age, helping you stand out in the interviews and shine as a skilled data scientist capable of tackling the toughest data problems. Spend about 2 months in this, you will get in the shape for sure. Now moving on to the step number four that is SQL. Learning SQL is essential for data scientists because it enables you to access, manage and manipulate data directly within databases where most real world data resides. With SQL, you can create new tables, alter existing ones, delete unnecessary records, and run queries to filter, sort, and aggregate data. These abilities allow you to retrieve, clean, and organize data effectively. Core skills needed for any data science role. It's easy, and you don't have to spend more than a month to have a deep understanding of it. Now, moving on to the fifth step that is mathematics and statistics. Mathematics and statistics are essential for data science because they form the backbone of data analysis, model building and interpretation. Topics like linear algebra, calculus, probability, and statistics gives data scientists the tools to understand data patterns, perform accurate analysis, and make datadriven decisions. Mastering these areas enables you to build robust models, validate results, and tackle complex problems confidently, making you a well-rounded and skilled data scientist. Make sure you spend two months to grabs this topics. Now moving on to the step number six that is data prep-processing and visualization. Learning data prep-processing and visualization is essential for a data scientist because these skills make you data accurate, insightful and easy to understand. Python libraries like NumPy and Panders are crucial for manipulating and creating data enabling you to handle missing values, filter out noise and prepare data for analysis. Once the data is ready, visualization lets you uncover patterns and communicate results effectively. Libraries like Mattplot tip and seaborn help create clear, impactful visuals, allowing you to interpret trends and convey insights in a way that's easily understood by others. Together with these tools make data prep-processing and visualization fundamentals for effective data science. If you have a solid foundation on Python and mathematics, you will get a good understanding of data prep-processing and visualization in a month or two. Now moving on to the seventh step that is machine learning fundamentals. Machine learning fundamentals involve understanding how algorithms enable computers to learn from data and make predictions on decisions without explicit programming. The two main categories are supervised learning and unsupervised learning. In supervised learning, models are trained on labelled data to make predictions. While in unsupervised learning, models find patterns in unlabelled data. Popular tools like TensorFlow, PyTorch help build and train complex models, especially for deep learning. While Skyit learn is essential used for simpler machine learning algorithms and data prep-processing. These tools make it easier to implement machine learning fundamentals effectively and build intelligent datadriven decisions. Dedicate about 3 months to understand the core of machine learning. Now coming to the next step that is deep learning. Deep learning is a subset of machine learning that focuses on algorithms inspired by the structures of the human brain called neural networks. Deep learning uses neural networks with multiple layers often dozens or hundreds to learn complex patterns from large data sets. Specialized types like convolutional neural networks that is CNN's are great for image processing while recurrent neural networks RNNs are used for sequence data like text or time series. Essential tools like TensorFlow, PyTorch make building, training and deploying deep learning models more accessible allowing you to create powerful AI solutions across various domains. I think it will take about two months to have a good hold on deep learning concepts and how to implement them. Now moving on to the ninth step that is specializations. Once you have grasped the deep learning, it's like reaching a new level as a data scientist. Just as doctors specialize in areas in nephrology and cardiology, data scientists often choose to specialize in fields like natural language processing or computer vision. Natural language processing focuses on teaching machines to understand and generate human language, enabling applications like chatbot, sentiment analysis, and language transition. It's about making computers read, write, and even interpret human emotions through text or speech. Computer vision, on the other hand, is all about enabling machines to see and interpret images or videos. This field powers innovations like facial recognition, object detection, and autonomous driving. Now you don't need to learn both. You can choose what interests you the most. Now spend one to two months diving deep into one of these areas. Now moving on to the last but not the least step that is big data. Big data refers to extremely large volumes of data generated rapidly from sources like social media and sensors. For data scientists learning to handle big data is crucial as it requires specialized tools like Hadoop and Spark to analyze and extract insights effectively. With companies relying on datadriven decisions, big data skills make you a highly in- demand professional in the field. Focus for about two months and you will be able to spot trends and patterns from data sets very easily. Once you're ready, it's time to build a killer resume packed with projects that showcase your new skills. Start applying to jobs on platforms like Noy and Indate and supercharge your LinkedIn. Connect with data scientists. See what skills they are mastering and learn from their journeys as well. Keep sharpening your own skills and when the time comes you will be ready to crush those interviews and land your dream data scientist role in 2025. >> Yeah, step by step we will go through all of this and uh we'll make sure that we learn everything and we bring everything together towards the end. Right, without further ado, let me just straight away deep dive to business, right? To learn data science, right? And with this data science there is also something which is prefixed which is applied data science and suffix for this is with Python right apply data science with Python right so there are there are two key concepts which are going to be a part of this course the first one is the knowledge about data science that what data science is and Then because we are doing an applied course right we are doing an applied course I will try to tie up these concepts which we will understand in data science with a tool right which is Python for you right we already know about uh 60 65% of Python right which is the fundamental Python and now we will be moving to the next step to advanced Python right and using Python Right. Leveraging Python we will be solving a lot of problems of data science right using this. Okay. So the first few sessions right the first few sessions will be about making you a breast with Python what Python is right what how and what packages do we have how do they work in reality right and all those things. And then we will be coupling it up with the data science concepts. And then finally towards the end of the session in the last few classes we will be doing uh we will be taking a real data set and on that data set we will be applying all these concepts right to understand the data better and we will be drawing inferences from that to convert that into information to take actionable insights or using that actionable insights taking a better decision right we'll do all of that in in the actual way okay so now guys If you understand this then the next point of contention is data science right one of the most famous keywords on the planet right now right one of the most famous keywords in the planet right now do you think that these two things okay let me put it different way what do you think that can be the possible explanation about this term data science you know data you know science what do you think is going to follow in these sessions. What is data science to you as per these two words? Okay. So this is people made up of two words, right? Data and science, right? So what we are trying to do is we are trying to understand data, right? We are trying to understand data, right? and then do something to it, right? Understanding its science, understanding the uh nature, the behavior of this data and converting it in something called as information, right? Do do we know difference between data and information? Data is something which is completely raw. Okay? It is completely raw. It has no meaning. Right? It has no meaning isn't it? For example, I give you these stats of some player like suppose say Dhoni, I give you stats of Mahindra Singh Dhoni, right? That what what what were his scores? Uh what is his name? what is his age and you know all those things now everything is there right but we don't know what to do about it right do you think the score of Dhoni has any context people it has any context no right but when I deep down but I when I go and deep dive about it right what is the first thing you find out of scores what is the first thing you find out of score scores you try to find the average of score isn't it that in last 10 innings Right? In last 10 innings before this also you need something which is called as a problem statement isn't it? Now for example the problem statement is selectors want to understand selectors wants to understand that whether Mahindra Singh Dhoni should be picked up. So there's a problem now right selectors want to see that whether Dhoni is fit for the next tournament or not. So what we will do we will now try to take the mean of the scores right for last 10 innings and if this score is suppose X we will try to compare this with Y. What is Y? Y is a reference right Y is a reference that we want to compare it against. Now when you are doing this comparisons when you are applying these techniques to this score this is now slowly becoming information right and at the end of the day once you have the strike rate once you have the mean score of Dhoni once you have his age once you have his fitness score all those things will now help you to take this particular decision because now what you have is called as information, right? Because this has context, right? This has meaning and this is usually processed, right? This is usually processed, right? This is usually processed. Now, what did we do? Now, what did we do here? If you will go and read about data science, data science says, data science says it is the art of collecting, right? Cleaning, analyzing, modeling, improving Right. And visualizing, right? Visualizing the data. Right? If a person is adept in doing all these things, this person people is cumulatively called a data scientist. Right? That person is called a data scientist. Right? So before going to the definition of data scientist now I will give you some more examples right I'll give you some more examples data science people as I said is a combination of these things right you have to collect the data right you have to collect the data right and this has a lot of things data can be connected from two types in two types one is primary and the second one is secondary Right. What is the primary way of collecting data? From your IoT devices, right? From sensors, from your in-built machines, right? Then from your surveys which you float, right? Questionnaires, right? All these things are primary ways. What is the secondary way of collecting data? Purchasing data. Right? Using internet data because you have not generated it. You are just using someone else's data. Right? Something like uh transfer learning. What is transfer learning? Transfer learning is a technique where suppose I am bank A and you are bank B. Right? So bank A has created some model right trained on their data. Now you are going to use the exact same model right you're going to use the exact same model right maybe you're not seeing the data but you are just using the property of data like mean median mode and a lot of modeling things which will come we will learn about them and you use this model on your particular data right so in a way you did not have enough data to create the model yourself but you are now using someone else's model to run your data on it right so this is called as transfer learning So this kind of collection is basically secondary data collection. So you can collect the data right then you can perform data analysis right you can perform data analysis right. How will you perform this data analysis using complex algorithms right? using complex algorithms, right? Some statistics, right? You can use artificial intelligence, right? Artificial intelligence, you can use machine learning. Right. Right. You can use all these things for data analysis. Then you can transform transform the patterns into predictions, right? You can transfer these patterns into predictions, right? Which can be used for business decision making. right for business decision making then you can validate the results right and present the results right so this is like a complete life cycle of a data scientist right so before going further let me give you what combinations do you need to have to become a data scientist the first one is domain knowledge right so what is domain knowledge first of all I told you right there will be a problem right you'll be solving a problem in any project of data science you'll be trying to solve a problem right and the problem will be belonging to a particular domain even if you're working for yourself right even if you're an entrepreneur then also you'll be solving a problem so this domain knowledge part includes things like understanding. It's a very important diagram. Understanding client requirement, right? Understanding the client requirement, right? Important criterians, right? Important criteria knowledge, right? For example, to give an example, suppose we have created a machine learning model. Okay. Understanding the data, we have created a machine learning model whose accuracy is 90%. Right? Is 90% a good accuracy? Yeah, fairly decent accuracy. Yes. Suppose you have to predict sales, right? You're selling something. Suppose you're selling clothes and you want to predict what will be the sales for the next week. When you use this model, whatever the output model gives you, what is going to be the accuracy of your output using this model? How much accuracy? 90%. But my question is model is 90%. But my question to you is that if 90% accuracy is on sales data, a person like me will be very very happy. Okay? Very very happy. I'll be probably dancing, right? But if you try to apply the same model right for a medical diagnosis case will you be interested in getting operated in such a hospital or an institution where the accuracy is coming as 90%. Domain knowledge right domain knowledge we need to understand what are the exact requirements we need to understand what are the exact expectations right and we need to know how much do we need to pivot right so the first thing in data science is these accuracies and everything are subjective right they are subjective so for that you need domain knowledge domain knowledge part very important guys very important these three things which I'm going to tell second part is people the game changer right the second part is computer science now if I take you back in history okay if I take you back in history in 1980s or somewhere then do you okay how many of you think that data science is a new concept how many of you think that data science is a new concept I hope you all know it's not a new concept everyone knows that yeah it has been happening for ages is just like you guys will be shocked if you already don't know AI was coined in the year 1956 1956 at the University of Damouth AI was coined by Paul McCarthy right and we saw the boom of AI in the year 2010 right such a long journey same case with data science because people back in the day data science was called as data mining Everyone heard about it data mining, knowledge databases. Yeah, we need we used to mine the data. Now what were the problems? What were the hiccups of data mining? The hiccups for data mining was that we were doing everything everything manually. Right now if I give you 100 points, can you calculate the mean? Or let's say if I give you two points to multiply 2 * 3, how much time will you take? 2 seconds. Yep. How much time a computer will take? 2 seconds. If I give you to multiply 2 489 * 200, how much time will you take to calculate this? Say 5 seconds. How much time computer will take? 2 seconds. Now if I give you to multiply 2 489 into 15 1 95 4386, how much time will you take to calculate this manually? Maybe say 1 minute 80 seconds. 1 minute. How much time a computer will take? Still 2 seconds, right? Still 2 seconds, right? And if I give you to calculate this over 200 times, you will take 200 minutes, right? using parallel computing computer will still take about 3 to 5 seconds. Right? So are you understanding the power of computer? Do you understand this concept in this relationship? What was happening back then? What computer science did people was it revolutionized the way data mining was happening and that thing now is called as that again that thing now is called as data science in which computer science is one of the most important contributors. So this is just one reason. Now manually right manually if I give you say 1 million rows of data right 1 million rows of data right so how many pages will you pages will you need to store this data suppose your notebook is like this right these boxes and here you are storing the data 1 million so maybe you can buy n number of notebooks but now do you think it's as easy in storing something in computer because back in the day people we had memory issues isn't it memory constraints? So there is something called as Mur's law right which says as the advancement in microprocessors will increase the price of microprocessor will decrease right so this is what is happening right now back in 1980s if I show you right guys right yeah 2.5 kg exactly right it was size of a fridge hard disk but now it fits in your palm right so this was enabled the storage techniques right the processing techniques the infrastructure, right? Things like big data. What kind of data do you think we will be dealing with people in data science? You all know the term, very famous term, the kind of data, big data, right? Everyone knows about big data. What is big data? Yes. A data which is fast, right? It has velocity, veracity, variety, right? So, this kind of data needs to be stored. this kind of data needs to be processed. So which thing brought all these things into data science? It was given to us by computer science, right? So computer science people included things like database management, right? Data validation, right? Data infrastructure, right? Data infrastructure, right? Then we had languages, computer languages which is Python right now for us. Right? Again, do you think when you do this thing manually, right? Suppose you do this thing manually, how easy do you think it will become using something like Python or any other computer language to create complex models. How easy it will be to do that to create the complexity in models, right? where you can capture the nonlinear nature. Isn't it people? Isn't it? For example, for example, let me tell you this. 2 4 6 8 10 dash. What do you think is the next number, guys? 12. If I tell you to define this to me in Okay, let leave that. What do you think is going to be the next number here? What is the next number? 11. Next number 25. Perfect. Now guys, if I ask you to write these numbers, right, the way you predicted them, can you give me a function f ofx is equal to what is the f of x here? It's 2x right? It's 2x. f of x is equal to 2x. If I tell you to create a function, it will be f of x is equal to 2x. What will be the function here guys? f ofx will be equal to x + 1. Yeah. x + 1. Yeah. And here f ofx will be equal to x². Yeah. Now the last example, right? Last example. What is the next number here? I don't want the number. I want the function. I want this so that I can generalize. Isn't it? How did you reach this figure? How many of you think it's not possible to determine this? How many of your think it is not possible to determine this. Yeah. How many of you think what if I just change this question and ask you how many of you think it is not possible to determine this manually? same response but now I say how many of you think it is not possible to determine this with computers will your answer still remain no do you think I cannot approximate this function using computers we have something called as deep neural networks and they are called as universal function approximators right so this is the problem people this is the problem right I will show this to you when the time comes right I will remember this example and I will show this to you but now what I'm trying to tell you is the things which seemed impossible manually was solved by what it was solved by computers the distribution of this is like this can you figure it out yourself no right we cannot isn't it we cannot do that so this kind of approximation will be given by what it will be only given by machines right and this is people what data science is all about right it is what computer science did inside data science right I hope this is clear I'm assuming a lot of you will be going for interviews and everything after these course right so this will be a very very important thing for you to know right often it is asked why data science is having computer science in it right the reason is this okay so this is the role of computer science inside data science. Now people the third thing the third circle which is one of the most parts is maths and stats right mathematics statistics which was optimization right optimization of your models right design of model. Right? Now guys, if you look carefully, if you look carefully, in order to approximate this, what you what will you be playing with? You will be playing with a lot of data. You will be playing with a lot of mathematical tool, mathematical concepts and statistical concepts, isn't it? How did you what do you call this? This is maths, right? This is statistics and mathematics. finding mean, median, mode, standard deviations, probability, statistics, all these will lead to this kind of result, isn't it? So, this becomes the third wheel of this particular uh of of this particular diagram. And this point of intersection, right? The sweet point of intersection is basically data science. Yeah. is particularly data science. Right? So now people this point okay this point is basically representing data engineering right data engineering right data engineering is people the part of data science which enables us to capture the correct data right how the data will flow how the data will be stored how the data will be cleaned right all this is done by home it is done by data engineering Just because so am I right there guys you go for interviews right after this and try to fetch yourself jobs in this domain data science AI ML if my understanding is correct is that the aim yes so now guys there will be three types of companies or let's say to simplify let's say two types one is small and the other one is big right so in a small organization if you become a part of small or organization and you are the data scientist there You can be involved in all of these things, right? All of these things possible, right? Your bosses and your management will expect you to construct all these flows, right? Know computer science, you should know maths and stats and you should have the domain knowledge and you will be asked to do all of this. But if you are going to become a part of a big organization, usually all these roles are fragmented. All of these roles are fragmented, right? There's a separate data infrastructure team. There's a separate data governance team. Now guys, when you go on to collect the data, can you collect any sensitive data about is it possible ethically it's not right? And legally also it's not right. My question to you is who will look after this compliance? Whose responsibility indeed it is to look after this compliance? Data scientist. So this is about fragmentation. If you are part of a big organization, this thing will be taken by someone else, right? But if you are a part of a small organization, you will be know you'll be e