Data Science Full Course 2025 | Data Science Tutorial | Data Science Training Course | Simplilearn

Key Takeaways

Welcome to the data science full course. Data science is one of the most in demand skills today. And in this course, we are going to help you unlock its full potential. Imagine turning raw data into powerful insights that help businesses make smarter decisions. That's exactly what data science is all about. We'll begin by unboxing the course and giving you a clear road map to become a data expert. You'll get hands-on with Python, the most popular language in data science. We'll guide you step by step through the installation and all the key fundamentals. You'll also get an introduction to the large language models, a game-changing technology that's transforming how data is used. And on the top of that, we'll walk you through common data science interview questions to help you prepare for real world opportunities. So, let's dive in. Now before we begin, if you're interested in learning how tools like Chad GPD work or you want to build a career in AI and data science, then this course is a great place to start. The professional certificate in data science and generative AI by Pura University online and simply learn is a six-month online program that covers Python, machine learning, deep learning, NLP, generative AI like prompt engineering and chbd. You'll also learn through live classes from Purdu and IBM expert. Complete 25 plus hands-on project and earn certificate from both PU University and IBM. Plus, you will get career support with resume help, mock interviews, and even job search guidance. So, what are you waiting for? Hurry up and enroll now and you can find the link below. >> Today, we are going to take a look at what this buzz is all about. So, what is the agenda for today? We will talk about what is the need for data science and then what exactly is data science some definitions. Then we'll talk about the prerequisites for learning data science and then what does a data scientist do? What are the activities performed by a data scientist as a part of his daily life and then we will talk about the data science life cycle with a quick example and briefly touch upon the demand or everinccreasing demand for data scientists. All right. So let's get started. Now you must have already heard about autonomous cars. So I'm sure you must be excited to have a car driving by itself which will take you from home to office or office to home right and that's where one of the examples where data science is used. Now the car needs to take a lot of decisions in this whole process whether to speed up whether to apply the brake take a left turn right turn or slow down. So all these decisions are basically a part of data science and there is a study that says that self-driving cars will minimize accidents and in fact it will root out more than 2 million deaths caused by car accidents annually. self-driving cars. Right now, there's a lot of research and there is a lot of testing going on and not a lot of cars are yet in production in terms of usage, but it's going to happen. Every automotive company worth its name is investing in self-driving cars. So, in about 10 to 15 years, some of the studies say that most of the cars will be autonomous or self-driving cars. Where else there are issues? For example, if we take airlines, this is another area where data science contributes in a big major way. Flights get delayed due to weather conditions because the weather is not predicted in time and the demand of passengers is not properly seen ahead of time. For all these you need data science. Then this could be improper route planning and uh some customers might miss some flights that again needs data science and similarly it could be incorrect decisions in selecting the right equipment. So which plane should fly in which route? That's the equipment that's being mentioned here. If that is not planned properly then you might end up in a situation where the plane is not available whereas you have planned for a flight in a particular route. So these are some of the challenges in one of the representative industries we are talking about which is airlines. So if we use data science properly all of these or most of these problems can be avoided and that will help in reducing the pain both for the airlines and also for the passengers. A few more examples. What else can we do? Here are some of the other things that we can do and we will stick to the airline industry. We can do better route planning so that there are less cancellations and less frustrated people. We can predict use predictive analytics and predict any delays that are there so that some flights can be rescheduled ahead of time and there are no lastm minute changes. Data science can also be used to make promotional offers and the last but not least is what kind of planes should be used or the different classes of planes that should be used in different routes for better performance. So these are some examples of how data science can be used in airlines. And another example or another industry where data science can be used and benefited would be in logistics. So companies like FedEx, they use data science models to increase their efficiency drastically to optimize the routes and cut costs and so on. So before their delivery truck actually sets out, they determine which is the best possible route to ship their items to the customers and based on various inputs they also predict or come up with the best suited time to deliver and last but not least they determine what is the best mode of transport for this delivery as well. So what is data science used for? These are some of the main areas where data science is used for better decision making. There are always tricky decisions to be made. So which is the right decision which way to go. So that is one area. Then for predicting for performing predictive analysis like for example can we predict delays like in the case of airlines can we predict the demand for certain products let's say in e-commerce. That is the second area. The third area is pattern discovery or pattern recognition. Is there a pattern in which people are buying items? For example, it could be seasonality. If you take the data, sales data for multiple years, there may be a pattern in the way people are buying that. That's a buying pattern. In certain months, probably the sales will increase, certain months, the sales will come down. Certain quarters traditionally the sales will be higher certain quarters. So that is a pattern and this pattern discovery is another area where data science is applied. So what is data science? Now let's take an example, a real life example. On a day-to-day basis, we use or we we try to make some decisions. Let's say we want to buy some furniture online for our new office. So how do you go about doing this? You need to take a bunch of decisions to actually do the purchase. So we start with which website, which portal or which website you should use. So we try to find out let's say you want to buy the furniture. Obviously you don't go to a online grocery store because you need furniture because there are several websites. So that is the first decision you need to take which website I should use. So once we have multiple websites you kind of discard all the websites which don't sell furniture and you stick to those websites which sell furniture. Now within that we try to find out what is the ratings of this websites. If the ratings is more that means they are reliable the quality probably is good and so on and so forth. So only then you want to buy from that particular website. So anything that doesn't satisfy this criteria you close all those websites. Close in the sense you close the browser right. So you still are left with maybe a few of these which satisfy your criteria that is which sell web pages or websites that sell furniture they have a rating of four and above and then you look for discount is somebody providing discount greater than 20%. Then again you filter out some of them which probably are not providing any discount and zero down to one or two websites which are probably providing those discounts and go ahead and select the furniture and purchase it. So this a very basic example you probably don't follow this always exactly the same way but just to illustrate drive home the point. So we can answer a lot of questions using data science. For example, when we take a cab, when we book a cab, now to go from location A to location B, what is the best route that the cab can take to reach in the fastest way or in the least amount of time? There could be several factors. There could be traffic, there could be bad road, there could be weather. Now, all these come as inputs and a decision needs to be taken as to which is the best route. Another example is TV shows. So Netflix and uh maybe even other a lot of other TV channels they have to perform this analysis to find out what kind of shows people are viewing, what kind of shows people are liking and and so on and so forth so that they can then sell this information to advertisers because their main source of revenue is advertising. So this is again major function of data science predictive maintenance. We need to find out will my car break down? Will my refrigerator break down in the next year or 2 years? Should I be prepared to buy a new refrigerator? You can potentially apply data science here as well. And then in politics, a lot of data science is applied in politics. You must have seen on TV about US elections or in UK or even in India. Nowadays, everybody is applying data science in elections and trying to capture the votes or rather the voters and influence the voters, personalized messages, providing personalized messages and and so on and so forth. And that is one. Not only that, people use data science to even predict who is going to win the elections. It's a different matter that probably not all predictions come out to be true. But then yes, this is they use data science to do this uh predictions. So what is the process or what are the various steps in data science? The first step is asking the right question and exploring the data. Basically you want to know what exactly is the problem you're trying to solve. So that is asking the right question. So that is the this circle out here. Then the next step is after exploring the data. So as the first step you will ask some questions. What exactly is the problem you're trying to solve? And then obviously you will have some data for that as input and you perform some exploratory analysis on the data. For example, you need to clean the data to make sure everything is fine and so on and so forth. So all that is a part of exploratory analysis and then you need to do the modeling. Let's say if you have to perform machine learning, you need to decide which algorithm to use and which model to use and then you need to train the model and so on and so forth. So that's all part of the modeling process and then you run your data through this model and then through this process and then you come out with the final results of this exercise which includes visualizing the results and preparing a way to communicate the results to the concerned people. So it could be in the form of PowerPoint slides or it could be in the form of a a dashboard which is basically what we call as a visualization. And so all the insights that have been gathered through this exercise that has to be communicated in a proper way in a easy to understand way which is again a a key part of this whole exercise communicating the results. So what are the prerequisites for data science? There are three essential traits required for to be a data scientist. One is curiosity. You need to be able to ask questions. The first step in data science is asking question. What is the problem we are trying to solve? If you ask the right question only then you'll get the right answer. Very often this is a very crucial step where a lot of data science projects fail because you're you may be asking the wrong question and then obviously when you get the answer that's not the answer you're looking for. So it is very important that you ask the right question. Needless to say then the second part or the second trait is common sense. So you need to be creative. You need to come up with ways to use the data that you have and try to solve the business problem on hand. In many cases you may not have all the data that you need. In many cases the data may be incomplete. So that is where you need to come up with ways what are the best ways to fill these gaps wherever this is missing. And that's where common sense comes into play. Last but not least after doing all this analysis if you're unable to communicate the results in the right way the whole exercise will fail. So communication is a key trait for a data scientist. Maybe technically you may be a genius but then if you are unable to communicate those results in a proper way once again that will not help. So these are the three main traits curiosity common sense and communication skills. In a way you can say these are the three C's. Okay. So what are the other prerequisites? The first one so machine learning machine learning is the backbone of data science. Data science involves quite a bit of machine learning in addition to the basic statistics that we do. So a data scientist needs to have a good hang or need to be very good at data science. The second part is modeling. So modeling is also a part of machine learning in a way. Um but you need to be good at identifying what are the algorithms that are most suitable to solve a given problem. what models can we use and uh how do we train these models and so on and so forth. So that is the second component. Then statistics. Statistics is like the core foundation of data science. So you need to understand statistics and you need to have a good hang of statistics in order to be a good data scientist and this will also help in getting good results. Programming is to some extent required at least some program or the other would be required as a part of executing the data science project. The most common programming languages are Python and R. Python especially is becoming a very popular programming language in data science because of its ease of learning because of the multiple uh libraries that it supports for performing data science and uh machine learning and so on. So, Python is by far one of the most popular languages in data science. If any one of you is is wanting to learn a new language, that should be Python. And then, of course, you need to understand databases, how databases work and how to handle databases, how to get data out of databases and so on. So, these are some of the key components of data science. Now coming to the tools and skills that are used in data science. These are some of the skills. From a language perspective, it is Python or R. And from a skills perspective, in addition to some of the programming languages, it would help if you have a good knowledge or good understanding of statistics. And what are the tools that are used in data analysis? SAS is one of the most popular tools. It's been there for very long time and uh that's the reason it is very popular and however this is compared to most of the other tools it is a preparatory software whereas Python and R are mostly open source the other tools are like Jupiter Jupyter notebooks you have R studio these are more development environments and development tools so Jupiter notebooks is a interactive development environment uh similarly R studio is for performing or writing our code and performing analytics and um performing data analysis and machine learning activities you can perform in R studio. It has a very nice UI and initially R was not so popular primarily because it did not have user interface and R studio is a relatively new addition and after the advent of R studio R became extremely popular and there are other tools like MATLAB and of course some people do with Excel as well. As far as data warehousing is concerned some of the skills that are required are ETL. So in order to extract data and uh transform load, ETL stands for extract transform load. So you have data in the databases like your ERP system or a CRM system. You need to extract that and then do some transformations and then load it into your warehouse so that all the data from various sources looks uniform. Then you need some SQL skills which is basically querying the data, writing SQL queries. Hadoop is another important skill especially if you are handling large amounts of data and also one of the specialtities of Hadoop is it can be used for handling unstructured data as well. So it can be used for large amounts of structured and unstructured data then spark is a excellent computing engine for performing data analysis or machine learning in a distributed mode. So if you have large amount of data, the combination of Spark and Hadoop can be extremely powerful. So you store your data in Hadoop HDFS and use Spark as your computation engine. It works in a distributed mode similar to Hadoop like a cluster. So that those are excellent skills for data warehousing and there are some standard tools that are available like Informatica, data stage, talent and also AWS redshift. If you want to do some on the cloud, I think AWS red shift is again a good tool. Data visualization tools for data visualization. Some of the skills that would be required are let's say R you R provides some very good visualization capabilities especially for for developing during development and then you have Python libraries mattplot lib and so on which provides very powerful visualization capabilities and that is from skills perspective whereas tools that can be used are uh Tableau is a very very popular visualization tool again that's a proprietary tool so it's a Little expensive maybe but excellent capabilities from a visualization perspective. Then there are tools like Cognos which is an IBM product which provides very good visualization capabilities as well. And then coming to the machine learning part of it the skills required there are Python which is more for programming part and then you will need some mathematical skills like algebra, linear algebra especially and then statistics and maybe little bit of calculus and so on. And the tools that are used for machine learning are spark mlib and apache maho and on cloud if you want to do something uh you can use microsoft as your ml studio as well. So these are by no means an exhaustive list there are actually many many tools and probably a few more skills also maybe there but this is this gives a quick overview like a summarizing of summarization of the tools and skills. Now moving on to the life of a data scientist. What does a data scientist do during the course of his work? So let's see. So typically a data scientist is given a problem, a business problem that he needs to solve. And in order to do that, if you remember from the previous slide, he basically asks the question as to what is the problem that he needs to solve. So that is the first thing he has got the problem. Then the next thing is to gather the data that is required to solve this problem. So he goes about looking for data from anywhere. It could be the enterprise. Very often the data is not provided in the nice format that he would like to have it or we would like to have it. So first step is to get whatever data that is possible what is known as raw data in whatever format. So it could be enterprise data it could be there is a probably a requirement to go and get some public data in some cases. So all that raw data is collected and then that is processed and analyzed and in prepared into a format in which it can be used and then it is fed into the analytic system be it a machine learning algorithm or a statistical model and we get the output and then he puts these output in a proper format for presenting it to the stakeholders and communicating those insights or the results to the stakeholder. So this is a a very high level view of like a a day in the life of a data scientist. So gathering data, raw data, performing some quick analysis on that and maybe processing or manipulating this data to bring it into a certain good format so that it can be used for the analysis. feeding this into that analysis system that has been designed be it mathematical models uh machine learning models and then get the results the insights and then present it in a nice way so that the stakeholders can understand how about machine learning algorithms. So let's see what are the various machine learning algorithms that would be required for a data scientist. So these are a few of the algorithms. Again this is not an exhaustive list. we have regression is one of the supervised learning models or techniques. So in case of regression you try to let's say come up with a continuous number. So the difference between regression and let's say a classification is that in case of classification those are discrete values whereas here we are talking about regression where you let's say you are trying to predict the temperature which is a continuous value or the share price which is a continuous value. So that is regression. So you need to know what is regression how to perform regression and we need to understand clustering. So clustering is one of the unsupervised learning techniques. In this case, there is no label data that is available and you get some data and then you want to put this into some shape so that you can analyze it and you try to make sense out of it. Let's say you have one example is you have a list of cricketers and they have not been marked as bowlers and batsmen or allrounders or whatever, right? So you just have their names and maybe how many runs they scored, how many wickets they have taken and so on. But there is no readily available information saying that okay this person is a batsman, this person is a bowler and so on. So I'm talking about cricket. Hopefully most of you are familiar with the game of cricket. So how do we find out? So then we put this into a clustering mechanism and then the system will say that okay these are the people who are all who have all scored good amount of runs so they belong to one cluster. These are all the people who have taken good amount of wickets. So they belong to one cluster. And maybe here are some people who have taken good amount of wickets and they have made good amount of runs. So they may be belonging to one group. And then we take a look at it and then we label them as okay people who have all together and those who have you know scored many runs they are we label them as batsmen. People who have taken a lot of wickets we label them as bowlers. and people who have taken good amount of wickets and also made some good runs. We label them as allrounders. But the system will just say okay this is cluster one, cluster two, cluster three. The names we give, we human beings have to give the names. Now decision tree is used for what is known as classification. Primarily it can also be used for regression but by and large it is used for classification. And here again it's a very logical way in which the algorithm goes about classifying the various inputs. One of the biggest advantages of decision tree is that it's very easy to understand and it's very easy to explain why a certain object has been classified in a certain way compared to maybe some of the other mechanisms like say support vector machines or logistic regression and so on. So that's the advantage of diction tree but that is also very popular algorithm. Then we have support vector machines primarily for classification purpose and uh then we have knives base. This is again a statistical probability based classification method. So these are a few algorithms. There are a few more that are not listed here but there are some more algorithms as well. And by the way, there are more detailed or there are detailed videos about each of these algorithms available. You can check in the playlist. So now let's talk about the life cycle of a data science project. Okay. The first step is the concept study. In this step, it involves understanding the business problem, asking questions, get a good understanding of the business model, meet up with all the stakeholders, understand what kind of data is available and all that is a part of the first step. So here are a few examples. We want to see what are the various specifications and then what is the end goal, what is the budget, is there an example of this kind of a problem that has been maybe solved earlier. So all this is a part of the concept study and another example could be a very specific one to predict the price of a 1.35 karat diamond and there may be relevant information inputs that are available and we want to predict the price. The next step in this process is data preparation, data gathering and data preparation also known as data munching or sometimes it is also known as data manipulation. So what happens here is the raw data that is available may not be usable in its current format for various reasons. So that is why in this step a data scientist would explore the data. He will take a look at some sample data. Maybe pick if there are millions of records, pick a few thousand records and see how the data is looking. Are there any gaps? Is the structure appropriate to be fed into the system. Are there some columns which are probably not adding value? May not be required for the analysis. Very often these are like names of the customers. They will probably not add any value or much value from an analysis perspective. the structure of the data. Maybe the data is coming from multiple data sources and the structures may not be matching. What are the other problems? There may be gaps in the data. So the data all the columns all the cells are not filled. If you're talking about structured data, there are several blank records or blank columns. So if you use that data directly, you'll get errors or you'll get inaccurate results. So how do you either get rid of that data or how do you fill this gaps with something meaningful? So all that is a part of data munching or data manipulation. So these are some additional subtopics within that. So data integration is one of them. If there are any conflicts in the data there may be data may be redundant. Yeah data resident redundancy is another issue. There may be you have let's say data coming from two different systems and both of them have customer table for example or customer information. So when you merge them there is a duplication issue. So how do we resolve that? So that is one data transformation. As I said there will be situations where data is coming from multiple sources and then when we merge them together they may not be matching. So we need to do some transformations to make sure everything is similar. We may have to do some data reduction. If the data size is too big, you may have to come up with ways to reduce it meaningfully without losing information. Then data cleaning. So there will be either wrong values or you null values or there are missing values. So how do you handle all of that? A few examples of very specific stuff. So there are missing values. How do you handle missing values or null values? Here in this particular slide we are seeing three types of issues. One is missing value. Then you have null value. You see the difference between the two right? So in the missing value there is nothing blank. Null value it says null. Now the system cannot handle if there are null values. Similarly there is improper data. So it's supposed to be numeric value but there is a string or a non- numeric value. So how do we clean and prepare the data so that our system can work flawlessly. So there are multiple ways and and there is no one common way of doing this. It can vary from project to project. It can vary from what exactly is the problem you're trying to solve. It can vary from data scientist to data scientist, organization to organization. So these are like some standard practices people come up with and and of course there will be a lot of trial and error. Somebody would have tried out something and it worked and will continue to use that mechanism. So that's how we need to take care of data cleaning. Now what are the various ways of doing you know if if values are missing how do you take care of that? Now if the data is too large and um only a few records have some missing values then it is okay to just get rid of those entire rows for example. So if you have a million records and out of which 100 records don't have full data. So there are some missing values in about 100 records. So it's absolutely fine because it's a small percentage of the data. So you can get rid of the entire records which have missing values. But that's not a very common situation. Very often you will have multiple or at least you know large number of data set. For example out of million records you may have 50,000 records which are like having missing values. Now that's a significant amount. You cannot get rid of all those records. Your analysis will be inaccurate. So how do you handle such situations? So there are again multiple ways of doing it. One is you can probably if a particular values are missing in a particular column you can probably take the mean value for that particular column and fill all the missing values with the mean value so that first of all you don't get errors because of missing values and second you don't get results that are way off because these values are completely different from what is there. So that is one way. Then a few other could be either taking the median value or depending on what kind of data we are talking. So something meaningful we will have put in there. If we are doing some machine learning activity then obviously as a part of data preparation you need to split the data into training and test data set. The reason being if you try to test with a data set which the system has already seen as a part of training then it will tend to give reasonably accurate results because it has already seen that data and that is not a good measure of the accuracy of the system. So typically you take the entire data set the input data set and split it into two parts and again the ratio can vary from person to person individual preferences. Some people like to split it into 50/50. Some people like it as 63.33 and 33.3. This is basically 2/3 and 1/3. And some people do it as 80/20. 80 for training and 20 for testing. So you split the data, perform the training with the 80% and then use the remaining 20% for testing. All right. So that is one more data preparation activity that needs to be done before you start analyzing or applying the data or putting the data through the model. Then the next step is model planning. Now this models can be statistical models. This could be machine learning models. So you need to decide what kind of models you're going to use. Again it depends on what is the problem you're trying to solve. So if it is a regression problem, you need to think of a regression algorithm and come up with a regression model. So it could be linear regression. Or if you're talking about classification, then you need to pick up an appropriate classification algorithm like logistic regression or decision tree or SVM and then you need to train that particular model. So that is the model building or model planning process and the cleaned up data has to be fed into the model. And apart from cleaning you may also have to in order to determine what kind of model you will use you have to perform some exploratory data analysis to understand the relationship between the various variables and u see if the data is appropriate and so on. Right? So that is the additional preparatory step that needs to be done. So little bit of details about exploratory data analysis. So what exactly is exploratory data analysis? It's basically to as the name suggests you're just exploring you just receive the data and you're trying to explore and uh find out what are the data types and what is the is the data clean in in each of the columns what is the maximum minimum value. So for example there are out ofthe-box functionality available in tools like R. So if you just ask for a summary of the table, it will tell you for each column it will give some details as to what is the mean value, what is the maximum value and so on and so forth. So this exercise or this exploratory analysis is to get an understanding of your data and then you can take steps to during this process you find there are a lot of missing values you need to take steps to fix those. You will also get an idea about what kind of model to be used and so on and so forth. What are the various techniques used for exploratory data analysis? Typically these would be visualization techniques like you use histograms. Uh you can use box plots, you can use scatter plots. So these are very quick ways of identifying the patterns or a few of the trends of the data and so on. And then once your data is ready, you you've decided on the model, what kind of model, what kind of algorithm you're going to use. If you're trying to do machine learning, you need to pass your 80% the training data or rather you use that training data to train your model. And the training process itself is iterative. So the training process you may have to perform multiple times and once the training is done and you feel it is giving good accuracy then you move on to test. So you take the remaining 20% of the data. Remember we split the data into training and test. So the test data is now used to check the accuracy or how well our model is performing and if if there are further issues let's say and model is still during testing if the accuracy is not good then you may want to retrain your model or use a different model. So this whole thing again can be iterative but if the test process is passed or if the model passes the test then it can go into production and it will be deployed. All right. So what are the various tools that we use for model planning? R is an excellent tool in a lot of ways. Whether you're doing regular statistical analysis or machine learning or any of these activities or in along with our studio provides a very powerful environment to do data analysis including visualization. It has a very good integrated visualization or plot mechanism which can be used for doing exploratory data analysis and then later on to do analysis, detail analysis and machine learning and so on and so forth. Then of course you can write Python programs. Python offers a rich library for performing data analysis and machine learning and so on. MATLAB is a very popular tool as well especially during education. So this is a very easy to learn tool. So MATLAB is another uh tool that can be used. And then last but not least SAS. SAS is again very powerful. It is a preparatory tool and it has all the components that are required to perform very good statistical analysis or perform data science. So those are the various tools that would be required for or that that can be used for model building. And uh so the next step is model building. So we have done the planning part. We said okay what is the algorithm we going to use? What kind of model we going to use? Now we need to actually train this model or build the model rather so that it can then be deployed. So what are the various uh ways or what are the various types of model building activities. So it could be let's say in this particular example that we have taken you want to find out the price of 1.35 karat diamond. So this is let's say a linear regression problem. You have data for various carats of diamond and you use that information you pass it through a linear regression model or you create a linear regression model which can then predict your price for 1.35 carat. So this is one example of model building and then little bit details of how linear regression works. So linear regression is basically coming up with a relation between an independent variable and a dependent variable. So it is pretty much like coming up with equation of a a straight line which is the best fit for the given data. So like for example here y is equal to mx + c. So y is the dependent variable and x is the independent variable. We need to determine the values of m and c for our given data. So that is what the training process of uh this model does. At the end of the training process, you have a certain value of m and c and um that is used for predicting the values of any new data that comes. All right. So the way it works is we use the training and the test data set to train the model and then validate whether the model is working fine or not using test data and uh if it is working fine then it is taken to the next level which is put in production. If not the model has to be retrained. If the accuracy is not good enough then the model is retrained maybe with more data or you come up with a newer model or algorithm and then repeat that process. So it is an iterative process. Once the training is completed training and test then this model is deployed and we can use this particular model to determine what is the price of 1.35 karat diamond. Remember that was our problem statement. So now that we have the best fit for this given data, we have the price of 1.35 karat diamond which is 10,000. So this is one example of how this whole process works. Now how do we build the model? There are multiple ways. You can use Python for example and use libraries like pandas or numpy to build the model and implement it. This will be available as a separate tutorial, a separate video in this playlist. So stay tuned for that. Moving on, once we have the results, the next step is to communicate this results to the appropriate stakeholders. So which is basically taking this results and preparing like a presentation or a dashboard and communicating these results to the concerned people. So finishing or getting the results of the analysis is not the last step. But you need to as a data scientist take this results and present it to the team that has given you this problem in the first place and explain your findings explain the findings of this exercise and recommend maybe what steps they need to take in order to overcome this problem or solve this problem. So that is the pretty much once that is accepted and the last step is to operationalize. So if everything is fine your data scientists presentations are accepted then they put it into practice and thereby they will be able to improve or solve the problem that they stated in step one. Okay. So quick summary of the life cycle. You have a concept study which is basically understanding the problem asking the right questions and trying to see if there is uh enough data to solve this problem and then even maybe gather the data. Then data preparation the raw data needs to be manipulated. You need to do data munching so that you have the data in a certain proper format to be used by the model or our analytics system. And then you need to do the model planning. What kind of a model? what algorithm you will use for a given problem and then the model building. So the exact execution of that model happens in step four and you implement and execute that model and uh put the data through the analysis in this step and then you get the results. These results are then communicated packaged and presented and communicated to the stakeholders and once that is accepted that is operationalized. So that is the final step. Now in the end let's take a quick look at the demand for data scientists. Data science is an area of great demand. The demand for data scientists is currently huge and the supply is very low. So there is a huge gap. So what are some of the industries with high demand for data scientists? I think gaming is definitely one area where it's a industry which is consumerf facing industry and a lot of people play games and growing industry and it requires a lot of data science. So that is an area where data scientists are in demand. Then we have healthcare for example data science is used for diagnosis and several other activities within healthcare predicting for example a disease. So healthcare is definitely finance definitely banks insurance companies all of these there is a huge demand for data scientists marketing is like a horizontal functionality across all industries there's a demand for data scientists there then of course in technology area so pretty much all of these areas there is a lot of demand globally there is a huge demand so this is a very very critical skill that would be required currently as well as in the future so Let's summarize what we have seen so far. We talked about the need for data science, what data science can do and what is data science and what are the prerequisites of uh data science in terms of the skills and um programming languages and tools and so on and so forth. We also talked about the various tools that are available like Python and R and and we did a comparison also between business intelligence and data science. And we did a detailed discussion about the life cycle of a data science project with an example. And last but not least, we talked about the demand for data scientists, the global demand. There's a huge demand for data scientists. We talked about that as well. >> Picture this. You're shopping online and suddenly you see a product that feels like it was made just for you. How did they know? It's not by chance. It's data science. Data science help businesses understand what you like, predict what you'll need next, and improve the way we shop and use technology. And here's the best part. Data science isn't just about watching Netflix. It's one of the fastest growing careers in the world right now. In fact, the US Bureau of Labor Statistic says that data science jobs are expected to grow 36% by 2033, way faster than most of the other jobs. Companies everywhere are using data to make smarter decisions. That means the demand for data scientists is huge. And let's talk about the salary. You're probably wondering how much can I earn actually. Well, for entry- level position, data scientists in the US are earning around $152,000 per year right now. And by 2025, some can make as much as $230,000. And in India, starting salaries range from 50,000 rupees to 1 lakh per month. And experienced professionals can earn more than 5 lakh rupees per month. That's impressive, right? But the best part is as a data scientist, you won't just stop here. The skills you develop in this role like machine learning, data visualization and statistics are highly transferable and crucial for moving into AI roles. So whether it's becoming an AI engineer or an AI specialist, the foundation you build in data science will help you level up and pursue exciting hyping opportunities in the AI field. Now if you're thinking this sounds great, but where do I even start? Well, that's exactly what the professional certificate course in data science from IIT Kpur and Simply Learn is designed to do. It will get you started and make sure you're ready for this booming industry. In this 11 month live online interactive program, you will learn the skills you need to become a data science professional. No more theory, no more fluff. You get hands-on projects, life classes and mentorship from IT Kpool faculty plus real world industry expert who will help you build your skills. So this course comes with exciting amazing features that makes learning even more impactful. Eight times high interaction in live online classes with industry expert. Regular live online classes conducted by experienced professionals who bring real world knowledge into every session. You'll also get to master 13 plus key skills including generative AI, prompt engineering, chart, explanable AI, conversional AI, NLP, and many more. These are the skills that top companies use every day. And you'll gain hands-on experience with 14 plus industry tools like Python, SQL, Tableau, DL2, MidJourney, TensorFlow, and more. And by the end of this course, you will be ready to tackle real world challenges using these powerful tools and techniques. But we are not just talking about textbook and theory. You'll work on 25 plus real world projects giving you hands-on experience with the tools and skills you'll use in the industry. For example, our first project would be about sales analysis. You'll use Python to analyze a clothing company fourth quarter sales data across Australian state, helping the company make informed decisions. The next project would be about employee performance analysis where you learn how to build machine learning models to understand the factors influencing employees turnover. Our third project would be about e-commerce which will help Amazon improve its recommendation engine to offer better recommendation to customers. Upon successful completion, you'll receive a program certificate directly issued by the ENI city academy IT Kpool within 45 days of completing your cohort. This prestigious certificate will help you boost your resume and show potential employees that you have mastered the skills needed to succeed. You'll also benefit from master classes delivered by distinguished IT kur faculty who bring their deep expertise in this course. Plus, you will be exposed to trending tools like charge to geni and prompt engineering. Now, if you're wondering who will be teaching all of this, then the IT kle faculty is here for you. These experts have been in this field for years and have worked with the top companies. You'll also get master classes from them and they will guide you through the learning process. You're not just learning from a textbook. You are learning from the people who have been there and done that. And the best part is once you have learned the skills, Simply Learn's career assistance team will help you take to the next step which will help you to build a killer resume, show you how to stand out top recruiters, and even give you access to mock interviews. Plus, you will also gain access to exclusive networking events and hackathons to connect with industry professionals. Along with simple learners job assistant, you'll also get access to ID KPUR's career services helping you connect with top recruiters and land interviews with leading tech companies. So, upon finishing this course, you'll be ready for top data science roles like data scientist, machine learning engineer, AI specialist and business analyst. The good news is top companies like Amazon, EY, Fidelity Investment, Johnson and Johnson, Borafhone, Accenture, Infosys and Nvidia is looking for professionals just like you. And with salaries in the US hitting around $230,000 plus and in India reaching up to five lakh per month, your career outlook looks great. So what will you be actually learning in this course? So here's a sneak peek of the syllabus which you'll be learning in this course which is the foundation in Python, SQL and mathematics, core data science like machine learning, data visualization, NLP, special topics like GNEI, charge GBT, prompt engineering and you'll also work on industry projects which we have already mentioned before. You'll also have the option to choose electives like data storytelling with PowerBI and business analytics with Excel to tailor your learning experience and focus on the areas that interest you the most. So don't wait, hurry up and enroll now and find the course link in the description box below and in the pin comments. So thank you for watching. If you have any question, leave them in the comment section below. I would love to help. Don't forget to subscribe for more career transforming opportunities by Simply Learn. Now moving on to the third step that is data structures and algorithms. Learning data structures and algorithms is crucial for becoming a data scientist because they provide the foundation for efficient data handling and problem solving. Data structures like arrays, stacks, cues, and trees help you store and organize data in ways that make it easier and faster to access, process, and analyze. Algorithms, on the other hand, give you strategies to perform tasks like searching, sorting, and optimizing data operations, which are essential for handling large data sets. While many candidates struggle with the essay, mastering it gives you an age, helping you stand out in the interviews and shine as a skilled data scientist capable of tackling the toughest data problems. Spend about 2 months in this, you will get in the shape for sure. Now moving on to the step number four that is SQL. Learning SQL is essential for data scientists because it enables you to access, manage and manipulate data directly within databases where most real world data resides. With SQL, you can create new tables, alter existing ones, delete unnecessary records, and run queries to filter, sort, and aggregate data. These abilities allow you to retrieve, clean, and organize data effectively. Core skills needed for any data science role. It's easy, and you don't have to spend more than a month to have a deep understanding of it. Now, moving on to the fifth step that is mathematics and statistics. Mathematics and statistics are essential for data science because they form the backbone of data analysis, model building and interpretation. Topics like linear algebra, calculus, probability, and statistics gives data scientists the tools to understand data patterns, perform accurate analysis, and make datadriven decisions. Mastering these areas enables you to build robust models, validate results, and tackle complex problems confidently, making you a well-rounded and skilled data scientist. Make sure you spend two months to grabs this topics. Now moving on to the step number six that is data prep-processing and visualization. Learning data prep-processing and visualization is essential for a data scientist because these skills make you data accurate, insightful and easy to understand. Python libraries like NumPy and Panders are crucial for manipulating and creating data enabling you to handle missing values, filter out noise and prepare data for analysis. Once the data is ready, visualization lets you uncover patterns and communicate results effectively. Libraries like Mattplot tip and seaborn help create clear, impactful visuals, allowing you to interpret trends and convey insights in a way that's easily understood by others. Together with these tools make data prep-processing and visualization fundamentals for effective data science. If you have a solid foundation on Python and mathematics, you will get a good understanding of data prep-processing and visualization in a month or two. Now moving on to the seventh step that is machine learning fundamentals. Machine learning fundamentals involve understanding how algorithms enable computers to learn from data and make predictions on decisions without explicit programming. The two main categories are supervised learning and unsupervised learning. In supervised learning, models are trained on labelled data to make predictions. While in unsupervised learning, models find patterns in unlabelled data. Popular tools like TensorFlow, PyTorch help build and train complex models, especially for deep learning. While Skyit learn is essential used for simpler machine learning algorithms and data prep-processing. These tools make it easier to implement machine learning fundamentals effectively and build intelligent datadriven decisions. Dedicate about 3 months to understand the core of machine learning. Now coming to the next step that is deep learning. Deep learning is a subset of machine learning that focuses on algorithms inspired by the structures of the human brain called neural networks. Deep learning uses neural networks with multiple layers often dozens or hundreds to learn complex patterns from large data sets. Specialized types like convolutional neural networks that is CNN's are great for image processing while recurrent neural networks RNN are used for sequence data like text or time series. Essential tools like TensorFlow, PyTorch make building, training and deploying deep learning models more accessible, allowing you to create powerful AI solutions across various domains. I think it will take about 2 months to have a good hold on deep learning concepts and how to implement them. Now moving on to the ninth step that is specializations. Once you have grasped the deep learning, it's like reaching a new level as a data scientist. Just as doctors specialize in areas in nephrology and cardiology, data scientists often choose to specialize in fields like natural language processing or computer vision. Natural language processing focuses on teaching machines to understand and generate human language enabling applications like chatbot, sentiment analysis, and language transition. It's about making computers read, write, and even interpret human emotions through text or speech. Computer vision on the other hand is all about enabling machines to see and interpret images or videos. This field powers innovations like facial recognition, object detection and autonomous driving. Now you don't need to learn both. You can choose what interests you the most. Now spend one to two months diving deep into one of these areas. Now moving on to the last but not the least that is big data. Big data refers to extremely large volumes of data generated rapidly from sources like social media and sensors. For data scientists, learning to handle big data is crucial as it requires specialized tools like Hadoop and Spark to analyze and extract insights effectively. With companies relying on datadriven decisions, big data skills make you a highly in- demand professional in the field. Focus for about 2 months and you will be able to spot trends and patterns from data sets very easily. Once you're ready, it's time to build a killer resume packed with projects that showcase your new skills. Start applying to jobs on platforms like Noy and Indate and supercharge your LinkedIn. Connect with data scientists. See what skills they are mastering and learn from their journeys as well. Keep sharpening your own skills and when the time comes, you will be ready to crush those interviews and land your dream data scientist role in 2025. >> Welcome to math refresher probability and statistics. In this lesson, we are going to explain the concepts of statistics and probability. Describe conditional probability. Define the chain rule of probability. Discuss the measure of variance. Identify the types of gshian distribution. Basic of statistics and probability. Probability and statistics. Data science relies heavily on estimates and predictions. A significant portion of data science is made up of evaluations and forecast. Statistical methods are used to make estimates for further analysis. Probability theory is helpful for making predictions. Statistical methods are highly dependent on probability theory and all probability and statistics are dependent on data. Data is information acquired for reference or research via observations, facts, and measurements. Data is a set of facts structured in the form that computers can interpret such as numbers, words, estimations, and views. Importance of data. Data aids in seeing more about the information by identifying possible connections between two features. Data assists in the detection of distortion by uncovering hidden patterns based on prior information patterns. Data may be utilized to anticipate the future or predict the current state of affairs. Also, data aids in determining whether two pieces of information have any instance in common or not. Types of data. Data might be quantitative. That is data that can be measured or counted in numbers. Or it may be qualitative which is data which is generally divided into groups or in simpler words which cannot be counted or measured in numbers. Let's consider an example. A customer information data of a bank may contain quantitative and qualitative data. Consider this snapshot where we have customer ID, surname, geography, gender, age, balance, has C or card is active member. Amongst these variables we can see surname is mostly qualitative as it cannot be counted and measured in numbers. Geography and gender are also qualitative as they cannot be counted in numbers and are mostly groups. has C or card that is has credit card and is active member although are containing numerical in form but these are categorical that means these have been divided into groups of one and zero that represent yes and no as an answer hence these two variables are also qualitative customer ID is again although a numerical data however the significance or intuition behind Customer ID is categorical. Hence, it may be kept in the qualitative data also. However, age and balance these are numerical information which have been measured or counted and numerical operations can be performed on them. Hence, these are under quantitative data categories. Introduction to descriptive statistics. Descriptive statistics. A descriptive measurement is summary measure that quantitatively portrays the most important features of a set of data allowing for a better comprehension of the information. Data can be measured as different levels. The levels of measurement describe the nature of information stored in the data assigned to the variables. Qualitative data can be measured as nominal or ordinal. Quantitative data can be measured in terms of interval and ratio type. Nominal data. The data is categorized using names, labels or qualities. For example, brand name, zip code, and gender. Ordinal data can be arranged in order or ranked and can be compared. Examples include grades, star reviews, position, and race, and date. Interval data is the data that is ordered and has meaningful differences between the data points. Example, temperature in Celsius and year of birth. Ratio data is similar to the interval level with the added property of inherent zero. Mathematical calculations can be performed on both interval as well as ratio data. For example, height, age, and weight. Population versus sample. Before analyzing the data, it's important to figure out if it's from a population or a sample. Population is a collection of all available items as well as each unit in our study. Sample is a subset of the population that contains only a few units of the population. Population data is used for study when the data pool is very small and can give all the required information. Samples are collected randomly and represent the entire population in the best possible way. Measures of central tendency. The central tendency is a single value that aids in the description of the data by determining its center position. Measures of central tendency are sometimes known as summary statistics or measures of central location. The most popular measurements of central tendency are mean, median, and mode. The normal distribution is a bell-shaped symmetrical distribution in which mean, median, and mode all are equal. The curve over here shows the bell-shaped curve or the normal distribution of variable X. The point over here that is X1 is the point which represents the mean, median and mode of this distribution. Mean mean is calculated by dividing these sum of all data values by the total number of data values. It gets affected when there are unusual or extreme values. It is sensitive to the outliers. Mean can be calculated as summation over all the values of X in a collection divided by the size of the collection. For example, we have a collection where we have values as 7 3 4 1 6 and 7. We find out the sum of these values which is 28 and there are total of six values. So 28 / 6 gives us a mean value of 4.66. Median, it is the middle value in the set of the data that has been sorted in ascending order. It is a better alternative to mean since it is less impacted by outliers and skewess. It is closer to the actual central value. Median is calculated differently for different sizes of data. Differentiated as if the total number of values is odd or if the total number of values is even. If the size of the data is odd. For example, in this case we have five elements. After sorting whatever middle value we get that means n + 1 by 2 term in this case 5 + 1 / 2 that is the third term which is four is the median value. In case when the total number of values is even like here there are six values. The average or the mean of the two central values is considered as the median. In this case the median is the mean of six and four which is five. Mode. Mode represents the most common value in the data set. It is not at all affected by extreme observations. It is the best measure of central tendency for highly skewed or non-normal distribution. Mode for categorical data is determined by estimating the frequencies for each categories and then the category with the highest frequency is considered to be mode. Like in this case seven has the highest frequency. Hence seven becomes the mode value. However, in case of continuous data or quantitative data, the calculation of mode is slightly different. The first step in calculation of mode is dividing the data into classes which are equal with then getting the frequency of data points lying in within that range of classes and finally selecting the class with the highest frequency. Using the range of that class and the frequencies, we can get the final mode value. Using the formula L plus F minus F_sub_1 * H / F minus F_sub_1 plus FM minus F_sub_2. Here L is the lower limit or the lower observation of the mode class. H is the size of the mode class. FM is the frequency of the mode class. F_sub_1 is the frequency of the class proceeding to mode and F_sub_2 is the frequency of the class succeeding to mode. This gives us the final mode value. Mean versus expectation. Now let's talk about mean versus expectation. So in general we use the expected value or expectation when we want to calculate the mean of a probability distribution that represents the average value we expect to occur before collecting any data. And mean on the other hand mean is basically used when we want to calculate the average value of a given sample. This represents the average value of raw data that we may have already collected. We can understand this by using a simple example. Now to calculate the expected value of this probability distribution, we can use a specific formula from the previous discussion. This is going to be the expected value where X is going to be the data value and this PX is the probability of value. For example, we could calculate the expected value for this probability distribution to be as shown. So here it will be 1.45 goals. So this represents the expected number of goals that the team will score in any given game. And then if you talk about calculating mean, so we typically calculate the mean after we have actually collected raw data. For example, suppose we record the number of goals that a soccer team will score in 15 different games. Now to calculate the mean number of goals scored per game, we can use the following formula where sum of x is basically the sum of all the goals divided by n and the number of records or we can say the sample size. It is as shown on the screen. So this represents the mean number of goals scored per game by the team. Measures of asymmetry. The difference between the three distinct curves can be studied in this image. The central curve is the normal or no skewess curve. Here mean, median and mode all lie on the same point. This normal curve is symmetrical about its mean, median and mode. That means the left hand side of the curve is a mirror image of the right hand side of the curve. However, in case of negatively skewed data, the tail is elongated on the left hand side and the mean is smaller than the mode and the median values or is on the left hand side of the mode. Hence indicating that the outliers are in the negative direction. On the other hand, in case of positively skewed, the data is concentrated on the left hand side of the curve. While the tail is elongated or longer on the right hand side of the curve, the mean is greater than the mode and median or is on the right hand side of the mode and median indicating that the outliers are in the positive direction. Let's consider an example. The graph here shows the global income distribution for the year 2003 2013 and a projection for 2035. If we see the global income distribution statistics for 2003, it is highly right skewed. We can observe in the previous graph that in 2003 the mean of 3,451 was higher than the median of $1090. The global income is definitely not evenly distributed. The majority of people make less than $2,000 each year, while only a small percentage of the population earns more than $14,000. Measures of variability. Measures of variability. Dispersion. The measure of central tendencies provide a single value that addresses the full worth. However, the central tendency cannot depict the viewpoint entirely. The metric of dispersion helps us focus on the inconsistency in the data spread. Measures of dispersion describe the spread of the data. The range, intercortile range, standard deviation and variance are examples of dispersion measures. Range. The range of distribution is the difference between the largest and the smallest amount of data. The range, for example, does not include all of a series positive aspects. It concentrates on the most shocking aspects and ignores that aren't considered critical. For example, for a set 13, 33, 45, 67, 70, the range is 57. That is the maximum of this which is 70 minus the minimum over here which is 13. Variance. Variance is the average of all squared deviations. It is defined as the sum of squared distance between each point and the mean or the dispersion around the mean. The standard deviation is used as variance suffers from a unit difference. Variance can be computed as sigma square summation over x - mu^ 2 divided by n where mu is the mean of the data, x is the individual data point and n is the size of the data. This representation is for a population data. for a sample data variance can be computed as X minus Xar whole square summation over it divided by n minus one. Here Xar is the mean of these sample data and n is the sample size. The units of values and variance are not equal. So another variability measure is used. Standard deviation. Standard deviation is a statistical term used to measure the amount of variability or dispersion around a mean. The standard deviation is calculated as the square root of variance. It depicts the concentration of the data around the mean of the data set. Standard deviation as indicated previously can be computed as square root of variance. For a population data, standard deviation sigma can be computed as square root of summation over x i minus mu^ square / n where mu is the mean of the data. x i are the data points and n is the size. Let's consider an example. Let's find out the mean, variance, and standard deviation for this data. The data values are 3 5 6 9 and 10. To find out the mean, we first find the sum of all these data values that is 33 and divide it by the count which is five. We get the mean of 6.6. To compute the variance, we start by computing the deviation. That is X minus the mean of X. Here 3 is one of the values of the data and 6.6 is the mean. So 3 - 6.6 squared and we do that. To find out sum of all the deviations divided by the count which is five we end up getting an overall variance of 6.64. Standard deviation as we know is measured at square root of variance that is square<unk> of 6.64 which amounts to 2.576. Measures of relationship. Measures of relationship coariance. Coariance is the measure of joint variability of two variables. It measures the direction of the relationship between the variables. It determines if one variable will cause the other to alter in the same way. Coariance between variable x and y can be computed as summation over the product of x i - xar and y i - y bar the whole divided by n minus one. Here xar and y bar are the mean of x and y respectively. The value of covariance can range from minus infinity to a plus infinity. Correlation. Correlation is normalized coariance. It measures the strength of association between two variables. The most common measure for correlation is the Pearson correlation coefficient. Correlation between two variables X and Y can be measured with respect to coariance as coariance between X and Y divided by the standard deviation of X and standard deviation of Y. The value of correlation ranges from a negative 1 to positive 1. Types of correlation. Correlation can be either a positive correlation, zero correlation or a negative correlation. The first picture over here represents a perfect positive correlation wherein a straight line with a positive slope is representing the relationship between the two variables. Zero correlation means that the line representing the relationship between the two variables is horizontal to the xaxis. Perfect negative correlation can be represented by a straight line with a negative slope. Correlation equals to 1 implies a positive relationship. That is when one variable increases the other variable also increases. A correlation value of negative 1 implies a negative relationship. That is when one variable increases the other decreases. The correlation coefficient of zero shows that the variables are completely independent of each other. Let's consider an example. Here we have two variables height and weight. To compute the correlation between height and weight, we use the correlation formula as covariance of X and Y divided by standard deviation of X and standard deviation of Y. Here height is the X variable and weight is the Y variable. First to compute coariance we compute the x - xar and y - y bar values and then the product of them. We then compute x - xr² and y - y bar square values to compute the standard deviations of height and weight respectively. Correlation as we know has been defined as covariance of X and I and Y divided by standard deviations of X and Y. This can also be represented as summation over x - xr multiplied to y - y bar divided by square root of summation over sum of squared deviations that is x - xr square multiplied to square root of summation over y - yar whole square that is sum of square deviations for y. Now let's find out values to put into this formula. First we find out the overall sum of height to get the mean of height which is 5.14. Similarly we get the sum of weight to get the mean of weight as 50. We now get the summation over x - xr multiplied to y - y bar to get the numerator for the formula. Then we compute x - xr square summation and y - y bar square that is sum of squared deviation of x and y respectively. Now we put in the values in this final correlation formula to get a correlation value of 0.889. This indicates that height and weight have a positive relationship. It is evident that as height grows, weight also increases. In this module, we will be talking about expectation and variance. So the expected value or we can say mean of a given variable that we can denote by X is a discrete random variable where it is a weighted average of the possible values that X can take and each value is going to be according to the probability of that specific event occurring. So usually the expected value of X is denoted by a simple formula where we can define the expectation based on the X parameter. which is going to be the sum of each possible outcome multiplied by the probability of the outcome occurring. So in more concrete terms, the expectation is what we would expect the outcome of an experiment to be on average. We can take an example for the coin. If a coin is being tossed 10 times, then one is most likely to get five heads and five tails. Same logic can be discussed if we talk about another example of rolling a dieice. So there are six possible outcomes when you roll a dieice. 1 2 3 4 5 6. And each of these has a probability of 1x 6 of occurring. So we can say that the expectation is going to be 1 multiplied by the probability of that happening which is going to be 1x 6 + 2x 6 + 3x 6 + 4x 6 + 5x 6 + 6x 6 and that is going to give us 3.5 as an output. The expected value is 3.5. So if you think about it, 3.5 is halfway between the possible values that I can take and this is what we should have expected. Next we talk about the concept of variance. So variance of a random variable allows us to know something about the spread of the possible values of the variable. So for a discrete random variable X the variances of X is going to be denoted by using a simple formula that is going to be var X equals E X - M the whole square where M is basically the expected value of the expectation of X. So this is more like a standard deviation of X which can also be represented by using this formula. So the variance does not behave in the same way as expectation when we multiply and add constants to random variables. So now there are two different type of variance that we can have a fair understanding on. First of all we have low variance and then we have high variance. So low variance simply means that there is a small variation in the production of the target function with changes in the trading data set and at the same time high variance as we can see here high variance shows a large variation in prediction of the target function with changes in the trading data set. So a model that shows high variance learns a lot and perform well with the training data set and it does not generalize well with the unseen data set and that's why as a result such a model gives good results with training data set but shows high error rates on the test data set and since the high variance a model learns too much from the data set it leads to an overfitting of the model. So model with high variance will be having couple of issues like it may lead to overfitting or it may also lead to increase in model complexities. Next we have skewess. So skewess in simple terms is basically a measure of asymmetry of a distribution. So distribution is asymmetrical when its left and right sides are not the mirror images. Right now this is a mirrored image and a distribution can have right positive or we can say negative or it can have zero skewess. So right skewed in this scenario is basically the distribution is longer on the right side of its peak and a left skew distribution is going to be we can say where it is longer on the left side. So we can see we have this one as a part of right side. It is more elongated towards the right side and this one is more elongated towards the left side. So we can think of skewess in terms of tails. A tail is long tampering and the end of a distribution. So it simply indicates that they are observations at one end of the distribution but that they are relatively infrequent. So a right skew distribution has a long tail on the right side as you can see here. So the number supports observed. Let's say we have a data on a per year basis. So again we can have a more skewess towards the right side where data is being dropping as we continue to increase the number of years. For example we may have a high sales towards the beginning of year suppose in 2022 but again as we proceed to 2023 second half we are seeing the dip in performance. So that is rightly skewed and same way let's suppose if we started with the sales figure it was really less in suppose 2002 but again as we proceeded to 2023 now our sales have been gradually increasing. So it's more like skew towards the left section as a part of negative skew. Next we have curtosis. So curtosis is basically a measure of the tailness of a distribution. So tailness is how often the outliers occur and act as curtis is the tailness of the distribution related to a normal distribution. So a distribution with medium curttosis is called as messortic. A distribution with low kurtosis like this one. This is called as the platicurtic and then distribution with high curtosis like this one. This is called as the leptocuric. So tails here they are tapering ends on either side of a distribution like this. So they represent the probability or the frequency of values that are extremely high or extremely low to the mean. In other words, tails here represents how often the outliers occur. So there are three type of curtosis. We have platicurtic which is negative, leptocortic which is a positive towards the upper end and then we have messertic which is a normal distribution. So messertic is the medium tail. So normal distributions they have a curtosis of three. So any distribution with a curtis of approx value of three is going to be messertic and curtosis is described in terms of excess curtises which is curtosis minus 3 and since normal distribution they have a curtosis of three axis curtises makes comparing a distribution curtosis to a normal distribution even easier. Introduction to probability. Probability theory. Probability is a measure of the likelihood that an event will occur. Let's consider an example of coin toss where the chances of getting heads on a coin are 1 by two or 50%. The probability of each given event is between zero and one both inclusive. Sum of an events cumulative probability cannot be greater than one. Hence the probability of an event X lies between 0 and 1. This means that the integral of probability of distribution over X equals to 1. Conditional probability. Conditional probability of any event A is defined as the probability of occurrence of A given that event B has previously occurred. Condition probability of event A given B can be estimated as probability of A intersection B that is probability of both A and B happening together divided by the probability of B. It is also written as that probability of A intersection B equals to probability of A given B multiplied to probability of B. Let's consider an example. In a coin, we are doing a two coin flip. Coin one gets heads, tails, heads, and tails in subsequent flips. while coin two gets tails, heads, heads, and tails in the subsequent flips. Now, the probability that coin one will get a head is 2 out of four. While the probability that coin two will get heads is again two out of four. The probability that both coin one and coin two will have a heads is just one out of the four flips. Hence the probability that coin one will get heads given that coin 2 is already heads can be computed as probability of coin one edge intersection coin 2 edge that is 1x4 divided by probability of coin 2 edge that's a given that is 2x 4 which is going to be 0.5 or 50% based base theorem Base theorem calculates the conditional probability of an event based on its prior probabilities. Basically base theorem incorporates the prior probability distribution to predict the posterior probabilities. Base theorem for conditional probability can be expressed as probability of A given B equals probability of B given A divided by probability of B multiplied to probability of A. Base theorem allows updating the probability values by using new information or evidence. Here probability of A is known as prior probability. That is the probability of event before any new data is collected. Probability of A given B is known as the posterior probability. It is the revised probability of an event occurring after taking into consideration the new information probability of B given A is known as the likelihood and probability of B is probability of observing an evidence B model. An example consider an example for calculating the likelihood of having diabetes based on frequency of fast food consumption. Here is the observed data. Let's say the fast food audience is 20%. Diabetes prevalence is 10% and 5% is fast food and diabetes. The chances of diabetes given fast food that is the conditional probability of D given B can be calculated as probability of diabetes and fast food together divided by probability of fast food. That means 5% divided by 20%. that equals 25%. Define an analysis can state eating fast food increases the chance of having diabetes by 25%. The multiplication rule of probability if events A and B are statistically independent and probability of A intersection B can be given as probability of A given B multiplied to probability of B. However, probability of A intersection B is also given as probability of A multiplied to probability of B. Here probability of A given B equals to probability of A when we assume that probability of B is non zero. Similarly, probability of B equals probability of B given A assuming probability of A is non zero. Chain rule of probability joint probability distributions over many random variables can be reduced into conditional distributions over a single variable. It can be expressed as probability of X1 X2 so on until Xn equals probability of X1 intersection probability of X I given probability of X1 till X I minus one. For example, the joint probability of A, B and C can be given as probability of A given B. C multiplied to probability of B given C multiply to probability of C. Logistic sigmoid. The logistics function is a type of sigmoid function that aims to predict the class to which a particular sample belongs. Its outcome is discrete binary value. a probability between zero and one. The logistic sigmoid is a useful function that follows the yes curve. It saturates when the input is very large or very small. Logistic sigmoid is expressed as sigma of x= 1 upon 1 + e to the power min - x. The logistic sigmoid can be expressed as sigmoid function of x is given as 1 upon 1 + e ^ min - x where e is the ooler's number. Gshian distribution. The gossian distribution is a type of distribution in which data tends to cluster around a central value with little or no bias to the left or right. It is often referred to as normal distribution. In absence of prior information, the normal distribution is frequently a fair assumption in machine learning equation. The formula for calculating Gaussian distribution is described as the normal distribution of X. That is the function of x given mean as mu and variance is sigma square can be calculated as 1 upon sigma square roo<unk> of 2 pi e to the power -/ x - mood / sigma square where mu is the mean or peak value which also is the expected value of x. Sigma is the standard deviation. Sigma square is the variance. A standard normal distribution has a mean of zero and a standard deviation of one. Gshian distribution can be univariate which describes the distribution of a single variable X. It can also be multivariate where it can just use to describe the distribution of several variables. It is represented in 3D of ND formats. Law of large numbers. Now let's talk about law of large numbers. The law of large numbers states that an observed sample average from a large sample will be close to the true population average and that it will get closer in the larger sample. So the law of large number does not guarantee that a given sample spatially a small sample will reflect the true population characteristics or that a sample does not reflect the true population will be balanced by a subsequent sample. This is for the law of large numbers to express the relationship between scale and growth rate. So there are multiple examples through which we can understand and it is widely used in statistical analysis in working with the central limit theorem in terms of the business growth. So there are multiple real time setup in which these are going to be used. So if you talk about tossing a coin so tossing a coin in a number of times will give us two different type of outcomes. the result will spread evenly between head and tails and the expected average value is going to be half. That means 50 times tails and 30 times heads. But again, if you toss a coin 1,000 times, then the result can be in different manners because out of 1,000, let's say 850 times it has been head and only 150 times it has been tails and so on. So that's why the possibility of one event occurring is going to be changed in large sample sets as compared to a small sample sets as in let's say 10 times. So the number of heads and tails unbalanced for lower number of trials. So we can see it is unbalanced. But again as soon as we toss more number of coins more leans towards the balance value or we can see the observed averages. Next we have p value. So p value is basically a number calculated from the statistical test that describes how likely we are to have found a particular set of observations if the null hypothesis were true. So p values are used in hypothesis testing to help decide whether to reject the null hypothesis. And the smaller the p value, the more likely we are to reject the null hypothesis. So we have a term called as null hypothesis. So all statistical tests they have null hypothesis. So for most tests the null hypothesis is that there is no relationship between our variables of in first or that there is no difference among groups. For example in a two-tail t test the non-hypothesis is that the difference between two groups is going to be zero. So p value is going to tell us how likely it is that our data could have occurred under the null hypothesis. It is done by calculating the likelihood of a test statistic which is the number calculated by a statistical test using our data. So p value tell us how often we would expect to see a test statistic as extreme or more extreme than one calculated by a statistical test. if the null hypothesis of the test was true. So there are multiple limitations as well. So first one is the results can be significant but again they are they may not be practical as we have compared it can be based on multiple hypothesis for a game for the healthcare test. If the test is going to be positive or not it may show even values of the effect of a variable but not the magnitude in real life. What exactly is going to be the application of a drug test being failed in pharma company? Therefore, it is recommended to use confidence and levels in addition to the p values to quantify or we can say to give a solid figure to the reserve which we are going to get. The p values they are interpreted as supporting or we can say refuting the alternative hypothesis. So p value can only tell you whether or not the null hypothesis is supported. It cannot tell us whether our alternative hypothesis is true or why. So the risk of rejecting the null hypothesis is often higher than the p value. So especially when we are looking at a single study or when using small sample sizes. So this is because the smaller frame of reference, the greater are the chance that as we stumble across a statistically significant pattern completely by accident. Key takeaways. Key takeaways. Probability and statistics structure the premise of the data. The data helps in anticipating the future or gauging in view of the past patterns of information. The central tendency is a single value that helps to describe the data by identifying these central positions. The mean, median, and mode are the measures of central tendencies. The distribution where the data tends to be around a central value with a lack of bias or minimal bias towards the left or right is called as gshian distribution. >> Let's move into the different types of deep learning. Neural networks are the main component of deep learning. But neural networks comprise three main types which contain artificial neural networks or ANN, convolution neural networks or CNN and recurrent neural networks or RNN. Artificial neural networks are inspired biologically by the animal brain. Convolutional neural networks surpass other neural networks when given inputs such as images, voice or audio. It analyzes images by processing data. Recurrent neural networks uses sequential data or series of data. Convolutional neural networks and recurrent neural networks are used in natural language processes, speech recognition, image recognition, and many more. Machine learning. The evolution of ML started with the mathematical modeling of neural networks that served as the basis for the invention of machine learning. In 1943, neuroscientist Warren McCullik and logician Walter Pittz attempted to quantitatively map out how humans make decisions and carry out thinking processes. Therefore, the term machine learning is not new. Machine learning is a branch of artificial intelligence and computer science that uses data and algorithms to imitate how humans learn, gradually increasing the systems accuracy. There are three types of machine learning which include supervised learning. What is supervised learning? Well, here machines are trained using label data. Machines predict output based on this data. Now coming to unsupervised learning. Models are not supervised using a training data set. It is comparable to the learning process that occurs in the human brain while learning something new. And the third type of machine learning is reinforcement learning. Here the agent learns from feedback. It learns to behave in a given environment based on actions and the result of the action. This feature can be observed in robotics. Now coming to the evolution of AI, the potential of artificial intelligence wasn't explored until the 1950s. Although the idea has been known for centuries, the term artificial intelligence has been around for a decade. Still, it wasn't until British polymath Alan Turing posed the question of why machines couldn't use knowledge like humans do to solve problems and make decisions. We can define artificial intelligence as a technique of turning a computer-based robot to work and act like humans. Now, let's have a glance at the types of artificial intelligence. Weak AI performs only specific tasks like Apple's Siri, Google Assistant, and Amazon's Alexa. You might have used all of these technologies, but the types I am mentioning after this are under experiment. General AI can also be addressed as artificial general intelligence. It is equivalent to human intelligence. Hence, an AGI system is capable of carrying out any task that a human can. Strong AI aspires to build machines that are indistinguishable from the human mind. Both general and strong AI are hypothetical right now. Rigorous research is going on on this matter. There are many branches of artificial intelligence which include machine learning, deep learning, natural language processing, robotics, expert systems, fuzzy logic. Therefore, the correct answer for which is not a branch of artificial intelligence is option A, data analysis. Now that we have covered deep learning, machine learning and artificial intelligence, the final topic is data science. Concepts like deep learning, machine learning and artificial intelligence can be considered a subset of data science. Let us cover the evolution of data science. The phrase data science was coined in the early 1960s to characterize a new profession that would enable the comprehension and analysis of the massive volumes of data being gathered at the time. Since its beginnings, data science has expanded to incorporate ideas and methods from other fields, including artificial intelligence, machine learning, deep learning, and so forth. Data science can be defined as the domain of study that handles vast volumes of data using modern tools and techniques to find unseen patterns, derive meaningful information, and make business decisions. Therefore, data science comprises machine learning, artificial intelligence, and deep learning. There are a lot of areas where data science can be used. One of the very common one is fraud detection or fraud prevention. There are a lot of fraudulent activities or transactions primarily on the internet. It's very easy to commit fraud and therefore we can use data science to either prevent or detect fraud. There are certain algorithms, machine learning algorithms that can be used like for example some outlier techniques, clustering techniques that can be used to detect fraud and prevent fraud as well. So who is a data scientist rather? It is actually a very generic role that defines somebody who is working with data is known as a data scientist. But there can be very specific activities and the roles can be actually much more specific. What exactly a person does within the area of data science can be much more specific. But broadly anybody working in the area of data science is known as a data scientist. So what does a data scientist do? These are some of the activities. Data acquisition, data preparation, data mining, data modeling and then model maintenance. We will talk about each of these in a great detail but at a very high level the first step obviously is to get the raw data which is known as data acquisition. It can be all kinds of format and it could be multiple sources but obviously that raw data cannot be used as it is for performing data mining activities or data modeling activities. So the data has to be planned and prepared for using in the data models or in the data mining activity. So that is the data preparation. Then we actually do the data mining which can also include some exploratory activities. And then if we have to do stuff like machine learning then you need to build a machine learning model and test the model get insights out of it. And then if um the model is fine you deploy it and then you need to maintain the model because over a period of time it is possible that you need to tweak the model because of change in the process or change in the data and so on. So that all comes under the model maintenance. So let's take deeper look at each of these activities. Let's start with data acquisition. So the stage of data acquisition basically the data scientist will collect raw data from all possible sources. So this could be typically an RDBMS which is a relational database or it can also be a non RDBMS or could be flat files or unstructured data and so on. So we need to bring all that data from different sources. If required, we need to do some kind of homogeneous formatting so that it all fits into in a looks at least format from a format perspective it looks homogeneous. So that may be requiring some kind of transformation. Very often this is loaded into what is known as data warehouse. So this can also be sometimes referred to as ETL or extract, transform and load. So a data warehouse is like a common place where data from different sources is brought together. so that people can perform data science activities like reporting or data mining or statistical analysis and so on. So data from various sources is put in a centralized place which is known as a data warehouse. So that is also known as ETL and in order to do this there can be data scientists can take help of some ETL tools. There are some existing tools that a data scientist can take help of like for example data stage or talent or informatica. These are pretty good tools for performing these ETL activities and getting the data. The next stage now that you have the raw data into a data warehouse, you still probably are not in a position to straight away use this data for performing the data mining activities. So that is where data preparation comes into play and there are multiple reasons for that. One of them could be the data is dirty, there are some missing values and so on and so forth. So a lot of time is actually spent in this particular stage. So a data scientist spends a lot of time almost 60 to 70% of the time in this part of the project or the process which is data preparation. So there are again within this there can be multiple sub activities starting from let's say data cleaning you will probably have missing values the data there is some columns the values are missing or the values are incorrect uh there are null values and so on and so forth. So that is basically the data cleaning part of it. then you need to perform certain transformations like for example normalizing the data and so on right so or you could probably have to modify a categorical values into numerical values and so on and so forth. So these are transformational activities then we may have to handle outliers. So the data could be such that there are a few values which are way beyond the normal behavior of the data for whatever reason either people have keyed in wrong values or for some reason some of the values are completely out of range. So those are known as outliers. So there are certain ways of handling these outliers and detecting and handling these outliers. So this is a part of what is known as exploratory analysis. So you quickly explore the data to find out are there. So and you can use visual tools like plots and identify what are the outliers and see how we can get rid of the outliers and so on. Then the next part could be data integrity. Data integrity is to validate for example if there are some primary keys that all the primary keys are populated. there are some foreign keys then at least most of the foreign keys should be populated and otherwise when we are trying to query the data you may get wrong values and so on. So that is the data integrity part of it and then we have what is known as data reduction. Sometimes we may have duplicate values we may have columns that may be duplicated because they're coming from different sources. The same values are there and so on. So a lot of this can be done using what is known as data reduction and thereby you can reduce the size of the data drastically because very often this could be written in data which can be removed and so on. So let's take a look at what are the various techniques that are used for data cleaning. So we need to ensure that the data is valid and it is consistent and uniform and accurate. So these are the various parameters that we need to ensure as a part of the data cleaning process. Now what are the techniques that that are used for data cleaning or uh so we will see what each of these are in this particular case and uh so what is the data set that we have we have data about a bank and its customer details. So let's take an example and see how we go about cleaning the data. And in this particular example we're assuming we are using Python. So let's assume we loaded this data which is the raw file CSV. This is how the customer data looks like and um we will see for example we take a closer look at the geography column we will see that there are quite a few blank spaces. So how do we go about when we have some blank spaces or if it is a string value then we put a empty string here or we just use a space or empty string. If they are numerical values then we need to come up with a strategy. Uh for example we put the mean value. So wherever it is missing we find the mean for that particular column. So in this case let's assume we have credit score and we see that quite a few of these values are missing. So what do we do here? We find the mean for this column for all the existing values and we found that the mean is equal to 638.6. six. So we kind of write a piece of code to replace wherever there are blank values. Nan is basically like null and uh we just go ahead and say fill it with the mean value. So this is a piece of code we are writing to fill it. So all the blanks or all the null values get replaced with the mean value. Now one of the reasons for doing this is that very often if you have some such situation many of your statistical functions may not even work. So that's the reason you need to fill up these values or either get rid of these records or fill up these values with something meaningful. So this is one mechanism which is basically using a mean. There are a few others as we move forward. We can see what are the other ways. For example, we can also say that any missing value in a particular row if even one column the value is missing you just drop that particular row or delete all rows where even a single column has missing values. So that is one way of dealing. Now the problem here can be that if a lot of data has let's say one or two columns missing and uh we drop many such rows then overall you may lose out on let's say 60% of the data has some value or the other missing 60% of the rows then it may not be a good idea to delete all the rows like in that manner because then you're losing pretty much 60% of your data. Therefore your analysis won't be accurate. But if it is only five or 10% then this will work. Another way is only to drop values where or rather drop rows where all the columns are empty which makes sense because that means that record is of really no use because it has no information in it. So there can be some situations like that. So we can provide a condition saying that drop the records where all the columns are blank or not applicable. We can also specify some kind of a threshold. Let's say you have 10 or 20 columns in a row. You can specify that maybe five columns are blank or null then you drop that record. So again we need to take care that such a condition such a situation the amount of data that has been removed or excluded is not large. If it is like maybe 5% maximum 10% then it's okay. But by doing this if you're losing out on a large chunk of data then it may not be a good idea. You need to come up with something better. What else we need to do next is so the data preparation part is done. So now we get into the data mining part. So what exactly we do in data mining? Primarily we come up with ways to take meaningful decisions. So data mining will give us insights into the data what is existing there and then we can do additional stuff like maybe machine learning and so on to get perform advanced analytics and so on. So the one of the first steps we do is what is known as data discovery and uh which is basically like exploratory analysis. So we can use tools like Tableau for doing some of this. So let's just take a quick look at how we go about that. So Tableau is excellent data mining or actually more of a reporting or a BI tool and you can download a trial version of Tableau at tableau.com or there is also Tableau public which is free and you can actually use and play around. However, if you want to use it for enterprise purpose then is a commercial software. So you need to purchase license and you can then run some of the data mining activities. Say your data source your data is in some Excel sheet. So you can select the source as Microsoft Excel or any other format and the data will be brought into the Tableau environment and then it will show you what is known as dimensions and uh measures. So dimensions are all the descriptive columns. So and Tableau is intelligent enough to actually identify these dimensions and measures. So measures are the numerical values. So as you can see here uh customer ID, gender, geography these are all dimensions non- numerical values whereas age, balance, credit score and so on are numeric values. So they come under measures. So you've got your data into Tableau and then you want to let's say build a small model and you want to let's say solve a particular problem. So what is the problem statement? All right, let's say we want to analyze why customers are leaving the bank which is known as uh exit and we want to analyze and see if what are some of the factors for exiting the bank and we want to let's assume consider these uh three of them like let's say gender, credit card and geography these as a criteria and analyze if these are in any way impacting or have some bearing on the customer exiting or the customer exit behavior. Okay. So let's um use Tableau and very quickly we will be able to find out how these uh parameters are affecting. All right. So let's see. So this is our customer data. So from our Excel sheet we have data set about let's say 10,000 rows and we want to find out what is the criteria. Let's start with gender. Let's say we want to first use gender as a criteria. So Tableau really offers an easy drag and drop kind of a mechanism. So that makes it really really easy to perform this kind of analysis. So what we need to do is exited says whether the customer has exited or not. So it has a value of zero and one and then of course you have gender and so on. So we will take these two and simply drag and drop. Okay. So exited and then we will put gender. And if we drag and drop into the analysis side of of Tableau. All right. So here what we are doing is we are showing male female as two different columns here and zero for people who did not exit and one for people who exited and that is colorcoded. So the blue color means people who did not exit and uh this yellow color means people who did exit. All right. So now if we pull the data here create like bar graphs this is how it would look. Uh so what is yellow? Let's go back. So yellow is uh who exited and uh for the male only 16.45% have exited and we can also draw a reference line that will help us or even provide aliases. So these are a lot of fancy stuff that is um provided by Tableau. You can create aliases and so that it looks good rather than basic labels and you can also add a reference line. So you add a reference line something like this. From here we can make out that on an average female customers exit more than the male customers. Right? So that is what we are seeing here on an average. So we have analyzed based on gender. We do see that there is some difference in the male and female behavior. Now let's take the next criteria which is the credit card. So let's see if having a credit card has any impact on the customer exit behavior. So just like before we drag and drop the credit card has credit card column if we drag and drop here and then we will see that there is pretty much no difference between people having credit card and not having credit card. 20.81% of people who have no credit card have exited and similarly 20.18% of people who have credit card have also exited. So the credit card is not having much of an impact. That's what this piece of analysis shows. Last we will basically go and check how the geography is impacting. So once again we can drag and drop geography column onto this side. And uh if we see here there are geographies like I think there are about three geographies like France, Germany and uh Spain. And um we see that there is some kind of a impact with the geography as well. Okay. So what we derive from this is that the credit card is really we can ignore the credit card variable or feature from our analysis because that doesn't have any impact but gender and geography we can keep and do further analysis. Okay. All right. So what are some of the advantages of data mining? Bit more detailed analysis can help us in predicting the future trends and it also helps in identifying customer behavior patterns. Okay. So you can take informed decisions because the data is telling you or providing you with some insights and then you take a decision based on that. If there is any fraudulent activity, data mining will help in quickly identifying such a fraud as well and of course it will also help us in identifying the right algorithm for performing more advanced data mining activities like machine learning and so on. All right. So the next activity now that we have the data we have prepared the data and perform some data mining activity the next step is model building. Let's take a look at model building. So what is model building? If we want to perform a more detailed data mining activity like maybe perform some machine learning then you need to build a model. And how do you build a model? First thing is you need to select which algorithm you want to use to solve the problem on hand and also what kind of data that is available and so on and so forth. So you need to make a a choice of the algorithm and based on that you go ahead and create a model train the model and so on. Now machine learning is kind of at a very high level classified into supervised and unsupervised. So if we want to predict a continuous value could be a price or a temperature or or a height or a length or things like that. So those are continuous values and if you want to find some of those then you use techniques like regression, linear regression, simple linear regression, multiple linear regression and so on. So these are the algorithms. On the other hand, there will be situations or there may be situations where you need to perform unsupervised learning. In case of unsupervised learning, you don't have any historical labeled data so to learn from. So that is when you use unsupervised learning. And uh some of the algorithms in unsupervised learning are clustering. K means clustering is the most common algorithm used in unsupervised learning. And similarly in supervised learning if you want to perform some activity on categorical values like for example it is not measured but it is counted like you want to classify whether this image is a cat or a dog whether you want to classify whether this customer will buy the product or not or you want to classify whether this email is spam or not spam. So these are examples of categorical values and uh these are examples of classification. Then you have algorithms like logistic regression, K nearest neighbor or KN&N and support vector machine. So these are some of the algorithms that are used in this case. And similarly in case of unsupervised learning if you need to perform on categorical values you have some algorithms like association analysis and hidden marco model. Okay. So in order to understand this better, let's take uh an example and uh take you through the whole process and then we will also see how the code can be written to perform this. Now let's take our example here where we want to perform a supervised learning which is basically we want to do a multilinear regression which means there are multiple independent variables and then we want to perform a linear regression to predict certain value. So in this particular example we have world happiness data. So this is a data about the happiness quotient of people from various countries and we are trying to predict and see whether our how our model will perform. So what is the question that we need to ask? First of all how to describe the data and then can we make a predictive model to calculate the happiness score. Right? So based on this we can then decide on what algorithm to use and what model to use and so on. So variables that are available or used in this model. This is a list of variables that are available. There is a happiness rank. I'll load the data and or I'll show you the data in a little bit so it becomes clear what are these. So there is what is known as happiness rank. Happiness score which is happiness score is more like a absolute value whereas rank is what is the ranking and then which country we are talking about and within that country which region and what kind of economy and whether the family which family and health details and freedom trust generosity and so on and so forth. So there are multiple variables that are available to us and uh the specific details probably are not required and there can be um in another example the variables can be completely different. So we don't have to go into the details of what exactly these variables are but it's just enough to understand that we have a bunch of these variables and now we need to use either all or some of these variables and then which we also sometimes refer to as features and then we need to build our model and train our model. All right. So let's assume we will use Python in order to perform this analysis or perform this machine learning activity and I will actually show you in our lab in in a little bit this whole thing we will run the live code but quickly I will run you through the slides and then we will go into the lab. So what are we doing here? First thing we need to do is import a bunch of libraries in Python which are required to perform our analysis. Most of these are for manipulating the data, the preparing the data and then scikitlearn or skarn is the library which we will use actually for this particular machine learning activity which is linear regression. So we have numpy, we have pandas and so on and so forth. All these libraries are imported and then we load our data and the data is in the form of a CSV file and there are different files for each year. So we have data for 2015, 16 and 17. And uh so we will load this data and then combine them, concatenate them to prepare a single data frame. And uh here we are making an assumption that you are familiar with Python. So it becomes easier if you are familiar with Python programming language or at least some programming language so that you can at least understand by looking at the code. So we are reading the file each of these files for each year and this is basically we are creating a a list of all the names of the columns we will be using later on you will see in the code. So we have loaded 2015 then 2016 and then also 2017. So we have created um three data frames and then we concatenate all these three data frames. This is what we are doing here. Then we identify which of these columns are required. Which for example some of the categorical values do we really need? We probably don't. Then we drop those columns so that we don't unnecessarily use all the columns and make the computation complicated. We can then create some plots using plotly library and it has some powerful features including creation or creation of maps and so on. just to understand the pattern the happiness quotient or how the happiness is across all the countries. So it's a nice visualization we can see each of these countries how they are in terms of their happiness score. This is the legend here. So the lighter colored countries have lower ranking and so these are the lower ranking ones and these are higher ranking which means that the ones with these dark colors are the happiest ones. So as you can see here Australia and maybe this side uh US and so on are the happiest ones. Okay. The other thing that we need to do is the correlation between the happiness score and happiness rank. We can find a correlation using a scatter plot and we find that yes they are kind of inversely proportion which is obvious. So if the score is high, happiness score is high then they are ranked number one. For example, highest is scored as number one. So that's the idea be behind this. So the happiness score given here and the happiness rank is actually given here. So they are inversely proportional because the higher the score the the absolute value of the rank will be lower. Right? So number one has the highest value of the score and so on. So they are inversely correlated but there is a strong what this graph shows is that there is a strong correlation between happiness rank and happiness score. And then we do some more plots to visualize this. we determined that probably rank and score are pretty much conveying the same message. So we don't need both of them. So we will kind of drop one of them and uh that is what we are doing here. So we drop the happiness rank and similarly. So this is one example of how we can remove some columns which are not adding value. So we will see in the code as well how that works. Moving on, this is a correlation between pretty much each of the columns with the other columns. So this is a correlation you can plot using plot function and uh we will see here that for example happiness score and happiness score are correlated strongest correlation right because every variable will be highly correlated to itself. So that's the reason so the darker the color is the higher the correlation and as so the and correlation in numerical terms goes from 0 to one. So one is the highest value and it can only be between 0 and one. Correlation between two variables can be only have a value between 0 and one. So the numerical value can go from 0 to one and one here is dark color and zero is kind of dark but it is blue color. From red it goes down the dark blue color indicates pretty much no correlation. So the from this heat map we see that happiness and economy and family are probably also health probably are the most correlated and then it keeps decreasing after freedom kind of keeps decreasing and coming to pretty much uh zero. All right. So that is a correlation graph and then we can probably use this to find out which are the columns that need to be dropped which do not have very high correlation and uh we take only those columns that we will need. So this is the code for dropping some of the columns. Once we have prepared the data when we have the required columns then we use scikitlearn to actually split the data. First of all, this is a normal machine learning process. You need to split the data into training and test data set. In this case, we are splitting into 80/20. So 80 is the training data set and 20 is the test data set. So that's what we are doing here. So we use train test split method or function. So you have all your training data in X_rain, the labels in Y train. Similarly, x test has the test data the inputs whereas the labels are in y test. So that's how and this value whether it is 8020 or 50/50 that is all individual preference. So in our case we are using 8020. All right. And uh then the next is to create a linear regression instance. So this is what we are doing. We're creating an instance of linear regression and then we train the model using the fit function and uh we are passing x and y which is the x value and the label data regular input and the label data label information. Then we do the test we run the or we perform the evaluation on the test data set. So this is what we are doing with the test data set and then we will evaluate how accurate the model is and using the scikit land functionality itself. We can also see what are the various parameters and what are the various coefficients because in linear regression you will get like a equation of like a straight line y is equal to beta 0 plus beta 1 x1 plus beta 2 x2 those beta 1 beta 2 beta 3 are known as the coefficients and beta 0 is the intercept. After the training you can actually get these information of the model what is the intercept value what are the coefficients and so on by using these uh functions. So let's take quickly go into the lab and take a look at our code. Okay. So this is my lab. This is my Jupyter notebook where the code I have the actual code and I will take you through this code to run this linear regression on the world happiness data. So we will import a bunch of libraries numpy pandas plot plotly and so on also. So yeah, scikit learn that's also very important. So that's the first step. Then I will import my data and uh the data is in three parts. There are three files, one for each year 2015, 2016 and 2017. And it is a CSV file. So I've imported my data. Let's take a look at the data. Quickly glance at data. So this is how it looks. We have the country, region, happiness rank, and then happiness score. there are some standard errors and then what is the per capita family and so on. So and then we will keep going. We will create a list of all these column names we will be using later. So for now just we I will run this code. No need of major explanation at this point. We know that some of these columns probably are not required. So you can use this drop functionality to remove some of the columns which we don't need like for example region and standard error will not be contributing to our model. So we will basically drop those values out here. So we use the drop and then we created a vector with these names column names that's what we are passing here. Instead of giving the names of the columns here we can pass a vector. So that's what we are doing. So this will drop from our data frame. It will remove region and standard error these two columns. Then the next step we will read the data for 2016 and also 2017 and then we will concatenate this data. So let's do that. So we have now data frame called happiness which is a concatenation of both all the three files. Let's take a quick look at the data now. So most of the unwanted columns have been removed and you have all the data in one place for all the three years. And this is how the data looks. And if you want to take a a look at the summary of the columns, you can say describe and uh you will get this information. For example, for each of the columns, what is the count? What's what is the mean value? Standard deviation, especially the numeric values, okay, not the categorical values. So this is a quick way to see how the data is and uh initial little bit of exploratory analysis can be done here. So what is the maximum value? What's the minimum value and so on for each of the columns. All right. So then we go ahead and create some visualizations using plotly. So let us go and build a plot. So if we see here now this is the relation correlation between happiness rank and happiness score. This is what we have seen in the slides as well. We can see that there is a tight correlation between them. Only thing is it is inverse correlation but otherwise they are very tightly correlated which also says that they both probably provide the same information. So there is no not much of value add. So we'll go ahead and drop the happiness rank as well from our columns. So that's what we're doing here. And now we can do the creation of the correlation heat map. Let us plot the correlation heat map to see how each of these columns is correlated to the others and we as we have seen in the slides. This is how it looks. So happiness score is very highly correlated. So this is the legend we have seen in the slide as well. So blue color indicates pretty much zero or very low correlation. Deep red color indicates very high correlation. And the value correlation is a numeric value and the value goes from 0 to one. If the two items or two features or columns are highly correlated then there will be as close to one as possible and two columns that are not at all correlated will be as close to zero as possible. So that's how it is. For example here happiness score and happiness score every column or every feature will be highly correlated to itself. So it is like between them there will be correlation value will be one. So that's why we see deep red color. But then others are for example with higher values are economy and then health and then maybe family and freedom. So these are generosity and trust are not very highly correlated to happiness score. So that is uh one quick exploratory analysis we can do and uh therefore we can drop the country and the happiness rank because they also again don't have any major impact on the analysis on our analysis. So now we have prepared our data. There was no need to clean the data because the data was clean. But if there were some missing values and so on as we have discussed in the slides, we would have had to perform some of the data cleaning activities as well. But in this case, the data was clean. All we needed to do was just the preparation part. So we removed some unwanted columns and we did some exploratory data analysis. Now we are ready to perform the machine learning activity. So we use scikitlearn for doing the machine learning. Scikitlearn is Python library that is available for performing our uh machine learning. Once again we will import some of these libraries like pandas and numpy and also scikitlearn. First step we will do is split the data in 2080 format. So you have all the test data which is 20% of the data is test data and 80% is your training data. So this test size indicates how much of it is in the what is the size of the test data. The remaining which is here we are saying 02 therefore that means training is8. So training data is 80%. All right. So we have executed that split the data and now we create an instance of the linear regression model. So lm is our linear regression model and we pass x and y the training data set and call the function fit so that the model gets trained. So now once that is done training is done, training is completed and now what we have to do is we need to predict the values for the test data. So the next step is using so you see here fit will basically run the training method. Predict will actually predict the values. So we are passing the input values which is the independent variables and we are asking for the values of the dependent variable which is which we are capturing in y prime and we use the predict method here lm.predict. So this will give us all the predicted y values and remember we already have y test has the actual values which are the labels so that we can use these two to compare and find out how much of it is error. So that's what we are doing here. We are trying to find the difference between the predicted value and the actual value. Y test is the actual value for the test data and Y predict is the predicted value. We just found out the predicted value. So we will run that and we can do a quick check as to how the data looks how is the difference. So in some cases it is positive, some cases it is negative but in most of the cases I think the difference is very small. This is exponential to the power of 0 - 04 and so on. So looks like our model has performed reasonably well. We can now check some of the parameters of our model like the intercept and the coefficients. So that's what we are doing here. So these are the coefficients of the various parameters that we or the coefficients of the various independent variables. Okay. So these are the values. Then we can quickly go ahead and list them down as well against the corresponding independent variables. So the coefficients against the corresponding independent variables. So 1.0051 051 is the coefficient for economy. N9983 is for family, coefficient for family and health and so on and so forth. Right? So that's what this is showing. Now we can use the functionality readily available functionality of scikitlearn and then plot that to find some of the parameters which determine the accuracy of this model like for example what is the mean square error and so on. So that's what we are doing here. So let's just go ahead and run this. So you can see here that the root mean square error is pretty low which is a good sign and uh which is a one of the measures of u how well our model is performing. We can do one more quick plot to just see how the actual values and the predicted values are looking. And once again you can see that as we have seen from the root mean square error root mean square error is very very low. That means that the actual values and the predicted values are pretty much matching up almost matching up. And this plot also shows the same. So this line is going through the predicted values and the actual values and the difference is very very low. So again this is actual data. This is one example where the the accuracy is high and the predicted values are pretty much matching with the actual values. But in real life you may find that these values are slightly more scattered and you may get the error value can be relatively on the higher side. The root mean square error. Okay. So this was a good and quick example of uh the code to perform data science activity or a machine learning or data mining activity. In this case we did what is known as linear regression. So let's go back to our slides and see what else is there. So we saw this these are the coefficients of each of the features in our code and uh we have seen the root mean square error as well and uh with we can take a few hundred countries certain values and actually predict to see if how the model is performing and I think we have done this as well and in this case as we have seen the pretty much the predicted values and the actual values are pretty much matching which means our model is almost 100% accurate as I mentioned it real life it may not be the case but in this particular case we have got a pretty good model which is very good also subsequently we can assume that this is how the equation in linear regression the model is nothing but an equation like y is equal to beta 0 plus beta 1 x1 plus beta 2 x2 plus beta 3 x3 and so on. So this is what we are showing here. So this is our intercept which is beta 0 and then we have beta 1 into economy value, beta 2 into the family value, beta 3 into health value and so on. So that is what is shown here. Okay. So I think the next step once we have the results from the data mining or machine learning activity, the next step is to communicate these results to the appropriate stakeholders. So that is what we will see here now. So how do we communicate? Usually you take these results and then either prepare a presentation or put it in a document and then show them these actionable results orable insights and uh you need to find out who are your target audience and uh put all the results in context and uh maybe if there was a problem statement you need to put this results in the context of the problem statement. what was our initial goal that we wanted to achieve. So that we need to communicate here based on you remember we started off with what is the question and what is the data and so on and then what is the answer. So we we need to put the results and then what is the methodology that we have used all that has to be put and clearly communicated in business terms so that the people understand very well from a business perspective. So once the model building is done, once the results are published and communicated, the last part is maintenance of this model. Now very often what can happen is the model may have to be subsequently updated or modified because of multiple reasons. Either the the data has changed, the way the data comes has changed or the process has changed or for whatever reason the accuracy may keep changing. Once you have trained the model the for example we got a very high accuracy but then over a period of time there can be various factors which can cause that. So from time to time we need to check whether the model is performing well or not. The accuracy needs to be tested once in a while and if required you may have to rebuild or retrain the model. So you do the assessment, you you see if it needs any tweaks or changes and then if it is required you need to probably retrain the model with the latest data that you have and then you deploy it. You build the model, train it and then you deploy it. So that is like the maintenance cycle that you may have to take the model. Data analyst versus data engineer versus data scientist. Which one to choose? This is one of the most popular questions asked by learners looking for a career in data and analytics. I'm sure you too would have come across these job roles in the ever growing data science landscape. Though they all deal with data, these jobs are not the same. There are significant differences between what a data analyst, data engineer, and a data scientist does. We will look at these job roles and the differences in detail. First, let's look at some data analytics and data science trends. The analytics and data science market is thriving. Data analytics, data engineering, and data science are the key trends in today's exhilarating market. As per statist.com, the global big data analytics market revenue will grow at a caggr of 30% with revenue reaching over 68 billion US by 2025. According to Technavio, the enterprise data management market is expected to increase by 64.08 billion US by 2025 as per markets and markets.com. The big data market size is projected to grow from 162.6 billion US in 2021 to $273.4 billion US in 2026. Now another report from research drive says that the data science platform market is estimated to reach 224.3 billion US by 2026. So with so much data available and companies making huge investments to drive business insights the job opportunities for data analysts, data engineers and data scientists are going to increase in 2022 and over the coming years. Now let's learn the major differences between data analyst versus data engineer versus data scientist. So who are they? A data analyst analyzes and interprets vast volumes of data in order to extract meaningful information out of it. They find solutions to a business problem and make critical business decisions. The insights provided by data analysts are important to companies that want to understand the needs of their end customers. But talking about who a data engineer is, a data engineer on the other hand builds infrastructure and scalable pipelines to manage the flow of data and prepare it for analysis. So basically they optimize the systems that enable data analysts and data scientists to perform their job efficiently. Data scientists are professionals who analyze and visualize existing data and use algorithms to build predictive models for making future decisions. They also engage with business leaders to understand their needs and present complex findings. With that, let's look at the primary roles and responsibilities of these three job roles. Data analysts are responsible to collect, clean, store and process data. They discover hidden patterns from data by performing exploratory data analysis and visualize data by creating charts and graphs. Acquiring data from primary and secondary sources is one of their key tasks. They build reports and dashboards and also maintain databases. Now talking about the roles and responsibilities of a data engineer. A data engineer performs data acquisition, the design, build and test data as well to develop and maintain data architecture. Data engineers are tasked with testing, integrating, managing and optimizing data from a variety of sources. So they integrate data into existing data pipelines, prepare data for modeling and perform various ETL operations. Now talking about the roles and responsibilities of a data scientist. So data scientists develop machine learning models to identify trends in data for making decisions. They develop hypothesis and use the knowledge of statistics, data visualization and machine learning to forecast the future for the business. Data scientists visualize data and use storytelling techniques and also write programs to automate data collection and processing. Now move on to the skills possessed by data analysts, data engineers and data scientists. To become a data analyst, you need to have good hands-on experience with writing SQL queries. You should have excellent Microsoft Excel skills for analyzing data. Data analysts are also good at programming and they need to know how to visualize data, solve business problems, and possess domain knowledge. Data engineers should have a solid understanding of SQL, MongoDB, and programming. They need to have a good command of data architecture, scripting, data warehousing and ETL. Data engineers are also good at Hadoop based analytics. Now talking about the skills for a data scientist. So a data scientist should have experience with programming in Python and R. They should have a very good understanding of mathematics and statistics as well. Data scientists need to possess analytical thinking and data visualization skills as well. Machine learning, deep learning, and decision-m are other critical skills every data scientist should have. Now we look at the salaries of a data scientist, a data analyst as well as a data engineer. So a data analyst in the United States earns over $70,000 peranom while in India a data analyst can earn nearly 7 lak 25,000 rupees peranom. A data engineer in the United States can earn over $112,500 per year and in India you can earn over 9 lakh rupees peranom. Talking about the salary of a data scientist, a data scientist in the United States earns over $117,000 peranom and in India a data scientist can earn over 11 lakh rupees peranom. Coming to the final section of this video, we'll look at the top companies hiring for data analysts, data engineers and data scientists. So we have the first company as Google. Then we have Tesla. Next we have the e-commerce giant Amazon. The internet giant Facebook or the social media giant Facebook. We have the tech giant Oracle. We also have Verizon and Airbnb. So these are some of the top companies that hire for the three roles. If you are now let's talk about the life cycle of a data science project. Okay. The first step is the concept study. In this step it involves understanding the business problem. Asking questions. Get a good understanding of the business model. Meet up with all the stakeholders. Understand what kind of data is available and all that is a part of the first step. So here are a few examples. We want to see what are the various specifications and then what is the end goal? What is the budget? Is there an example of this kind of a problem that has been maybe solved earlier. So all this is a part of the concept study and another example could be a very specific one to predict the price of a 1.35 karat diamond. And there may be relevant information inputs that are available and we want to predict the price. The next step in this process is data preparation. Data gathering and data preparation also known as data munching or sometimes it is also known as data manipulation. So what happens here is the raw data that is available may not be usable in its current format for various reasons. So that is why in this step a data scientist would explore the data. He will take a look at some sample data. Maybe pick there are millions of records. Pick a few thousand records and see how the data is looking. Are there any gaps? Is the structure appropriate to be fed into the system. Are there some columns which are probably not adding value, may not be required for the analysis. Very often these are like names of the customers. they will probably not add any value or much value from an analysis perspective. The structure of the data maybe the data is coming from multiple data sources and the structures may not be matching. What are the other problems? There may be gaps in the data. So the data all the columns all the cells are not filled. If you're talking about structured data there are several blank records or blank columns. So if you use that data directly, you'll get errors or you'll get inaccurate results. So how do you either get rid of that data or how do you fill this gaps with something meaningful. So all that is a part of data munching or data manipulation. So these are some additional subtopics within that. So data integration is one of them. If there are any conflicts in the data, there may be data may be redundant. Yeah, data resident redundancy is another issue. There may be you have let's say data coming from two different systems and both of them have customer table for example or customer information. So when you merge them there is a duplication issue. So how do we resolve that? So that is one data transformation. As I said there will be situations where data is coming from multiple sources and then when we merge them together they may not be matching. So we need to do some transformations to make sure everything is similar. We may have to do some data reduction. If the data size is too big, you may have to come up with ways to reduce it meaningfully without losing information. Then data cleaning. So there will be either wrong values or you null values or there are missing values. So how do you handle all of that? A few examples of very specific stuff. So there are missing values. How do you handle missing values or null values? Here in this particular slide, we are seeing three types of issues. One is missing value. Then you have null value. You see the difference between the two, right? So in the missing value, there is nothing blank. Null value, it says null. Now the system cannot handle if there are null values. Similarly, there is improper data. So it's supposed to be numeric value, but there is a string or a non-numeric value. So how do we clean and prepare the data so that our system can work flawlessly. So there are multiple ways and and there is no one common way of doing this. It can vary from project to project. It can vary from what exactly is the problem we're trying to solve. It can vary from data scientist to data scientist, organization to organization. So these are like some standard practices people come up with and and of course there will be a lot of trial and error. somebody would have tried out something and it worked and it'll continue to use that mechanism. So that's how we need to take care of data cleaning. Now what are the various ways of doing you know if if values are missing how do you take care of that? Now if the data is too large and um only a few records have some missing values then it is okay to just get rid of those entire rows for example. So if you have a million records and out of which 100 records don't have full data. So there are some missing values in about 100 records. So it's absolutely fine because it's a small percentage of the data. So you can get rid of the entire records which have missing values. But that's not a very common situation. Very often you will have multiple or at least you know large number of data set. For example out of million records you may have 50,000 records which are like having missing values. Now that's a significant amount. You cannot get rid of all those records. your analysis will be inaccurate. So how do you handle such situations? So there are again multiple ways of doing it. One is you can probably if a particular values are missing in a particular column, you can probably take the mean value for that particular column and fill all the missing values with the mean value. So that first of all you don't get errors because of missing values and second you don't get results that are way off because these values are completely different from what is there. So that is one way. Then a few other could be either taking the median value or depending on what kind of data we are talking. So something meaningful we will have put in there. If we are doing some machine learning activity then obviously as a part of data preparation you need to split the data into training and test data set. The reason being if you try to test with a data set which the system has already seen as a part of training then it will tend to give reasonably accurate results because it has already seen that data and that is not a good measure of the accuracy of the system. So typically you take the entire data set the input data set and split it into two parts and again the ratio can vary from person to person individual preferences. Some people like to split it into 50/50 some people like it as 63.33 and 33.3 is basically 2/3 and 1/3 and some people do it as 80/20 80 for training and 20 for testing. So you split the data, perform the training with the 80% and then use the remaining 20% for testing. All right. So that is one more data preparation activity that needs to be done before you start analyzing or applying the data or putting the data through the model. Then the next step is model planning. Now this models can be statistical models. This could be machine learning models. So you need to decide what kind of models you're going to use. Again, it depends on what is the problem you're trying to solve. If it is a regression problem, you need to think of a regression algorithm and come up with a regression model. So, it could be linear regression. Or if you're talking about classification, then you need to pick up an appropriate classification algorithm like logistic regression or decision tree or SVM and then you need to train that particular model. So that is the model building or model planning process and the cleaned up data has to be fed into the model. And apart from cleaning you may also have to in order to determine what kind of model you will use you have to perform some exploratory data analysis to understand the relationship between the various variables and u see if the data is appropriate and so on. Right? So that is the additional preparatory step that needs to be done. So little bit of details about exploratory data analysis. So what exactly is exploratory data analysis? It's basically to as the name suggests you're just exploring you just receive the data and you're trying to explore and uh find out what are the data types and what is the is the data clean in in each of the columns what is the maximum minimum value. So for example there are out ofthe-box functionality available in tools like R. So if you just ask for a summary of the table, it will tell you for each column it will give some details as to what is the mean value, what is the maximum value and so on and so forth. So this exercise or this exploratory analysis is to get an understanding of your data and then you can take steps to during this process you find there are a lot of missing values you need to take steps to fix those. You will also get an idea about what kind of model to be used and so on and so forth. What are the various techniques used for exploratory data analysis? Typically these would be visualization techniques like you use histograms. Uh you can use box plots, you can use scatter plots. So these are very quick ways of identifying the patterns or a few of the trends of the data and so on. And then once your data is ready, you you've decided on the model, what kind of model, what kind of algorithm you're going to use. If you're trying to do machine learning, you need to pass your 80% the training data or rather you use that training data to train your model. And the training process itself is iterative. So the training process you may have to perform multiple times and once the training is done and you feel it is giving good accuracy then you move on to test. So you take the remaining 20% of the data. Remember we split the data into training and test. So the test data is now used to check the accuracy or how well our model is performing and if if there are further issues let's say and model is still during testing if the accuracy is not good then you may want to retrain your model or use a different model. So this whole thing again can be iterative but if the test process is passed or if the model passes the test then it can go into production and it will be deployed. All right. So what are the various tools that we use for model planning? R is an excellent tool in a lot of ways. Whether you're doing regular statistical analysis or machine learning or any of these activities or in along with our studio provides a very powerful environment to do data analysis including visualization. It has a very good integrated visualization or plot mechanism which can be used for doing exploratory data analysis and then later on to do analysis, detail analysis and machine learning and so on and so forth. Then of course you can write Python programs. Python offers a rich library for performing data analysis and machine learning and so on. MATLAB is a very popular tool as well especially during education. So this is a very easy to learn tool. So MATLAB is another uh tool that can be used. And then last but not least SAS. SAS is again very powerful. It is a preparatory tool and it has all the components that are required to perform very good statistical analysis or perform data science. So those are the various tools that would be required for or that that can be used for model building. And uh so the next step is model building. So we have done the planning part. We said okay what is the algorithm we going to use? What kind of model we going to use? Now we need to actually train this model or build the model rather so that it can then be deployed. So what are the various uh ways or what are the various types of model building activities. So it could be let's say in this particular example that we have taken you want to find out the price of 1.35 karat diamond. So this is let's say a linear regression problem. You have data for various carats of diamond and you use that information you pass it through a linear regression model or you create a linear regression model which can then predict your price for 1.35 carat. So this is one example of model building and then little bit details of how linear regression works. So linear regression is basically coming up with a relation between an independent variable and a dependent variable. So it is pretty much like coming up with equation of a a straight line which is the best fit for the given data. So like for example here y is equal to mx + c. So y is the dependent variable and x is the independent variable. We need to determine the values of m and c for our given data. So that is what the training process of uh this model does. At the end of the training process, you have a certain value of m and c and um that is used for predicting the values of any new data that comes. All right. So the way it works is we use the training and the test data set to train the model and then validate whether the model is working fine or not using test data and uh if it is working fine then it is taken to the next level which is put in production. If not the model has to be retrained. If the accuracy is not good enough then the model is retrained maybe with more data or you come up with a newer model or algorithm and then repeat that process. So it is an iterative process. Once the training is completed training and test then this model is deployed and we can use this particular model to determine what is the price of 1.35 karat diamond. Remember that was our problem statement. So now that we have the best fit for this given data, we have the price of 1.35 karat diamond which is 10,000. So this is one example of how this whole process works. Now how do we build the model? There are multiple ways. You can use Python for example and use libraries like pandas or numpy to build the model and implement it. This will be available as a separate tutorial, a separate video in this playlist. So stay tuned for that. Moving on, once we have the results, the next step is to communicate this results to the appropriate stakeholders. So which is basically taking this results and preparing like a presentation or a dashboard and communicating these results to the concerned people. So finishing or getting the results of the analysis is not the last step. But you need to as a data scientist take this results and present it to the team that has given you this problem in the first place and explain your findings explain the findings of this exercise and recommend maybe what steps they need to take in order to overcome this problem or solve this problem. So that is the pretty much once that is accepted and the last step is to operationalize. So if everything is fine your data scientists presentations are accepted then they put it into practice and thereby they will be able to improve or solve the problem that they stated in step one. Okay. So quick summary of the life cycle. You have a concept study which is basically understanding the problem asking the right questions and trying to see if there is uh enough data to solve this problem and then even maybe gather the data. Then data preparation the raw data needs to be manipulated. You need to do data munching so that you have the data in a certain proper format to be used by the model or our analytics system. And then you need to do the model planning. What kind of a model? what algorithm you will use for a given problem and then the model building. So the exact execution of that model happens in step four and you implement and execute that model and uh put the data through the analysis in this step and then you get the results. This results are then communicated packaged and presented and communicated to the stakeholders and once that is accepted that is operationalized. So that is the final step. Let's begin this lesson by defining the term statistics. Statistics is a mathematical science pertaining to the collection, presentation, analysis, and interpretation of data. It's widely used to understand the complex problems of the real world and simplify them to make well-informed decisions. Several statistical principles, functions, and algorithms can be used to analyze primary data, build a statistical model, and predict the outcomes. An analysis of any situation can be done in two ways. Statistical analysis or a non-statistical analysis. Statistical analysis is the science of collecting, exploring, and presenting large amounts of data to identify the patterns and trends. Statistical analysis is also called quantitative analysis. Non-statistical analysis provides generic information and includes text, sound, still images, and moving images. Non-statistical analysis is also called qualitative analysis. Although both forms of analysis provide results, statistical analysis gives more insight and a clearer picture, a feature that makes it vital for businesses. There are two major categories of statistics, descriptive statistics and inferial statistics. Descriptive statistics helps organize data and focuses on the main characteristics of the data. It provides a summary of the data numerically or graphically. Numerical measures such as average, mode, standard deviation or SD and correlation are used to describe the features of a data set. Suppose you want to study the height of students in a classroom. In the descriptive statistics, you would record the height of every person in the classroom and then find out the maximum height, minimum height, and average height of the population. Inferial statistics generalizes the larger data set and applies probability theory to draw a conclusion. It allows you to infer population parameters based on the sample statistics and to model relationships within the data. Modeling allows you to develop mathematical equations which describe the inter relationships between two or more variables. Consider the same example of calculating the height of students in the classroom. In inferial statistics, you would categorize height as tall, medium, and small, and then take only a small sample from the population to study the height of students in the classroom. The field of statistics touches our lives in many ways. From the daily routines in our homes to the business of making the greatest cities run, the effect of statistics are everywhere. There are various statistical terms that one should be aware of while dealing with statistics. population, sample variable, quantitative variable, qualitative variable, discrete variable, continuous variable. A population is the group from which data is to be collected. A sample is a subset of a population. A variable is a feature that is characteristic of any member of the population differing in quality or quantity from another member. A variable differing in quantity is called a quantitative variable. For example, the weight of a person, number of people in a car. A variable differing in quality is called a qualitative variable or attribute. For example, color, the degree of damage of a car in an accident. A discrete variable is one which no value can be assumed between the two given values. For example, the number of children in a family. A continuous variable is one in which any value can be assumed between the two given values. For example, the time taken for a 100 meter run. Typically, there are four types of statistical measures used to describe the data. They are measures of frequency, measures of central tendency, measures of spread, measures of position. Let's learn each in detail. Frequency of the data indicates the number of times a particular data value occurs in the given data set. The measures of frequency are number and percentage. Central tendency indicates whether the data values tend to accumulate in the middle of the distribution or toward the end. The measures of central tendency are mean, median, and mode. Spread describes how similar or varied the set of observed values are for a particular variable. The measures of spread are standard deviation, variance and quartiles. The measure of spread are also called measures of dispersion. Position identifies the exact location of a particular data value in the given data set. The measures of position are percentiles, quartortiles and standard scores. Statistical analysis system or SAS provides a list of procedures to perform descriptive statistics. They are as follows. Proc print, proc contents, proc means, proc frequency, proc univariant, proc gart, proc box plot, proc g-plot, proc print, it prints all the variables in a SAS data set. Proc contents, it describes the structure of a data set. Proc means it provides data summarization tools to compute descriptive statistics for variables across all observations and within the groups of observations. Proc frequency it produces oneway to inway frequency and crosstabulation tables. Frequencies can also be an output of a SAS data set. PROC univariat. It goes beyond what proc means does and is useful in conducting some basic statistical analyses and includes highresolution graphical features. Proc G- chart. The G-chart procedure produces six types of charts. Block charts, horizontal vertical bar charts, pie doughut charts, and star charts. These charts graphically represent the value of a statistic calculated for one or more variables in an input SAS data set. The variables can be either numeric or character. Proc box plot. The box plot procedure creates sidebyside box and whisker plots of measurements organized in groups. A box and whisker plot displays the mean, quartiles, and minimum and maximum observations for a group. Procplot G-plot procedure creates two-dimensional graphs including simple scatter plots, overlay plots in which multiple sets of data points are displayed on one set of axis, plots against the second vertical axis, bubble plots, and logarithmic plots. In this demo, you'll learn how to use descriptive statistics to analyze the mean from the electronic data set. Let's import the electronic data set into the SAS console. In the left plane, rightclick the electronic.xlsx data set and click import data. The code to import the data generates automatically. Copy the code and paste it in the new window. The proc means procedure is used to analyze the mean of the imported data set. The keyword data identifies the input data set. In this demo, the input data set is electronic. The output obtained is shown on the screen. Note that the number of observations, mean, standard deviation, and maximum and minimum values of the electronic data set are obtained. This concludes the demo on how to use descriptive statistics to analyze the mean from the electronic data set. So far you have learned about descriptive statistics. Let's now learn about inferial statistics. Hypothesis testing is an inferial statistical technique to determine whether there is enough evidence in a data sample to infer that a certain condition holds true for the entire population. To understand the characteristics of the general population, we take a random sample and analyze the properties of the sample. We then test whether or not the identified conclusions correctly represent the population as a whole. The population of hypothesis testing is to choose between two competing hypotheses about the value of a population parameter. For example, one hypothesis might claim that the wages of men and women are equal, while the other might claim that women make more than men. Hypothesis testing is formulated in terms of two hypotheses. Null hypothesis, which is referred to as Hnull. Alternative hypothesis, which is referred to as H1. The null hypothesis is assumed to be true unless there is strong evidence to the contrary. The alternative hypothesis is assumed to be true when the null hypothesis is proven false. Let's understand the null hypothesis and alternative hypothesis using a general example. Null hypothesis attempts to show that no variation exists between variables and alternative hypothesis is any hypothesis other than the null. For example, say a pharmaceutical company has introduced a medicine in the market for a particular disease and people have been using it for a considerable period of time and it's generally considered safe. If the medicine is proved to be safe, then it is referred to as null hypothesis. To reject null hypothesis, we should prove that the medicine is unsafe. If the null hypothesis is rejected, then the alternative hypothesis is used. Before you perform any statistical tests with variables, it's significant to recognize the nature of the variables involved. Based on the nature of the variables, it's classified into four types. They are categorical or nominal variables, ordinal variables, interval variables, and ratio variables. Nominal variables are ones which have two or more categories, and it's impossible to order the values. Examples of nominal variables include gender and blood group. Ordinal variables have values ordered logically. However, the relative distance between two data values is not clear. Examples of ordinal variables include considering the size of a coffee cup, large, medium, and small, and considering the ratings of a product, bad, good, and best. Interval variables are similar to ordinal variables except that the values are measured in a way where their differences are meaningful. With an interval scale, equal differences between scale values do have equal quantitative meaning. For this reason, an interval scale provides more quantitative information than the ordinal scale. The interval scale does not have a true zero point. A true zero point means that a value of zero on the scale represents zero quantity of the construct being assessed. Examples of interval variables include the Fahrenheit scale used to measure temperature and distance between two compartments in a train. Ratio scales are similar to interval scales in that equal differences between scale values have equal quantitative meaning. However, ratio scales also have a true zero point which give them an additional property. For example, the system of inches used with a common ruler is an example of a ratio scale. There is a true zero point because 0 in does in fact indicate a complete absence of length. In this demo, you'll learn how to perform the hypothesis testing using SAS. In this example, let's check against the length of certain observations from a random sample. The keyword data identifies the input data set. The input statement is used to declare the aging variable and cards to read data into SAS. Let's perform a t test to check the null hypothesis. Let's assume that the null hypothesis to be that the mean days to deliver a product is 6 days. So null hypothesis equals 6. Alpha value is the probability of making an error which is 5% standard and hence alpha equals 0.05. The variable statement names the variable to be used in the analysis. The output is shown on the screen. Note that the p value is greater than the alpha value which is 0.05. Therefore, we fail to reject the null hypothesis. This concludes the demo on how to perform the hypothesis testing using SAS. Let's now learn about hypothesis testing procedures. There are two types of hypothesis testing procedures. They are parametric tests and non-parametric tests. In statistical inference or hypothesis testing, the traditional tests such as test test and ANOVA are called parametric tests. They depend on the specification of a probability distribution except for a set of free parameters. In simple words, you can say that if the population information is known completely by its parameter, then it is called a parametric test. If the population or parameter information is not known and you are still required to test the hypothesis of the population, then it's called a non-parametric test. Non-parametric tests do not require any strict distributional assumptions. There are various parametric tests. They are as follows. TA test, ANOVA, chai squared, linear regression. Let's understand them in detail. Test. A t test determines if two sets of data are significantly different from each other. The test test is used in the following situations. To test if the mean is significantly different than a hypothesized value. To test if the mean for two independent groups is significantly different. To test if the mean for two dependent or paired groups is significantly different. For example, let's say you have to find out which region spends the highest amount of money on shopping. It's impractical to ask everyone in the different regions about their shopping expenditure. In this case, you can calculate the highest shopping expenditure by collecting sample observations from each region. With the help of the t test, you can check if the difference between the regions are significant or a statistical fluke. ANOVA ANOVA is a generalized version of the t test and used when the mean of the interval dependent variable is different to the categorical independent variable. When we want to check variance between two or more groups, we apply the ANOVA test. For example, let's look at the same example of the t test example. Now you want to check how much people in various regions spend every month on shopping. In this case, there are four groups, namely east, west, north, and south. With the help of the ANOVA test, you can check if the difference between the regions is significant or a statistical fluke. Chi square. Chiquare is a statistical test used to compare observed data with data you would expect to obtain according to a specific hypothesis. Let's understand the Chiquare test through an example. You have a data set of male shoppers and female shoppers. Let's say you need to assess whether the probability of females purchasing items of $500 or more is significantly different from the probability of males purchasing items of $500 or more. Linear regression. There are two types of linear regression. Simple linear regression and multiple linear regression. Simple linear regression is used when one wants to test how well a variable predicts another variable. Multiple linear regression allows one to test how well multiple variables or independent variables predict a variable of interest. When using multiple linear regression, we additionally assume the predictor variables are independent. For example, finding relationship between any two variables, say sales and profit is called simple linear regression. Finding relationship between any three variables, say sales, cost, telemarketing is called multiple linear regression. Some of the non-parametric tests are will coxin rank sum test and crus Wallace H test. Will coxin rank sum test. The Woxin signed rank test is a nonparametric statistical hypothesis test used to compare two related samples or matched samples to assess whether or not their population mean ranks differ. In woxin rank sum test, you can test the null hypothesis on the basis of the ranks of the observations. Cruscol Wallace H test. Cresco Wallace H test is a rank-based non-parametric test used to compare independent samples of equal or different sample sizes. In this test, you can test the null hypothesis on the basis of the ranks of the independent samples. The advantages of parametric tests are as follows. Provide information about the population in terms of parameters and confidence intervals. easier to use in modeling, analyzing, and for describing data with central tendencies and data transformations. Express the relationship between two or more variables. Don't need to convert data into rank order to test. The disadvantages of parametric tests are as follows. Only support normally distributed data. Only applicable on variables, not attributes. Let's now list the advantages and disadvantages of non-parametric tests. The advantages of non-parametric tests are as follows. Simple and easy to understand. Do not involve population parameters and sampling theory. Make fewer assumptions. Provide results similar to parametric procedures. The disadvantages of non-parametric tests are as follows. Not as efficient as parametric tests, difficult to perform operations on large samples manually. >> We'll discuss the types of distribution in statistics. But before we move ahead, let's have a brief introduction on what is probability distribution. A probability distribution is a list of all of the possible outcomes of a random variable along with the corresponding probability values. And it is used in many fields, but we rarely do explain what they are. So in this video we'll discuss the three main types of probability distribution that is normal, binomial and poison distribution. So let's move ahead. So what is normal distribution? Normal distribution is a continuous probability density that has a probability density function which gives us a symmetrical bell curve. Now data can be distributed or spread out in different ways but there are many cases where the data tends to be around a central value with no bias to the left or right which means that it doesn't show any particular spikes towards the left or the right and it gets close to a normal distribution. Half of the data will fall on the left of the mean and the other half will fall on the right. Now let's take a look at a graph which shows the height distribution in a class. As you can see, the average height is in the middle and the data to the left of the average height represents the short people and the data to the right of it represents the taller people. The y-axis shows us the likelihood of any of these heights occurring. The average height has the most distribution or it has the most number of cases in the class. And as the height decreases or increases the number of people who have that height also decreases. This kind of a distribution is called a normal distribution where the average or the mean is always the highest point and any other point after that or before that is significantly lower. The resulting data gives us a bell curve. And as you can see there is no abrupt bias or spike in the data anywhere except for the average height. So this kind of a curve is called a bell curve and it's usually seen in a normal distribution. The reason we call this a normal distribution is because the data is normally distributed with the average being the highest and all the other data points having a lower likelihood. Now we came across two terms which are associated with normal distribution. Continuous probability density and probability density function. What is continuous probability density? Continuous probability density is a probability distribution where the random variable X can take any given value. Because there are infinite values that X could assume. The probability of X taking on any specific value is zero. For example, let's say you have a continuous probability density for men's height. What is the probability that a man will have the exact height of 70 in? It is impossible to find this out because the probability of one man measuring exactly 70 in is very low. It is more probable that he will measure around 70.1 in or maybe 69.97 in. And it doesn't stop there. The fact is that it's impossible to exactly measure any variable that's on a continuous scale. And because of this, it's impossible to figure out the probability of one exact measurement which is occurring in a continuous probability density. Next, we have the probability density function. It's nothing but a function or an expression which is used to define the range of values that a continuous random variable can take. An example of this would be to godge the risk and reward of a stock. A probability density function is a statistical measure which is used to gge the likelihood of a discrete value. A discrete variable can be measured exactly while a continuous variable can have infinite values. However, for both continuous as well as discrete variables, we can define a function which gives us the range of values within which these variables will fall. And that function is known as the probability density function. Now let's take a look at standard deviation. What is standard deviation? Standard deviation is used to measure how the values in your data differ from one another or how spread out your data is. A standard deviation is a statistic that measures the dispersion of a data set relative to its mean. The standard deviation is calculated as the square root of variance by determining each data point's deviation relative to the mean. If the data points are further from the mean, that means that there's a higher deviation within the data set and then the data is set to be more spread out. This leads to a higher standard deviation too. Let's take an example of income in rural and urban areas. In rural areas, let's say such as farming areas, the income doesn't differ that much. more or less everyone earns the same. Because of this, our bell curve has a very low standard deviation and it has a very narrow peak. However, in urban areas, the wealth distribution is very uneven. Some people can have very high incomes and can be earning a lot while other people can have very low incomes. The furthermore the data distribution between these two income points is going to be more spread out because there are lot more people living there who work in various fields and who have various incomes. Because of this, our standard deviation is more spread out and our bell curve will also have a wider peak. Now, how can we find the standard deviation? Standard deviation is obtained by subtracting each data value from the mean and finding the squared average of these values. Let's look at how we can do this with the help of an example. These values correspond to the height of various dogs. We can find the mean by finding the average of all these values which is nothing but adding all the values and dividing it by the total number of values. The mean that we get is 394. This means that the average height of a dog is 394 mm. To find the standard deviation, first we need to subtract the height from the mean. This will tell us how far from the mean our data points actually are. Next, we will square up all of these differences and add them up and again divide it by the total number of values that we have. This is called the variance. The variance that we get in this case is 21704. Finally, when we find the square root of this value, we will get the standard deviation. The standard deviation here is 147. The standard deviation will tell us how our data points differ from the average. And it gives us a basic value suggesting how spread out our data is from the very middle or from the mean. So when we plot these values, this value 147 will mean that a curve will have a width of 147 points around the mean. Now what is the standard normal distribution? The standard normal distribution is a type of normal distribution that has a mean of zero and a standard deviation of one. This means that the normal distribution has its center at zero and it has intervals which increase by one. All normal distributions like the standard normal distribution are unimodel and symmetrically distributed with a bell-shaped curve. However, a normal distribution can take on any value as its mean and standard deviation. In the standard normal distribution, however, the mean and standard deviation are always fixed. When you standardize a normal distribution, the mean becomes zero and the standard deviation becomes one. This allows you to easily calculate the probability of certain values occurring in your distribution or to compare data sets with different mean and standard deviations. The curve shows a standard normal distribution. As you can see again the data is centered at zero. This does not mean that the data necessarily starts at zero. This means that after standardizing this point is where our mean will lie. In a standard normal distribution the standard deviation is one. So all the data points will increase or decrease in steps of one. Let's better understand a standard normal distribution with the help of an example. Again as you can see the data is centered around zero which is nothing but the mean. Let's again consider the weights of students in class 8. The average weight here is around 50 kgs and the data increases and decreases in steps of five. The data over here in this curve is evenly distributed along these steps. This is what a standard normal distribution will look like. We already know that the mean of our data is 50. And because the data is increasing and decreasing in equal steps, we can just standardize it and take it to mean that the data is increasing and decreasing in steps of one. This is what a standard normal distribution looks like. And when you have a data which looks like this, you can always standardize it and convert it into a standard normal distribution. Now standard normal distribution has a couple of properties which makes calculation comparatively easy. The first one is that 68% of the values fall within the first standard deviation. Which means that 68% of all data values on this curve will fall between the range of minus1 to 1 or the first interval ranging from minus1 to 1. The second probability is that 95% of the rest of the values are within the second standard deviation or from the second negative point to the second positive point. And finally 99.7% of the values fall within the third standard deviation or from the third negative point to the third positive point. This makes calculations on standard normal distribution fairly easy. You can compare scores on different distributions with different means and standard deviations. You can normalize scores for statistical decision making using standard normal distribution. You can find the probability of observations in a distribution which fall above or below a given value. And finally, you can find the probability that a mean significantly differs from a population mean. Now let's take a look at zcore. So what is a zcore? A zcore is used to tell us how far from the mean a data point actually is. It is calculated using the mean and standard deviation. So it can be said that the zed score is how many standard deviations below the mean our data is. Basically by using the zed score we can get an approximate location of where our data point lies on the graph with regards to the mean. Now the zed score is given by subtracting the data point from the mean and dividing it by standard deviation. This can also be written as x minus mu divided by sigma. Now any normal distribution can be standardized by converting its values into zed scores. The zed score will tell you how many standard deviation from the mean each values lie. While data points are referred to as X in a normal distribution, they are called zed or zed scores in the zed distribution. A zed score is a standard score that will tell you how many standard deviations away from the mean an individual point will lie. A positive zed score will mean that your x value is greater than the mean and a negative zed score will mean that your x value is less than the mean. A zed score of zero will mean that your x value is equal to the mean. And again to standardize a value from a normal distribution all we have to do is convert it to a zed score by subtracting the mean from our individual value and dividing it by the standard deviation. Now let's see how we can find the zed score from data points with the help of a solved example. Let's do a case study. In this case study we'll be taking the summary of daily travel time of a person who's commuting to and from work. All these values are in minutes and using these values we have to calculate the mean, the standard deviation and the zed score. These values are as shown. As we can see there are 13 values in total. Let's start by finding the mean. The mean is the average and it can be gotten by adding all of these values and dividing it by the total number of values. This gives us a value of 38.6. The mean tells us the average of all our data points, which means on an average, he travels for 38.6 minutes to reach work. Next, let's subtract the individual values from our mean and calculate the variance and standard deviation. The values on the left give us the values that we get after subtracting it from the mean. And the variance can be calculated by squaring all of these values, adding up all of the squared values, and dividing it by the total number of values. At the end of the day, we get a variance of 140. To calculate the standard deviation, all we have to do is take a square root of the variance, which gives us a value of 11.8. Now, the mean signifies the average of our values, and we already know this. It gives us the average time which is taken to travel. But the standard deviation will tell us the average value of how much our data points differ from the mean. It tells us the deviation within our own data and it tells us how far away on an average a point is from the mean. Now the value that we get is 11.8 which means that on an average a single data point is around 11.8 data points away from the mean. Now let's calculate the zed score. The zed score is given by subtracting individual data points from the mean and dividing it by the standard deviation. We know that we have a standard deviation of 11.8 and a mean of 38.6. Using these values, we can calculate the zed scores for individual x values. Now we know that a negative zed score means that our x value is lower than our mean. But what does the number 1.06 mean? This means that the zed score for 26 is 1.06 standard deviations away from the mean. The negative symbol here means that our x value is less than the mean. And by how less? 1.06 times the standard deviation. Now we know that the negative value of a zed score means that our x value is less than our mean. But what does the number 1.06 mean? This means that the zed score is 1.06 times the standard deviation less than the mean. The same thing can be said for the zed score of 33. It is 0.47 times the standard deviation less than the mean. The zcore of 65 is 2.23 times the standard deviation more than the mean. That means it has to be added to the mean. The reason that we know it's more than the mean is because this has a positive value. So this means that using zed scores, we can know where our data points fall relative to other points on the graph. The zed score will tell us how far away from the mean a point is in steps of our standard deviation. basics and terminology. The first one is outcome. Whenever we do an experiment like flipping a coin or rolling a dice, we get an outcome. For example, if we flip a coin, we get an outcome of heads or tails. And if you roll a dieice, we get an outcome of 1 2 3 4 5 or six. Random experiment. A random experiment is any well- definfined procedure that produces an observable outcome that could not be perfectly predicted in advance. A random experiment must be well defined to eliminate any vagueness or surprise. It must produce a definite observable outcome so that you know what happened after the random experiment is run. Random events. Consider a simple example. Let us say that we toss a coin up in the air. What can happen when it gets back? It either gives a head or a tail. These two are known as outcome. And the occurrence of an outcome is an event. Thus, the event is the outcome of some phenomenon. The last one is sample space. A sample space is a collection or a set of possible outcomes of a random experiment. The sample space is represented using the symbol S. The subset of all possible outcomes of an experiment is called events. And a sample space may contain a number of outcomes that depends on the experiment. If it contains a finite number of outcomes, then it is known as a discrete or finite sample spaces. Now let's discuss what is random variable. A random variable is a numerical description of the outcome of a statistical experiment. A random variable that may assume only a finite number of values is set to be discrete. One that may assume any value in some interval on the real number line is set to be continuous. Let's see an example. Let X be a random variable defined as a sum of numbers when two dices are rolled. X can assume the values 2 3 4 5 6 7 8 9 10 11 and 12. Notice there's no one here because the sum on the two dice can never be one. Now that we know the basics, let's move on to binomial distribution. The binomial distribution is used when there are exactly two mutually exclusive outcomes of a trial. These outcomes are appropriately labeled success and failure. The binomial distribution is used to obtain the probability of observing X successes in n number of trials with the probability of success on a single trial denoted by P. The binomial distribution assumes that P is fixed for all the trials. Here's a real life example of a binomial distribution. Suppose you purchase a lottery ticket. Then either you are going to win the lottery or not. In other words, the outcome will be either success or failure that can be proved through banimal distribution. There are four important conditions that needs to be fulfilled for an experiment to be a binomial experiment. The first one is there should be a fixed number of entries carried out. The outcome of a given trial is only two that is either a success or a failure. The probability of success remains constant from trial to trial. It does not changes from one trial to another. And the trials are independent. The outcome of a trial is not affected by the outcome of any other trial. To calculate the binomial coefficient, we use the formula which is ncr into p ^ r into 1 - p to the power n minus r where r is the number of success in n number of trials and p is the probability of success. 1 - p denotes the probability of a failure. Now let's use this formula to solve an example. Suppose a dice is tossed three times. What is the probability of no five turning up 1 five and 35 turning up? To calculate the no five turning up here r is equal to 0 and n is equal to 3. Substituting the value in the formula we have 3 0 into 1x 6 ^ 0 into 5x 6 ^ 3 where 1x 6 is the probability of success and 5x 6 is the probability of failure. Calculating this equation we'll get the value to be 0.5787. In a similar manner to calculate the probability of 1 15 turning up we'll replace r with one and n will be three. So px1 will be equal to 3 c1 into 1x 6 ^ 1 into 5x 6 ^2 which will come out to be 0.347 and for 35 turning up we substitute r equal to 3 and the formula will remain the same and we'll get the value to be 0.0046. Now that we are done with the concepts of binomal probability distribution, here's a problem for you to solve. Post your answers in the comment section and let us know. A poison distribution is a probability distribution used in statistics to show how many times an event is likely to happen over a given period of time. To put it another way, it's a count distribution. Poison distribution are frequently used to comprehend independent event at a constant rate over a given interval of time. The poison distribution was developed by French mathematician Simon Dennis poison in 1837. A poison distribution is used in cases where the chances of any individual event being a success is very small. The number of defective pencils per box of a 6,000 pencil. the number of plane crashes in India in one year or the number of printing mistakes in each page of a book. All of these example can have use of poison distribution. The poison distribution can be used to calculate how likely it is that something will happen X number of times. A random variable X has a poison distribution with parameter lambda. And the formula for that is e to the power minus lambda into lambda to the power x divided by x factorial where x can be the number of times the event is happening. The value of e is taken as 2.7182. Let's discuss some application of poison distribution. If you want to calculate the number of deaths per day or week due to rare disease in a hospital, you can use a poison distribution. In a similar manner, the count of bacteria perceive >> are an aspiring data scientist who's looking out for online training and certification in data science from the best universities and industry experts then search no more simply learns post-graduate program in data science from Caltech University in collaboration with IBM should be the right choice. For more details on this program, please use the link in the description box below. Hello everyone, welcome to another session by simply learn. Today we are going to discuss the base theorem, an important subtopic that comes under probability theory. We'll start this video by talking about probability and conditional probability. After that we'll move on to the base theorem and understand its formula and a real life example where the base theorem can be used. So let's get started. What is probability? Probability is a branch of mathematics concerning numerical descriptions of how likely an event is to occur or how likely it is that a proposition is true. The probability of an event is a number between 0 and one. Well, roughly speaking, zero indicates the impossibility of the event and one indicates certaintity. The higher the probability of an event, the more likely is that the event will occur. Let's look at an example. A simple example is the tossing of a fair unbiased coin. Since the coin is fair, the outcome that is heads and the tails are both equally probable. The probability of heads equals the probability of the tails. And since no other outcomes are possible, the probability of either heads or tails can be set to be 1 by 2, which is also 50%. The probability of an event can be calculated by number of ways it can happen divided by the total number of outcomes. Now that we know about the probability, let's see if you can answer this question. What is the probability of drawing a jack and a queen consecutively from a deck of 52 cards without replacement? Here are your options. Post your answers in the comment section and let us know. Now let's move on to conditional probability. Let A and B be the two events associated with a random experiment. Then the probability of A's occurrence under the condition that B has already occurred and probability of B is not equal to zero is called the conditional probability. It is denoted by P A/B. Thus we can say that P A/B is equal to P A intersection B divided by P of B where P A/B is the probability of occurrence of A given that B has already occurred and PB is the probability of occurrence of B. To know more about conditional probability, you can check our previous video which is specifically on conditional probability. Now let's move on to base theorem. The base theorem is a mathematical formula for calculating condition probability in probability and statistics. In other words, it is used to figure out how likely an event is associated on its proximity to another. B law or base rule are the other names of this theorem. The formula for the base theorem can be written in a variety of ways. The most common version is P A/ B is equal to P of B / A into P of A divided by P of B. Where P A/B is the conditional probability of event A occurring given that B is true and P A and P of B are the probabilities of A and B occurring independently of one another. Let's solve a problem using the base theorem to understand it better. There is a cricket match tomorrow and in recent years it has rained only 5 days each year. Unfortunately the meteorologist has predicted the rain for tomorrow. Now when it rains the mologist correctly forecast rain 90% of the time and when it doesn't rain he incorrectly forecast rain 10% of the time. Let's calculate what is the probability that it will rain on the mash day. So the two sample spaces here are the events that it rains and it does not rain. Additionally, a third event is also there that mologist predicts the rain. So the notation for these events appear below. Event A1 is equal to it rains on the match day. Event A2 that it does not rain on the match day and event B is the meteorologist predicting the rain. Now in terms of probability we know the following probability of A1 is 5 by 365 that it rains 5 days in a year which will come out to be 0.0136. P A2 is 360 by 365 that is no days for 360 days in an year which will come out to be 986. PB/ A1 is.9. This signifies when it rains the meteorologist predicts the rain 90% of the time. In a similar manner, PB BY A2 is.1 that it does not rain. The meteorologist predicts the rain 10% of the time. Combining all this, we can calculate P A1/B that is the probability it will rain on the given match day given a forecast of rain by mologist. The answer can be determined using the base theorem as shown below. So here's the formula of the base theorem and putting all the values that we have calculated in the previous slide. The probability that it will rain on the match day given a forecast of the rain by meteorologist will come out to be 0.111 which will be equal to 11.11%. So there's an 11% chance that it will rain on the match day given that the meteorologist has predicted the rain. I hope this example is clear to you. >> It's a weekend and John decided to watch the latest movie recommended by Netflix at his friend's place. Before heading out, he asked Siri about the weather and realized it would rain. So, he decided to take his Tesla for the long journey and switched to autopilot on the highway. After coming home from the eventful day, he started wondering how technology has made his life easy. He did some research on the internet and found out that Netflix, Siri, and Tesla are all using AI. So, what is AI? AI or artificial intelligence is nothing but making computers based machines think and act like humans. Artificial intelligence is not a new term. John McCarthy, a computer scientist, coined the term artificial intelligence back in 1956. But it took time to evolve as it demanded heavy computing power. Artificial intelligence is not confined to just movie recommendations and virtual assistants. Broadly classifying, there are three types of AI. Artificial narrow intelligence, also called weak AI, is the stage where machines can perform a specific task. Netflix, Siri, chatbots, spatial recommendation systems are all examples of artificial narrow intelligence. Next up, we have artificial general intelligence, referred to as an intelligent agent's capacity to comprehend or pick up any intellectual skill that a human can't. We are halfway into successfully implementing this space. IBM's Watson supercomputer and GPT3 fall under this category. And lastly, artificial super intelligence. It is the stage where machines surpass human intelligence. You might have seen this in movies and imagined how the world would be if machines occupy it. Fascinated by this, John did more research and found out that machine learning, deep learning, and natural language processing are all connected with artificial intelligence. Machine learning, a subset of AI, is the process of automating and enhancing how computers learn from their experiences without human help. Machine learning can be used in email spam detection, medical diagnosis, etc. Deep learning can be considered a subset of machine learning. It is a field that is based on learning and improving on its own by examining computer algorithms. While machine learning uses simpler concepts, deep learning works with artificial neural networks which are designed to imitate the human brain. This technology can be applied in face recognition, speech recognition, and many more applications. Natural language processing, popularly known as NLP, can be defined as the ability of machines to learn human language and translate it. Chatbots fall under this category. Artificial intelligence is advancing in every crucial field like healthcare, education, robotics, banking, e-commerce, and the list goes on. Like in healthcare, AI is used to identify diseases, helping health care service providers and their patients make better treatment and lifestyle decisions. Coming to the education sector, AI is helping teachers automate grading, organizing, and facilitating parent guardian conversations. In robotics, AI powered robots employ real-time updates to detect obstructions in their path and instantaneously design their routes. Artificial intelligence provides advanced data analytics that is transforming banking by reducing fraud and enhancing compliance. With this growing demand for AI, more and more industries are looking for AI engineers who can help them develop intelligent systems and offer them lucrative salaries going north of $120,000. The future of AI looks promising with the AI market expected to reach $190 billion by 2025. >> We know humans learn from their past experiences and machines follow instructions given by humans. But what if humans can train the machines to learn from their past data and do what humans can do and much faster? Well, that's called machine learning. But it's a lot more than just learning. It's also about understanding and reasoning. So today we will learn about the basics of machine learning. So that's Paul. He loves listening to new songs. He either likes them or dislikes them. Paul decides this on the basis of the song's tempo, genre, intensity, and the gender of voice. For simplicity, let's just use tempo and intensity for now. So here tempo is on the x-axis ranging from relaxed to fast whereas intensity is on the y-axis ranging from light to soaring. We see that Paul likes the song with fast tempo and soaring intensity while he dislikes the song with relaxed tempo and light intensity. So now we know Paul's choices. Let's say Paul listens to a new song. Let's name it as song A. Song A has fast tempo and a soaring intensity. So it lies somewhere here. Looking at the data, can you guess whether Paul will like the song or not? Correct. So Paul likes this song. By looking at Paul's past choices, we were able to classify the unknown song very easily. Right? Let's say now Paul listens to a new song. Let's label it as song B. So song B lies somewhere here with medium tempo and medium intensity. Neither relaxed nor fast, neither light nor soaring. Now, can you guess whether Paul likes it or not? Not able to guess whether Paul will like it or dislike it. Are the choices unclear? Correct. We could easily classify song A. But when the choice became complicated as in the case of song B, yes, and that's where machine learning comes in. Let's see how. In the same example for song B, if we draw a circle around the song B, we see that there are four votes for like whereas one vote for dislike. If w