Machine Learning With Python Full Course 2026 | Python Machine Learning For Beginners | Simplilearn

Key Takeaways

learning using Python course. Let me ask you something interesting. You type a question into ChatgBT and within seconds it gives you a detailed answer. You upload a photo to an AI tool and suddenly it recognizes object face or even generates a completely new image. You open Spotify and somehow it already knows exactly what song you want to hear next. So what's really happening behind the scenes? Now these systems are not just running simple programs. They are learning patterns from massive amounts of data. This ability for computers to learn from data and improve over time is what we call machine learning. And today one of the most powerful tools used to build these intelligent system is Python. Python has become the backbone of modern AI development powering everything from recommendation system, chatbots, fraud detection, self-driving technology and companies like Google, Netflix, Amazon, OpenAI, they rely heavily on machine learning models built using Python to process huge data sets and make intelligent predictions. But here's where things get really interesting. Machine learning is not just for big tech companies anymore. Today, anyone with the right knowledge can build models that can predict house prices, detect email spams, recognizes images, analyze customer behavior, or even build their own AI powered applications. And that's exactly what this course is going to help you understand. So before we begin, let's quickly look at what we'll be covering. First, we'll understand what machine learning actually is and how machines learn patterns from data. Second, we'll explore why Python has become the most popular programming languages for machine learning. Next, we'll learn about different types of machine learning techniques and algorithms used to solve real world problems. Next, we'll understand how data is prepared and processed before training a machine learning model. And finally, we will see how machine learning models are built, trained, and evaluated using Python. Also, if you are interested in mastering the future of technology, then the professional certificate course in generative AI and machine learning is the perfect opportunity for you. This is offered in collaboration with ENIC Academy, IT, Kpur and it's an 11-month live online interactive program which provides you hands-on expertise on cutting edge areas like genai, machine learning tools like chat GBT, DAL to hugging face. You'll be gaining practical experience through 15 plus projects, integrated labs, life master classes delivered by esteemed IT Kpur faculty alongside earning a prestigious certificate from IIT Kpur. You will receive official Microsoft badges for Azure AI courses and career support through Simply's job assist program. So what are you waiting for? Hurry up and enroll now. The course link is mentioned below. Now before we get started, here's a quick quiz question for you. Which programming language powers many of today's most popular AI tools and machine learning applications? Your options are Python, MSWord, Photoshop or Excel. Let me know your answers in the comment section below. So without any further ado, let's get started. >> I'll be your mentor or trainer for this course of uh machine learning and I hope we have this uh wonderful interactive exciting journey. All right. So let's let's begin with this journey and try to understand what is machine learning and why is there so much buzz around it. Let me first uh let me go through what all we would be covering in this particular course and then we begin the discussion. So if we talk about a learning path you know the first uh you know uh topic that we will be covering today is introduction to machine learning which focuses on the basics of machine learning. Second is supervised learning regression and application which focuses on supervised learning with an emphasis on understanding and implementing different types of regression models. Third is supervised learning classification applications like basically we're trying to cover that what are the different learning techniques available in the machine learning. So first technique that we would be covering up is supervised learning. Under supervised learning we would be covering up regression as well as classification. Then we would be moving on to the ensemble learning method which focuses on advanced ensemble methods to enhance the performance and robustness of the models. Then we would be moving on to unsupervised learning and finally recommener system along with the application. What is machine learning and why is there so much of buzz? Why are you here to learn machine learning? Let's let me put this in another way. So basically we want the machine to get trained with our data. We want the machine to learn from the data so that it can predict data. What kind of data that we want to predict? Why do we want to have want these predictions? Right? Because now are we living in the digital technology where data is all around us. Even this you know uh you know session is data right when all the material that has been sent to you is data anything on the news which is coming is data anything doing for entertainment is data e-commerce is data you know your work profile is data because we are living in huge amount of data data is all around us do you think is there any escape from data now No, not now. Uh when I used to take sessions 5 years, you know, and 7 years uh before, you know, uh the scenario was little trying to adapt. But I don't think so. There is now any survival without data. Can you survive without this data? Not moving on social media. And you know one day, you know, as it says, you know, if the internet stops, do you think your life also stops? your your mobile phone is lost everything is lost isn't it the data has become a lifeline you know and now we see several applications which are working or the concepts that are being bas you know based on data now it automatic translation translation has not become difficult difficult if you want to convert something from English to German to Spanish any language Language translation is right there. You know, we speak to the machine and we get the translation. Virtual personal assistants are there. Image recognition. Email spam filtering. That's that's actually comes under the domain of machine learning that it you know based on the algorithm or the pattern or the text or the words which are there. It is able to filter out whether the mail is a spam or not. So what do you think would be the criteria? Generally the males with spam are saying that you have a lottery system other system or there is a bonus. So that becomes your email spam filtering. Then we have the text and the speech recognition. Medical diagnosis, online fraud detection is there you know where we want to detect uh the how is the online fraud happening. Web search uh search and recommendation engines and of course what is going to be the traffic prediction not only the traffic prediction on the roads but it also relates to the traffic going onto a particular website whether that website is going to get crashed or not. Data becoming now difficult to handle because it's digital data. You know understanding data everything numerically is difficult. You know earlier the data was this much it could be handled. Now it's becomes like this much and this much and this much and it's increasing. So we need certain algorithm we need technologies which can help to analyze so that we can improve our performance. But on if you if you look at on the overall scenario still there is a lot of confusion about the AI the machine learning and the deep learning. Yeah. What is the you know the subset or the issues like what is the artificial what is artificial intelligence machine learning and deep learning? Are you able to distinguish between the three? So this is one of the initial chess you know developed by the computers IBMD blue chess program developed in 1997. When was it developed? It was developed in 1997 by IBM and this particular program was strong enough to defeat the uh world chess champion at that particular point. Uh his name is Gary Kasparov. Okay. He was capable of defeating the world chess champion. Getting my point right and then we have this IBM Watson under machine learning. So AI is the bigger branch. All right. Artificial intelligence is the bigger branch right under which we have the machine learning subset right which we are going to study. In machine learning basically it works on statistical algorithm. That is why we say that before learning machine learning it is important to have a good concepts and knowledge of statistics that helps to understand. So based on statistical foundation, statistical algorithm uh you know the machine is capable of doing Google search algorithm, Amazon recommendation and email spam filtering. How like we just saw that it is capable of filtering the email. And then finally we have the deep learning which is also under machine learning in which we have the alpho the natural speech recognition and the level four automated driving system. The scenario has changed and we are into the AI revolution. So the bigger branch is still AI. We are seeing we are the you know witnessing this revolution in front of us. Under AI we have the machine learning under machine learning we have the deep learning and under deep gen AI task. Right? So when we talk about the gen AI task generation fine-tuning we have the agents automation and the virtual assistants. And this genai is now capable of even doing lot of text generation that we see we ask a lot of things to the chat GPT other uh models like Gemini copilot also we want to image uh generate images video generation all these things are being possible AI that we are talking at the moment is only and only related to the software through software we are able to or give intell intelligent answers. But what is basically the difference between a traditional programming and a machine learning programming? So let me explain you with this particular con uh example. So in traditional programming again we have this data right that this is my data which is 1 2 3 and 4 right and over here we have machine learning algorithm where we have 1 2 3 and four okay the data has not changed I'm giving you a very simple uh you know example now if we talk about the program initially the program when we talk about C, Pascal, Photron, even C++ and Java and even Python they are capable of logic. Now if I want to distinguish that what are the numbers what is the logic behind that these uh you know numbers are even or odd. So simply we understand the logic that if I if I talk about uh Python I percentage 2 is equal equal to zero that means if the remainder divided by two is zero then the number it is odd right else it is odd getting my point. Yes or no? This point is getting clear. So now even if a number 10, it would automatically give me that this is going to be an even number. And if I give the number 57, it is going to be odd. But now machine learning how things happen. I give the output along with it. I say that one is odd, two is even, three is odd and four is even. Got it? Now the computer based on certain statistical concepts algorithm will try to detect that when I feed the number 10 over that it is going to be even. Can it predict me as odd also? Yes, the prediction can be wrong also. Clear? But if the algorithm has to be good enough that it is predicting that 10 is even and 57 is odd. So coming on to the concept the first kind of learning is known as supervised learning. What is it known as? Supervised learning. that if this is my known data, this is already now images that this is my input and IO feed the output. When I feed the in as well as output to the machine, it becomes my labeled data, right? That means this is an image of an apple. I feed it into the machine and when I feed this apple it says predicts that is this apple or not and it predicts it's an apple but it can predict wrong also. So might have seen sometimes the chart GPT also predicts gives wrong answer the image uh development jibli and all all give wrong answers. It's the biggest fear you know that these if the if a wrong or a incapable data is uh said to uh them then it can give false and nonsensical and fabricated information also. So this is the fake fear that we are now moving ahead right AJ Char hardik but if we talk about system if we talk about the traditional systems expert systems expert systems were working that we have a user we are trying to give the query and then try to get the output out of it right if this is my query and this is how I get the output out of it and then I tried to infer what is it but inference is not coming from the data but this data it's coming from data but this data is created from an expert now what do I mean by that let me explain this to you expert can be a cardiologist a lawyer in different domains right a cardiologist a lawyer somebody from in the finance domain maybe for 30 years 30 plus years of experience and weated if else knowledge you know inference knowledge that whether you are capable of getting a loan or not and if I have a cardiologist some information has been fed that maybe your BP rating is this much or your um terms if that is matching to the inference engine that's how it will give the output so what happens for example if you are a user and you enter uh to the user interface. Maybe all your uh you know blood reports, your BPS and your test reports all things are given as the user interface right and then we do the inference engine and based on this knowledge base we get the output right. So what has been replaced now rather than knowledge base it is completely based on the original data and things have become complicated on images on you know nonstructured data. So uh do we understand supervised learning now? Can I say more the data more my system becomes intelligent is only capable of recognizing hexagon triangle and square and if I add a shape of a circle maybe a parallelogram maybe a rectangle it's capable of analyzing that. So that is where you know the systems are getting modeled because the data is becoming huge day by day right. So this is supervised learning. So under supervised learning we have the label data. What is label data? We have the input as well as the output where we try to train the model. After the model has been trained based on the test data we try to do the prediction whether it is a square or a triangle. Clear? We will initially start with supervised learning that we have the input the output we will try models. So where will we do the model? That means now the data the 70% of the data will be used for training and 20 to 30 uh you know percent will be used for testing then it will predict the output. Okay. And as I have been telling you there is a very strong relationship between machine learning as well as statistics part of it. Right? So statistics is a field of mathematics but when it is combined with computer science you know the machine learning it becomes statistics and machine learning. The idea of statistics is that it helps us to draw inferences relationship between variables whereas machine learning gives optimization prediction accuracy etc. Right? And then we have prior assumptions about the data. Some knowledge about the population usually required. This is none. Dimensionality of the data usually applied to the low uh dimensional data. And knowledge overlap there's no ML knowledge required when we study statistics. But in machine learning some statistics knowledge is usually needed as it is becomes the foundation for few algorithms. So if you do not know much about statistics there is nothing to worry. It's not very difficult. Definitely concepts of probability would be required. So I would just require request you the learners to get familiar with the concepts of probability conditional probability basian theorem and probability distribution. So this is what I expect from you all. Got it? Right. And to make the picture a little more clear, you know, the boundaries are not very crisp now because there is data everywhere. But to make the picture a little more clear that you know when we start with initially Python course, you know, we are doing visualization, exploratory data analysis, maths and statistics and when we try to overlap do overlap between AI, machine learning, deep learning, we get this data science. So data science becomes the foundation for AI, ML and deep learning. If I technically ask you what is learning and now you might have been hearing this word agent you know it could be a human agent, it could be a robotic agent, it could be an AI agent. Yes. So if we talk about a little more technical definition of learning basically we are trying to improve the behavior based on the experience. When we say you are a very learner thing you experience that means uh different types of knowledge. The range of behaviors is expanded. The agent can do more. Right? The range of behav behavior is expanded and agent can do more. The accuracy on the task is improved. The agent can do things better and the speed is improved. The agent can do things faster. So what is the idea of learning? If you see this means that DS okay so if we talk about learning from the machine as well as from the human point of view this definition is valid that it is the ability to improve our behavior based on our experience right it's not always about acquiring new skills that's one of the thing that range of behavior that you are you know driving you know swimming you know stitching you know cooking of course the range is increased but also So accuracy by doing the thing again and again doing learning increases my accuracy as well as speed. And if we now see technically in machine learning the different types of learning techniques are supervised learning, unsupervised learning, semi-supervised learning and reinforcement learning. So all these of actually learning come from the fact that how we as humans also learn. So do you also agree only agree through training and testing? Training and testing happens when we are going to school we are trained and then we are tested in universities, colleges or even this session. Is there any other way we learn also? Do we learn through observations? Do we learn through our mistakes? Do we do we learn? There are different kinds of learning also possible. And exactly the same thing we also try to replicate in our machines also. So one of the learning method is observation. So one uh basic difference between supervised learning and unsupervised learning. What do we mean by supervised and unsupervised learning? Supervised learning says that this is my data right are my apples right and I tell them that this is my image these are apples I 70% of my data is used for training and 30% of the data is used for testing label data is given in absolutely correct mega label data means supervised is learning and then the model predicts me that it is an apple and what is unsupervised learning? Have I have do I have a label data? No, I have not told that this is an apple or this is a banana or this is a peach. But the model or the algorithm, sorry guys, is capable enough to distinguish between the three of them that this is my apple, this is my peach and this is my banana. Right? So initially we will you know build concepts on this supervised as well as unsupervised learning. The machine is given huge sets of data that are not labeled as inputs to analyze. The machine needs to figure out the output its own where it identifies the patterns. And the two types of algorithm which come under unsupervised learning are association and clustering and K means for clustering problems and a priority algorithm for association role learning problems. Right? And then we have supervised learning. The input is in the form of raw data that is labeled. The machine is already fed with the required feature set to classify inputs divided into two types of problems. Regression and classification. And then we will try to understand different regression algorithm. This is the first stage that we are going to work on. Clear? So now let's understand what is selfsupervised learning. You know here we have the input data. Is it labeled? No. This is not labeled data. But we have partial label data that this is an orange and this is a banana and its quantity is less. And here both the types of data are mixed machine learning model and this is my unlabelled data to predict the output. So it's an apple. So this is my input data, right? This is my partial data. And when I combine them together, prepare the model, I get the output, clear and reinforcement learning. The most effective way of learning from which we as humans learn a lot. That means from our mistakes, from our feedbacks, from our punishments, from our rewards, right? So if this is the input given to the machine and the machine predicts that it is a mango and I give a feedback wrong, it's an apple, it notes it down. And now when I feed the apple again to the machine, it says that it is an apple. So take making the overall picture a quick recap that basically you know if we talk about machines there are three types of learnings we have the supervised learning where we have the input and output and this supervised learning is capable of calculating error that is output minus the input uh uh you know and what is the because why are we capable of calculating error in supervised learning because we have the actual output over here right and based on the actual output is my machine predicting the correct my error. So the biggest advantage or the simplest way to learn is supervised learning where we are capable of calculating the errors. Right here we have unsupervised learning where we do not know the output. It's just trying to do the clustering and association between the objects. And the other type is reinforcement learning that is not going to be part of this journey. That's generally taken into deep learning concepts that where the machine learns from its you know uh punishments and rewards right and of course that we are capable of calculating error. Again I'm telling you in this particular course we are going to try to cover supervised and unsupervised learning uh you know concepts algorithms in detail. Now moving ahead to supervised learning there are two types of supervised learning. We have regression and then we have classifification. We have regression as well as classification. Now what is the difference between the two? Please try to understand. Under supervised learning, if the output, it's all about output. If the output is numerical then this is known as regression and if the output is categorical then it is known as classification. Getting my point learners? What we are trying to achieve that what is going to be the temperature tomorrow. If it it gives me that tomorrow it is going to be 84° F or any other value whether the temperature is going to be cold or hot. This is known as categorical data. Clear? Two types of numerical data. One is discrete and other one is continuous. Do we understand that? And if we talk about categorical data, do we understand nominal and ordinal data, this is what is covered in data science class. So basically, you know, data types are divided as qualitative and quantitative. Very very important when you um divide the data as qualitative, it's categorical. Order data is something you know like rating, ranking, they come under order. Feedbacks of uh like movies. Nominal is that there is no order as such the color of the eyes or nationality. And quantitative is numerical values continuous which can be divided such as distance, salary, price and something which cannot be uh divided is example cats etc. So now I hope the concept of regression and classification is clear to everybody right. So uh making your concept more clear that regression is something when the task of predicting a continuous uh you know quantity that I want to predict the price of the house that price of the house in 2014 was this much. Then in 2024 it is this much right and then what will happen? What will be the price in 2034? Clear? So since price is a numerical quantity it comes under regression and classification as I told you that if I want to separate whether the male is a spam or not that comes under a classification problem. Clear? So now we can we will begin our journey in this machine learning through supervised learning. First we will try to complete algorithms which are like simple linear regression, multiple linear regression, polomial, support vector, decision tree, random forest. We are not covering neural network. All the others will be covered. Similarly in classification we will cover logistic K nearest neighbor support vector machine nave bias decision tree random forest. Again neural networks are not covered. Yeah. So with this uh we come to the end of the introduction and if you will now look at your u slide the ebooks that is lesson number two. Now we can begin with lesson number two. So just have prepared the foundation for that. So let me show you the lesson number two. All right. So we are now starting with this. I hope now you would be able to locate this particular file in your LMS in your material. Please look at that. So analyze the distinctions and applications of machine learning, deep learning, artificial intelligence through the real world examples of various technical applications. Differentiate among various machine learning models and explore each model learns from data to predict outcomes. Explore Python libraries for effective data manipulation, visualization, implementation and machine learning algorithm. So the business scenario says that ABC is an e-commerce company which is struggling with a surge in fraudulent transactions on its website. The manual review process for transaction has caused delays in order processing and led to negative customer experience. To address this, ABC will use machine learning algorithms to detect realtime fraudinal transactions. So machine learning algorithms are capable for detecting the fraud transaction and these algorithms will be integrated into the company's transaction processing systems to flag suspicious transactions and prevent fraud. Additionally, the company will use these algorithm to predict the customer behavior on the past purchase history thereby improving the recommendation engine's performance. Now everybody is using mixture. Everybody wants the best you know the more you learn the best output you get. And when I say best output you want the prediction to be highly accurate. So when your report goes to a machine learning or a AI machine it it has to give accurate result that whether you have whether you are you know that predicting that you know it should be accurate enough to predict that you know yes you know your you are you know uh cap you have a tendency to have cancer or not you know so no so not only one technique will be used it will try to use mixture of techniques to get the best results. Got it? Now, so now are we clear? What is machine learning? So, machine learning is a subset of AI that assist systems to learn and improve automatically from the experience without being explicitly programmed. Arthur Samuel coined the term machine learning in 1959. It enables programs to learn automatically making computers more intelligent without human intervention. But human feedback is extremely extremely important because see ultimately machines are not genius. We have to tell them that this is you know a an acceptable result or not. So who is known as the father of machine learning? It's Arthur Samuel and he coined the term machine learning in 1959. Father of AI that's John Mcathi in 1956. Yes, John Mcathi in 1956. Are we now more clear what is the difference between the traditional approach where we had the data and where we were programming explicitly the output right and the machine learning approach based on the algorithm based on the data the patterns it understands it predicts the output that is why statistical techniques algorithms are the foundation for machine learning it automatically learns the features and reduces the need of manual featuring clear. It handles complex and unstructured data such as images, text, audio without requiring extensive pre-processing. Performance improves with more data and learning iterations enhancing accuracy and generalization. Right? And now do we understand this ven diagram also that machine learning deep learning AI are often used interchangeably. So AI encompass encompasses simulation of human intelligence and machine. So self-driving cars are you know AI all the robotics come under the category of AI but machine learning Amazon Alexa where we are giving it specific instructions and it gives us output. And when I talk about deep learning, deep learning involves neural network which is going to be your next stage after uh you know machine learning to understand how neural network uh you know work. How are they capable of understanding complex pattern recognition such as recognizing patterns in images, speech, text etc. So where are what are the examples of machine learning in a chess game between a computer and a person? Why do you think that? Why is this chess game always coming into the picture? Why do you think is the chess game always coming into the picture? Well, when we talk about human beings, a person who plays uh chess well is said to be intelligent. It's an intelligent game. Agreed. It's said to be intelligent. And even if the results go wrong, there's no harm. There are no catastrophic results. you know even if the person is winning and says the machine is winning it is not that harmful. Yeah. So that is why you know that that is was one of the way where AI exploded in you know intelligent gaming systems theorem solving that is why chess alph you know read or see when we talk about a little history of AI. So in a chess game uh between a computer and a person, the computer uses AI to analyze the game, predict moves, decide its decision. AI uses machine learning to figure out the opponent is a beginner, intermediate, and an advanced level. How the AI uses machine learning to identify whether you are a beginner, intermediate, or an advanced player? By predicting our moves, you know, based on our moves, it will immediately judge, you know, immediately judge that you are a beginner or a intermediate or an advanced level, right? And you might be playing a lot of games where there is AI and other, you know, graphics games over there, you know, especially the young generation, right? Well, I don't, but you can, you know, how smartly, you know, they are capable of hiding things and they become smarter. The level changes you know as you are also becoming smart the level of the game becomes smarter right we all observe that so AI decides its next move against the oppo opponent using a complex neural network that learns various features patterns from the data right then of course many applications are there we see machine learning all around us in spam filtering spam filtering actually use Live based classifier, social media analysis, customer service, chat bots, now a lot of available online recommendation. We all see that sentiment analysis. How does the sentiment analysis happen? Anybody who has an idea based on the words and especially the emojis. Is it a smiley? Is it a angry face? Is it a sad face? Crying face. All these things are taken into account. So these processes allow computers to learn patterns from the data, make predictions, predict outcomes, classify target feature and improve performance. So that's what uh we want machine learning that they help to predict the outcome, what is going to be the price of the house or any other thing after 10 years down the line. classify target features based on the similarities and improve the overall performance of the system. And we have seen is there a very very strong relationship between the data and the output. Yes, the amount of data if it is more of course the output or the quality or the prediction also increases where the red line is uh the quantity and if if it is high quality data we get better results better insights and better predictions of the output. So of course maintaining the quality, authenticity and removing the errors. All these points have to be taken into account when we are talking about the data and the machine learning algorithms. Clear? Right. So when we talk about types of machine learning, ML can be divided into four main categories each characterized by its capacity to predict the our conditions or identify the patterns to produce outcomes such as what is supervised learning, unsupervised learning, semi-supervised learning and reinforcement learning. Now I think the distinction is clear. Supervised learning, we've done this. What are the three main points under supervised learning? First is the label data. What do we mean by label data? It will have the input as well as the output. Second point is splitting of the data into training and testing. Right? Generally training happens on most of the data and testing of on the rest of the data. And third is calculation of the error because in supervised learning we know the actual output also and whether it is predicting it right or wrong. Clear? And some commonly known uh you know uh supervised learning algorithms are linear regression, decision trees, logistic regression, support vector machines etc. So some examples of supervised learning are predicting temperature rise based on the yearly temperature trends. Predicting why supervised learning again predicting temperature rise it comes under regression because it's a numerical problem right numerical output predicting crop yield based on the seasonal crop quality changes. Again regression sorting waste based on the known waste items corresponding to the waste type. This types comes under classification or filtering. So under supervised learning calculation of error is also going to be an important aspect of machine learning because we have to be very clear that the output of machine learning will always not be 100% correct. Clear? Now this example are these example making the picture more clear. And if we talk about unsupervised learning very much used to I you know uh identify different parts of the object image segmentation for object detection. Identific identification of user groups based on commonalities. Identification of anomalies over geographical landscapes based on the data patterns. The unlabelled data set is provided to an unsupervised learning algorithm to discover hidden patterns and to recognize their relationship. So it's not that unsupervised learning is not important. It is equally important to analyze different features, different relationships in the data. Right? Rather this is the algorithm which helps us to discover the hidden patterns and discover the relationships. Clear? And now coming on to the unsupervised learning example. It automatically groups the images based on the similarity that this is unlabelled data. Based on that it is capable of distinguishing the middleage people, old age people, the young, the infants, the teenage etc. Got it. And what is semi-supervised learning? As I've already told you, it uses a combination of small amount of label data. Sometimes, you know, the data is that's one of the constraints that we see in data is not completely labeled. A large amount of unlabelled data is used for training. Like supervised learning, it aims to learn from a function that can accurately predict the output variable from the input variable. It uses the unlabelled input to assist the learning process by collecting more information improving model generalization. So it falls between supervised and unsupervised learning. So suppose this is my raw data and I have partial that this is adults and these are kids then the machine automatically distinguishes between babies, teens and tween and also distinguishes between the senior citizens, youth and adults. Got it? So it automatically learns the correct uh you know uh groupings of the kids uh or the different people into teens uh tween and babies and the adult ones into into these category clear and another example of semi-supervised learning which we see it practically that was your question Mega that Google photos is popular example of semi-supervised learn learning that when a picture is taken, it gets stored in the Google cloud platform and from slowly the Google tracks you know whose picture it is at what place it was taken. So in various instances uploaders label images despite Google's lack of knowledge regarding image names its algorithm can identifying by analyzing visual features and shapes and colors and it does that it does a lot of lot for me it is able to it's able to identify my friends my family in which location I was and where and reinforcement learning is a type of machine learning where algorithms learn from the environment by performing actions and receiving either rewards or penalties as feedback. If the pro program finds correct solution, the interpreter rewards the algorithm. If the outcome is incorrect, the algorithm is penalized for incorrect predictions. It must I reiterate until it finds a better result. Right? So ultimately reinforcement learning involves an agent. It interacts with an environment learning from the rewards and states to choose from and then based on the output it gives the best action and if it is an error it learns it again. All right. So this is my input raw data based on the environment reward state and action it's capable of detecting them separately. Okay. So the example is this type of learning is seen in YouTube recommendation where the user searches for a particular song the program shows the list of available song. So when a user selects a specific song the system trains itself to remember and deliver similar results for future s searches based on the user's interaction like views, shares etc. So this is the concept on which recommendation systems work. So other examples of reinforcement learning are game where players can play with bots, order correct tools, search recommendation income uh engines, self-driving cars and then the Python packages that we would be doing for machine learning. We are aware about NumPy. It's a very powerful tool for numerical Python computing. Mattplot lip for drawing data visualization pandas. So I hope you all are aware about numpy mattplot lip pandas. But we will be working more on the scikitlearn uh uh you know uh file which consists of different algorithm but the pre-processing the other part is also being taken care before we feed in into the algorithm. So a quick recap. So machine learning refers to the machine's ability to learn from the data and replicate human behavior. AI includes machine learning, deep learning each with unique capabilities for simulating intelligence. There are four main types of machine learning. Supervised, unsupervised, semi-supervised, and reinforcement learning. Python packages are folders with modules that organize code for easy reuse and maintainance, improving the de development efficiency. Now, clear? So, now let's go in for a knowledge check. Question number one. Yes, learners, are you there? Which of the following best describes the machine learning? A, B, C, and D. Question number two, which example illustrates the use of machine learning to enhance customer experience in an e-commerce company? What distinguishes between deep learning, machine learning, and AI? Yes, it is a subset of ML that uses multiple layers for complex pattern recognition such as recognizing patterns in images, speech, and text. Right? So with this we come to the end of the very introduction and basics of machine learning. Yeah. So what I'm looking forward is this is the overall uh picture of statistics that uh we have. The learners who have already done data science are aware about it. The ones who are not aware about it. The different types of statistics that we have is descriptive. Under descriptive we have measures of central tendency and measures of variability. Under measures of central tendency we have the mean, mode and median. Under measure of variability we have the range, variance and dispersion. And here we have the inferial statistics how we infer the results. So this is the important part that we are looking at that that includes confidence interval, hypothesis testing. So you can do a lot of search on inferial statistics and uh this is what I am saying that uh statistical inference constructing confidence and intervals on population hypothesis testing that is what I'm looking and what is the advantages of these particular uh program because ultimately you know probability in data science and AI play a very very major role in understanding uncertain certainty predicting outcomes how probable correct the output is even the LLM models the chart GPT is predicting on the probability okay this is the next word so if it is 70% above then let's predict the outward how do we model complex system enhancing AI and it is the statistics and the probability together which help in exploratory data analysis and give meaningful full insights and features. So as we are going through this flow we understand we are going to start with supervised learning. What is supervised learning? Before supervised learning you know uh I would like you to cover we would uh you know the cover the basic concepts of machine learning which I have prepared through my PPT and then we will move on to what they have shared. Okay. So the first thing is regression. You know what are the two main algorithms which come under supervised learning? Yeah, regression and classification. What is the difference between regression and classification? Regression happens when the output is numerical and classification happens when the output is categorical. All right. So I start with my PPT again so that it helps and it gives you strong foundation clear concepts so that we can move along with that. So if we talk about the first knowledge check what is machine learning? Correct answer it is see that it is an autonomous acquisition of knowledge through the use through the use of computer programs. Second question, what is the key difference between supervised and unsupervised learning? A and B are both correct. What's the key benefit of using deep learning for task like rec? How do I u Okay, you've written yes. Okay. What's the key benefit of using deep learning for task like recognizing images? Yeah, they can learn from complex details from data on the own. That's the idea of deep learning. That's the use of neural network. Absolutely correct. All right. So now we start with the concepts of machine learning under supervised learning which are valid for regression and classification algorithms. Whenever you will say you know machine learning these are the basic questions that will be asked right. So if I talk about uh machine learning or supervised learning what are we trying to uh you know do in supervised learning what is the main aim what is our main objective of supervised learning if but prediction of the data. Okay, we would call it prediction of the data future based on given data. Right? We want to predict and is this prediction always correct or it can be wrong. Can be wrong. They can be errors. Right. Right. And so they there should be a limit of accepting the errors or rejecting the result. Right? There should be some way of accepting and rejecting the results. We all understand this very in a conceptual subjective matter. Now let's try to understand it on the basis of mathematical functions. Right? So here we have the plus over here and here we have the minus over here. Right? This is my data set. This is my x-axis or and this is my yaxis. These are given as my data. Right? And what are we trying to do over here? We are going we are trying to predict the output. So what are we trying to predict the data that what is the value of this question mark? What is the value of this question mark? All right. And what is the value of this question mark? Right? Is this question mark a plus sign or a minus sign? Can you tell me what is this question mark? A plus or a minus sign? What about this? What about this? So this is my first question mark. Second question mark. Third question mark and fourth. So based on your observation, can you tell me what is this first? Do you think it is a plus or a minus sign? What does this particular data point represent? Is it a plus or a minus? Because this this particular data point is more close to the plus. So there are chances that this is going to be plus sign more chances. Yes, it can be negative also but we can say 70 to 75% or 90 to 95% chances are that this is going to be plus. What about the fourth one? This is negative. Now what about the second and the third one? What about the second? What about the second and the third? This is going to be a little difficult to say that this is going this can be plus this can be minus based on the way the neighbors I am selecting what will be my output agreed but but if technically now if I use a straight line function ma mathematically how do we use a straight line function that y is equal to mx + c. So any all the points which lie on the left side of this line they all are known as plus they all will come under the category of plus and the data points. Now all the points which are lying on the right side of the line they will be all negative because most of the points are minus or the red points. And I say that all the points which are lying on the left side are plus and all the points which are lying on the right right side are negative. There are very few now this is there is only one plus sign and if you see all the points belong to the red red negative class one way I have to do something but again the question is why this straight line that how do I decide the straight line? So one point to understand is that when data points are given that is known as a hypothesis space represented by capital H and the line is represented by small H right and it is going to be this red line which is going to divide the data points into two different classes. So there can be more than one solutions to a problem or infinite solutions. I can draw infinite straight lines. But which one to accept? The line which will be accepted is going to be one with gives me the minimum error. Right? The error word will be different for different algorithms. But I will accept that straight line which will give me the minimum error or I can say the maximum accuracy. Clear? We will calculate as I told you in supervised learning we keep the track how we can have maximum accuracy and minimum error. So what is the capital H over here? Hypothesis space is the set of all possible legal hypothesis. This is the set from which the machine learning algorithm would determine the best possible only one which would describe the target function and small h is a hypothesis function that best describes the target in supervised learning algorithm. So now technically what does supervised learning mean that the hypothesis or the small edge there can be several small edge that an algorithm would come up depends upon the data also depends upon the restrictions bias that we have imposed on the data. So next thing is we need to find out mathematical functions which give relationships between the data points. Right? So basically now the idea is that we want to find a mathematical function where you know uh we want the error to be minimum or the result to have maximum accuracy. Again I'm repeating we want to find a simple mathematical function which gives me minimum error or maximum accuracy. Clear? Thank you at Thank you for understanding. Thank you learners. So now what are the different stages that we are looking at? Now suppose data points can be of any kind. So if this is my data points over here and if I want to solve it with with a simple linear function see straight line function is a simple mathematical form uh you know calculation that y is equal to mx + c. Okay. So when y is equal to mx + c, right? And if these are my data points. Now how do I calculate my error? This is going to be actual minus the predicted. It will have very high error. And this particular concept is known as underfitting. Do you think that this straight line is consistent covering all the data points? No. Right? So the simple straight line equation or function is not capable of covering all the data points that is known as underfitting. Other way is that if I draw a function which passes through all the data points so it makes it a very very highly complicated mathematical function with high degree. Right? So it makes a complicated function with highderee mathematical function that also we don't want. Why we don't want because if the data point is out of all the data points then this is going to give me my maximum error. So we want a mathematical function which is simple and which covers all my data points. So something like this exponential right and this gives me a good fit or a good balance now clear. So AJ says no. So the model even fails to predict labels what it learned right it will it will predict the labels but with very low accuracy the error will be high. So we we will not accept models with high errors or low accuracy. So overfitting is a situation. How do I know the error is high overfitting? So one way is that error is equal to if I talk in technical terms. So how do we know that it it is underfitting overfitting? So that's what I'm telling you. Generally the error is the formula for error is bias²ared plus variance. What is it equal to? It is bias squared plus variance. But the equation is in good balance not linear. Yeah it it it's not necessary that it has to be a linear equation. No, we want an equation which is simple and nice. So how do we know? So now if I say that error is equal to bias squared plus variance over here right and this is you know so how do I know whether it's overfitting or underfitting it's not about the straight line but when I have variance if if the variance of the data goes very high then I know that it is an overfitting model if the bias of my data goes very high I know it's an underfitting model and to keep my error low I want the bias as well as variance to be low. Getting my point right now what do we mean by bias and variance? A very very typical example. A bias means how far are we away from the original data points. So bias means we are close to the center. Variance what does variance mean? The spread of the data is also low. So this is the ideal situation we always want the data points to be to be in. This is where we will say okay a good fit has been achieved where we have the low bias and low variance. Okay. And when we have the low bias and when we have the high variance low bias means that the data points are near to my actual target points but spread out. So what is the case happens over here? This is known as the overfitting case. And what is high bias and low variance that over here these are my data points and this is high bias. So this is underfitting situation. So in whole of machine learning we don't want underfitting we don't want overfitting. Of course this is a worst case where the bias is more variance is both. The idea is to have low bias as well as low variance. Clear? Okay. Now pre I'll explain the previous example with this explanation after this punit I think. So this will make things more clear. So these are my data points over here a linear model is not a good fit. Under fit that means over here what is more my bias is more. What is bias? That my data points are very far away from the actual points. Bias means we are far away from the actual points. Right? This is high bias. And what is variance? Variance is when the spread of the data is more. Right? And when spread of the data is more or mathematical function is too complex then it is overfitting. When my mathematical function is too simple, it is underfitting and when it is a balance, it gives me the right data. So now when I talk about complexity, now when I say overfitting means more complexity that generally linear models give me underfitting, right? We start with simple models but generally they do underfitting. Then we will move on to nonlinear models, support vector machines, treebased models, deep learning models. Why are we in doing this? Because it they give me more results, better results, more accurate result, less of error. But what is the cost that I my model has become complicated. I'm losing the interpretability of the model. What do I mean by interpretability? That if this is my input, right? How do I know that this is going to be my output? That's possible in linear model because I know y is equal to mx plus c that if this is my input, I will get this output otherwise I will not. Clear? And to make it more clear you know the overall picture that I was talking about please look at this slide. So ultimately where is the trade-off? Where are we fighting in the whole of machine learning models? If life was so easier wouldn't have the problems be solved by now but no there is still a trade tradeoff which is going so how will I know it is underfitting though it is showing in diagram is there any yes yes they are mathematical way but first let's get the grasp of the thing okay that ultimately the idea is to get highest accuracy right and interpretability is also important Why? Because we want to see why am I getting this output. So we always start with linear regression or logistic regression problems. That is what we do. That is what we are going to do in our uh goal of this journey. And why uh regression linear regression? Because they are linear and smooth well definfined relationship easier to compute. And as we move along this journey, decision trees provide good accuracy with high interpretability. KNN clustering clustering and KN&N algorithm are mid-range. Interpretability is okay and accuracy is also okay. But if I want really good accurate results that is why new algorithms complex algorithms were built with kernel based approach for support vector machines onsembled methods and of course neural networks now clear is this slide getting clear to everybody? That is why complicated neural networks are capable of solving nonlinear relationships, non smooth relationships and long computation time. Clear? Okay. If I want interpretability of the models then I'll might go in for decision trees but I will have to compromise on my accuracy. But if I am not interested in interpretability that we are not. We always want best result. That is why neural networks, deep learning have taken the market. It is we as users who will decide. What is the difference between accuracy and interpretability? Interpretability is how am I going to find out the relationship of output given a particular input. So if I know a mathematical equation okay this is related with beta x1 beta_2 x1 I know why am I getting the result but if it is some integrable model like neural networks working in layers in different layers some in some layer differentiation integration addition subtraction is happening I will not be able to interpret the result but it is giving me highest result then of course I will use it if I have a very mathematical complication complicated ated system. See, now try to understand the whole story. Again, let me just uh repeat the story because this is very important in the whole of machine learning and these concepts are valid for deep learning also. Okay, the story says that we want to predict the output or the data. You're clear. The first point says we want to predict the output of the data with minimum error or maximum accuracy. First two points are clear. The the threshold Simon is maximum accuracy and minimum error. That point is clear right. And which is the simplest mathematical function that we have? We always talk about a straight line. If that straight line is giving me minimum error, is it so is it that only we will always apply one algorithm or is it a different algorithms or a different hit and trial methods that we have to apply? It is different hit and trial methods algorithm that we will try to get the output. It's not just you know oh god regression and decision tree is working okay this is the end of it. No, you might go in for advanced algorithms and that's where the research is going. Why do you think that the problems are not solved? Because every time every algorithm will have its own pros and cons and the research is going at each NLE level and it's still going you know we are just trying to improve on the algorithms every time. Why? Because we still see the LLMs the chat GPT giving us wrong answers wrong predictions. So there is still a lot of scope of incre improvement you know where we have to work clear. So life was if the thing was so easy I think so by now everything would have been solved by AI and machine learning algorithm. No. Why is it not simple? First data is variable uncertain. It keeps on changing. Secondly every algorithm has its pros and cons. So the errors the accuracy the the the my requirements upon the product you know on the data or the algorithm keeps on changing and that is why this change happens right. So they are pilotra of algorithms which have been built. As in machine learning we will always start with linear regression algorithms because of the simple mathematical function and high interpretability but they have minimum accuracy. But as we move along with the journey of decision trees clustering kernel based we have we want maximum accuracy. But the trade-off is that we decrease on the interpretability of the model. Right? So as I have been telling you splitting the data for machine learning in supervised learning, we always and always want to select the data randomly. So there is a function train test_plit that we would be using to split the data. train test_split which is part of the skarn library. This is my data and we will be randomly selecting it and dividing into training and testing data. Now clear and most of you you see this is around 70 to 80% of the data and this is around 30 to 20% of the data. Punit says just wondering if interpretability is more important as we go to complex problems. Not really. Not really. you know ultimately punit uh not really because we are always interested in the final result it's something like as I give my example suppose you know you have a magician and you know the magician turns a pigeon into a flower are we interested how does how does it does or are we interested in the output as a flower okay so we are always interested in the output got it now better yeah so if logist istic regression is having minimum accuracy then it is a no no the accuracy is less over here it doesn't mean that it's a wrong prediction logistic regression might not give you the best results for the prediction and maybe using a neural network would give you you know it's something like n uh you know using logistic regression is giving you 70% of accuracy and neural network is giving you 99 97%. So that's the difference but of course neural networks are better that means right got it so yeah there are a lot of terms which you have to be associate you have to understand it in the correct sense I understand I'm giving you time for doing that yeah now coming on to the part that you all are interested that ma'am what is the mathematical equation you know how do we know the mathematical term which is used to see the output. Now first of all we are going to calculate error right in regression the error is known as mean square error. Now let's start understanding this. So we have this mean square error. Okay the error name is that that I'll explain you technically when we move on to regression. Now in supervised learning whether it's classification or regression we we know the data is divided into training and testing. Now tell me which error is more important. Now let me put it in this way that you go to a class you know and you are pursuing some course you know you you you do errors you are learning and you do errors while training right and still you don't perform well in the test and there is somebody who doesn't attend even one single class and performs well in the test. So which error is more important? It is the training error which is important or the testing error. Among the two of course both are important but among the two which one is important. So what is more important to have the training error less or the testing error less? The training error is less but the testing error is very high. Is it a good thing? It's a even big failure. So again I'm asking you this question is the training error important or the testing error important? Of course the final test if you're not performing well on the final day of the test then the whole training is useless. Agreed? So when my training MSSE is high, of course I can reduce my training by giving in more data, more data and my testing MSE is less, what is this case? This is the case of underfitting. Of course, the training error should not be high than the testing MSE. The model is too simple for it to solve and then I will say the model is underfitting. Now clear that was the question I think so so now clear mathematically so this is how I will I will judge based on my training and uh and testing that my model is underfitting so I will now not use a straight line to solve my data points I will not use okay other is my training MSE is zero excellent but my testing MSE is so high absolutely not acceptable case. So this is the case of overfitting which is high variance. And what do you think the data or the model goes through? Is it more of underfitting problem or an overfitting problem? What do you think most of the models and machine learning face which kind of problem? Reducing the training error is easy but failing on the testing is a total failure and that is what we will will we will look at most of the algorithms that how to prevent overfitting in the models that is why I was telling you these concepts are very very common to machine learning deep learning models so are you getting a grasp of it and I don't straight away start with regression come on let's start doing at other things. You have to have good foundation of machine learning concepts so that you know you understand other materials the indepth knowledge how we accept and reject the particular model. Training and testing that we will decide because when the data is there we will split the data into training and testing and then we will calculate the training error separately and the testing error separately. Okay. We will have the suppose I said 1,000 rows 700 rows will be used for training then we will have uh 300 rows used for testing then calculate the errors for training and testing separately and then decide whether it's underfitting or overfitting. So what will be the best case where my training error is also low, testing error is also low or almost equal to each other. So again looking at the concept xaxis we have the model complexity and then we have the predictive error. So over here on the x-axis this is my underfitting over here and if what is overfitting that when I start with my model simple model my errors are high both the bias and the both the test and as well as the training as my training increases the training error or the model complexity increases the training error decreases but the test error will dip. and then increase because of high variance. So we want to find out a model. How do I decide which model is best for my data where the training and the testing error are minimum or close to each other. So this is what I'm trying to explain you. Please try to understand that as the model complexity is less here we have the bias more as the complexity increases the bias decreases. When the interpretability is high the variance is low and as the complexity increases the variance increases but we are looking at a point where these two inter intersect to find the optimal model complexity. The variance is the spread of the data. Variance is a statistical term which means the spread of the data. So as my model complexity is going to increase the the the complexity as the complexity of the model increases the variance or the spread of the data also increases. Got it? That's the relationship. Again I am telling you we are trying to create a graph between error or accuracy is going to be opposite of this and this is my model complexity. We always start with minimum complex model. This is the case of underfitting. Agreed. when my model is simple but my bias is high as the complexity of the model increases my bias decreases but my variance increases so I don't want the overfitting situation or the underfitting situation I am looking at a situation where my error is minimum so the test error is U-shaped in curve and where the value is minimum that becomes comes my optimal model complexity. Now clear to everyone? Okay. Now uh okay now since the uh topic has been taken now let's do one thing. If you have downloaded the uh ebooks material I want everybody to make a folder in the desktop and move under lesson number three. something like this. See what I would see is let me share my screen in work in data brick. How do I understand that? So once you are into now we starting with lesson number three. So do you see this 3.1 and 3.2 learners? So how do you go about opening it? Open the Jupiter notebook environments, install and open the file along with me. So chapter number three, we have to start with lesson three. Lesson three is supervised learning regression and its application. Yeah. Are we all able to open this? Let me guide them that how do we go about it? See the prerequisite for the course I've been telling it is Python. So we always we like there are lot of tools like PyCharm, VS Code, you can do it through uh other tools but generally we use Anaconda Navigator download. Please go ahead and go to the Anaconda, fill in your uh like the details, the email id and I want everybody to download the Anaconda right and uh based on your uh system whether it's Mac, Linux or Windows and go in for a full distribution. Don't go for mini Google Collab. If you're aware about Google Collab, that's also one of the online tools u available you know which helps you to run the code in the Jupy Jupiter kind of environment. Let me show it to you. Anaconda distribution. Go and download it. Install it. Or the other one is go to Google Collab over here. Just login through your Google account. That's the another uh way to go about it. And from here I will upload the file. Which file? The file which I have already downloaded on my ebooks. That's there on my desktop. machine learning third chapter 3.1 and I open it. Yeah. So there are two three methods whichever one you are comfortable with that's not an issue. If you clear you're comfortable Anush on VS code that's not an issue. Got it? Now everybody is there with me now. So this is one of the very safest tool. You don't need to uh you know download the Anaconda on your desktop through the Google Collab through your good internet connection. You will be able to run the code. Yeah. So this particular file is uh you know basically if I talk about 3.1 again I'm repeating go on to your learning management system on the reference material please download the ebooks the Jupyita notebooks and then try to open. So if we talk about if we are back to 3.1 let's let's quickly go through the file that file is completely theoretical okay nothing great in that file it talks about what supervised learning is and what are the two types of algorithms in supervised learning it's regression and classification what is the difference between regression and classification and supervised learning in regression the output is always numeric IC and classification output is always categorical. So when the target variable is categorical we do classification. So this example we are very clear. So when it's predicting numerical value right that is regression. When we are predicting a categorical outcome determining the whether tomorrow it is going to be hot or cold. visualization. The same thermometer scale divided into two colored regions cold and red for hot where the threshold separating is classification. And if you look at the applications of supervised learning, do you think HR people use machine learning to um uh identify the different job profiles? Uh shortlisting of the rumés. HR is also using machine learning now. Yeah. In finance, what is the use of machine learning and predicting of fraudulent data is are you know ready for that loan approval or not. Right. And um this is similar to how to how a credit card company determines your credit worthiness issuing before issuing a card. Then emails this is about spam filtering whether it's a spam or not. manufacturing. Anybody in manufacturing? So in manufacturing, supervised learning is also used to inspect the quality, classify the products into different grades. For example, a factory might use a machine learning model to check for defects in products and ensure they meet quality standards. Much like the quality control inspector, maritime industry, it helps in forecasting the historical events, weather condition. So precautionary accident it's capable of predicting any kind of you know weather which might get wrong also which we have seen and supervised learning techniques like regression model can be used to predict tidal currents forecast demand and supply reducing inventory losses think of it as how weather forecast predicts rain based on past weather patterns broad predictions we use this in agriculture field too I'm into automating parking so under supervised algorithm you know there are lot of algorithms that we will begin tomorrow again as you understand my style we'll start with regression then linear regression logistic nave bias k&n under classification they all come under classification so under linear regression we would be working on multiple linear regression and before we move on to classification there are lot of concepts that we will study such as excuse Okay, excuse me. Cross validation, regularization, hyperparameter tuning, skarn pipelines. So there is lot more to be explored tomorrow. We will be concentrating on linear regression and its concepts. Tomorrow, right? And linear regression is now very clear. Whether output is numerical, then like predicting housing pricing, we will use regression. After doing supervised uh regression in supervised learning then we will move on to classification algorithm. Under classification pelra of algorithms that need to be done initially we'll start with logistic regression, knive bias, KN&N decision trees, random forest and support vector machines. So it's a long journey. Tomorrow we'll do linear regression. Next weekend we'll we will be doing classification and then we will move on to the next stages of ensemble learning. then it is uh unsupervised learning right so I think so you all are excited about this journey so it's a long journey I don't want to burden you today with all the concepts so let's go slow and steady that's my rule slow and steady wins the race but I hope you got the crux of today's class right so this is 3.1 right and um we'll be beginning with uh 3.2 two. >> We started with the very first session of machine learning where we understood what machine learning is, what are the different types of learning techniques, what are the different types of m machine learning techniques, supervised learning, unsupervised learning and reinforcement learning. What is supervised learning? Supervised learning means it consists of label data which consists of input and output. We split calculate the errors. The two main algorithms under supervised learning are regression and classification. Then we stood unsupervised learning. Unsupervised learning consist of unsupervised learning consist of unlabelled data where based on the similarity of the data the groups and clusters are formed. So two main algorithms are clustering and association. And last but not the least is reinforcement learning that based on the feedback positive and negative the machine learns. Are we good to go? Everybody is clear with these concepts of machine learning the basic concepts and then we started with the concepts of overfitting, underfitting and a good fit. Yes learners now you will tell me what do we mean by underfitting fitting in machine learning very very important question it will definitely be asked if you say that you know the concept of underfitting and overfitting. Yes learners can you tell me what do we mean by underfitting and overfitting? Underfitting is the case where the error is high. What error is high? Bias is high. We are very far from the actual data point. But we talk about simple mathematical function. What is the advantage of simple mathematical function? That it gives us more interpretability of the model. What do we mean by more interpretability? That it gives the relationship between the output. And if we talk about overfitting again the error becomes high because the variance goes very high, right? and definitely it makes and more complex. So we don't want a algorithm to be or model to be underfitted or overfitted. We want a good balance fit. That is how do we judge that that the training error and the testing error should be similar. Right? So today we will start with the first uh you know learning technique that is supervised learning technique and the regression model. Are we good to go? So as now you are getting more familiar with my style you know first we will try to cover the concepts and then move on to the practical aspect because unless and until you what is happening how is how are we going to interpret the model there is no point moving on to the code but definitely we'll move on to the code but let's start with the concept of regression so here we go with the regression so what is regression now if I ask you what is regression? Tell me. So when we're talking about regression learners, please be clear with this concept that always the target variable or the dependent variable are all numeric values or continuous value. Don't say the term data. Right? See everything is data. But if you're not very spec technically you go very wrong right so you have to understand now data we're talking about structured data which cons in tabular form it consists of input as well as output values right whatever are the input or the output value x over here is referring to your independent variables right but whenever we are choosing any model or algorithm we always and always take into consideration the dependent variable. So it is very important that the dependent variable has to be numerical or continuous right. So we are trying to find out so whole so basically what are we trying to find out we are trying to find out the relationship between the independent variable based on the output. Clear? So let's try to break the term linear and regression. Linear we understand from simple mathematics that anything of degree one any function which has value one is set to be linear. So this means that progressing from one stage to another in a single series of steps sequential extending along a straight line or nearly a straight line. So we all understand the term linear right and as very rightly said by nanda regression is a statistical terminology which is used to find out the relationship between one dependent variable. we always have one output. If the output is categorical, we will go in for classification algorithm. And if the outputs are continuous or numerical, we will go in for linear or not linear but regression algorithms. So moving ahead does a linear rel regression means a linear approach for so when I combine the two terms linear regression it means that the linear approach for modeling the relationship between the dependent and independent quantitative variables. It will plot a straight line because that's the simplest linear uh approach as a best fit along the data points to predict the target value. Right? So what are the different types of linear regression available to us? So a simple linear regression is with one depend obviously the dependent variable is always one that's the target or the output but if the independent variable is one it is simple linear regression and it is always represented by a straight line. So beta kn is the intercept and this refers to the slope. So this refers to beta kn plus beta 1 x1. getting my point? And then we have multiple linear regression that is over here when independent variable then this straight line turns into a hyper plane. So line in higher dimension two dimension becomes a plane and in higher dimension it becomes a hyper plane. Getting my point? Are you all getting the concept mathematical equation along with the graphical view and of course with the code try to I'm trying to explain you from all aspects all right and when I say polomial sure I can do that right over here we are talking about linear regression linear means something of degree one so do you see all the variables have all the coefficients have the degree E1. Do you see this? Right. These are known as the regression. So the term linear regression means we are trying to find out relationship between the independent uh variables with the dependent variable. So a simple linear regression is beta KN plus beta 1 X1. Clear? That is when we have one input uh variable and one output variable. Mathematically it is represented by a straight line where beta kn represents the intercept of the line and sorry the beta kn represents the intercept and beta 1 represents the slope of the line. Clear? Have multiple linear regression. Graphically a straight line now becomes a hyper plane that is a multiple linear regression. Of course you know generally practically we will have more than one independent variables with output variables. And then we have polomial linear regression that is y is equal to beta kn beta 1 x1 beta 1 x² and q. Clear now? So now let's try to understand it from now the same thing from the machine learning point of view. So in simple terms it is a finding a ba best straight line fitting the given data set. So in simple terms find a simple straight line which tries to find the relationship between the independent and the dependent variable and best linear relationship. How do I decide the term best? Something with minimum error or maximum accuracy. Right? Something with minimum error or maximum accuracy is taken in terms of simple terms. And if I talk about technical terms, it is a supervised machine learning algorithm that finds the best fit relationship on the given data set between independent and dependent variable. almost the same thing but the basic concept or the algorithm it works on it is OS the ordinary le squared method also known as the sum of the square residuals clear so to explain these terms graphically mathematically I want everybody to concentrate here on the graph or the slid share so this is an equation of a straight line. What is beta KN? Beta KN is the intercept. What do I mean by intercept? What is the value of y when x is = 0? Are you all getting this point? What does the term intercept mean? That what is the value of y when x is equal to zero? Clear? Is this point getting clear to everyone? And if I talk about beta 1, beta 1 refers to the slope of line. Is this term getting clear to everybody? Since I have only one input variable or independent variable, it represents a straight line. Clear? Now once we have understood the equation of the straight line, what do these blue data point uh blue dots represent? the actual data points available right that the first data point has value one y is equal to three this is value four and value is equal to six all right now this is where how we are going to calculate the error what is the error the actual value this you can consider this y i as my actual value and the red dot on the straight line is the predicted value if I want to fit a straight line on these data points. Right? So this is my actual output and this is my predicted output. Clear? The error is clear. How are we calculating the error? Now what algorithm are we technically using over here? ordinary le square method or the residual. Residual means the error sum of square error. So what does this mean? It is equal to the sum of the square of error which is actual minus predicted value. Getting my point? So and why are we taking the square of the error? There is a story behind that also. Why? Because is + two for example and this error is minus2. So the total error is zero. So what I'm trying to say is if the error is plus over here and minus and if I add them it will show me zero error. It does it is not zero error. So I will try to take the square of the value it will become four + 4 that is the error is equal to 8. So always the errors are taken as square of errors. And we are looking at minimum square of errors. That is why it is known as the ordinary le square method. We want the sum of the squared error to be the least. This is known as the RSS. Getting my point learners? So this is the total RSS. by the total for this particular data point. This is going to be my first error. For this data point, my second error for the third one, fourth one, fifth one. And if they are infinite points, then infinite data points are calculated. Clear to everybody? So why y Okay. Now because this is my actual y1 and what is my predicted y1? This is my predicted value right base because we are using nanda this particular function. This is my predicted value and predicted values are represented as ycap. So for the first one it will become beta minus beta 1 x1 then y2 beta beta 1 x2 for each data points. Now clear this is how based on this mathematical function I am going to predict my output for each data points. Clear? So the formal statement of simple linear regression I'm talking about simple linear regression with one input and one output. Y1 is the value of the response variable. Again, predict, target, predicted, you know, dependent, they all mean the same thing. Do not get confused with the terminologies in the IAT trial. Beta kn and beta 1 are the parameters. Xi is the value of the predictor variable in the IAT trial and epsylent I is the random error with mean E. epsylon i is equal to zero and variance is equal to square. Clear? Okay. Now this random error this is very very important to understood that a random error we are talking about are we talking about the whole data set or a part of data set or a sample of data set over here. This is my population and if I take a part of it for training and as well as testing do I always consider the whole data set or a part of data set it's always sample and when I and is my sample and we want the sample to be the true representative of the population but since it is a part of the data there will be some error also uh you know associated with it which is irreducible that is represent resented by epsylon right so epsylent I is the random error with mean this is the random error which is always going to be introduced it's it's not reducible like bias and variance or overfitting but it's part of the whole linear regression where the mean of that error is equal to zero and the variance is equal to epsylon square now clear so The idea of whole uh linear regression or significance model is that it is highly interpretable. What is the idea behind this? That it is why the error is zero. We always want the error to be zero. In this case we are considering that uh you know the average of the error not the square the the sum of the square or the average of the error is equal to zero not the square. Okay. So now the significance of linear regression lies in the fact that we can easily interpret and understand the marginal changes with with with input to the output. So linear regression is an highly interpretable model. How is it interpretable? That if we increase the value of X1 by one unit keeping the other variables constant, then the total increase in the value of Y will be beta 1 and the intercept term beta KN is the response when all the predictor terms are set to zero and are not considered. Right. So how much is my y value change will depend on how much is my beta 1 is it positive or negative beta_2 and beta n. So now what are we trying to find out in linear regression the values of beta kn beta 1 and beta n right where the error is minimum. Clear? Right? I understand you know there are a lot of concepts lot of mathematical terms coming. So just hold on till we start moving on to the practical part and you know you will see them practically just try to grasp as much concepts as you can. So the different assumptions which we follow in linear regression is linearity that is the relationship between the features and the target homoidastiticity. The error term has this is where you know the linear regression has its assumption or constraint and the the term is known as homoidasticity that the error term is constant variance throughout the whole data. The error term will be constant. Multi-olinearity, there is no multi-olinearity between the features. And if we talk about independence, observations are independent of each other. getting my point? Normality. This is where the uh point I was saying that the error residuals follow normal standard distribution where mean is equal to zero and the standard deviation is equal to one. Okay. So multicolinearity means that we can have more than one input. Are they related with each other? No. every in in uh first input should be independent with the second one. Second one should be independent with the third. The input should not be uh dependent on each other. They should all be directly dependent on the output. First let's try to understand the concept. So how do we go about in the supervised uh learning process that we have this full data set right and we will after doing all the pre-processing so today I told you to revise all your EDA that we will load the data set using pd dot read csv command do head and tail try to find out null values work on them uh do uh encoding if required and then split the data into training and testing. First step is getting clear and of course training can be se it's generally 70 it's generally 7 to 2 to 80. If you want to keep it 50/50 also nobody is stopping you or 6040 that's completely up to you. Okay. And the function which is used in Python in the sklearn prep-processing library is train test_split. X is the first parameter which represents the features of the input data. So this is how we will create a list or array which will contain all the input. Y refers to the target or the label vector of the input data. Size of the test data test underscore size. And finally we have random underscore state. What is this random state? It is the seed that initializes the pseudo random number generator. What does that mean? That I will give any fixed number. It could be 42. It could be three. It could be zero. It could be 500. Right? I want to keep this number constant so that the shuffling or the randomization of the data happens in the same way. so that I can check the accuracy and do the comparisons. Clear? Right? So again remember this point whenever data is given to you you have to identify it in terms or split it in terms first in terms in number of inputs and the output. always separate the outputs from the input and then we will use this function and then we will get four outputs. Train test_split gives four output that is x train y train x test and y test. Clear? Training test_split is used to split the data. Right? We are preparing the data which will be used for training and testing separately. So the parameters of this function are input output how much we want to give for testing. It could be 3/4 half completely up to you no rules for that and random state gives me the seed that initializes or does the shuffling of the data in the same manner. And the output of this function is four output. I get input training and output training data and input training and output testing data. Clear? And now if I talk about multiple linear regression, right? In threedimensional setting with two predictors, one response, the le squared regression line becomes a plane. So this is a multiple linear regression model. epsylon is the normalized error. Okay, which is irreducible. Nothing can be done much about it. And now we will practically deal with data which has three inputs. The budget of TV, budget of radio, budget of newspaper and which one of them affects the sales most. So this is the first uh you know algorithm that we are going to study. Okay. So this is simple this is polomial and these are the errors. So let's start with the practical work and then get back to the different errors and analysis. Got it? Have you got an idea what linear regression is all about? So I request you all learners to from the LMS2 open 3.2 to supervised learning regression linear regression five. Okay. So now do you understand the term independent independent variables straight line when we have in two dimensional when we have one input one output it's a straight line blue dots are my data points and this is a straight line or the line of regression which I want to fit on the data points. This graph is now very clear to everybody. I'm starting with the uh pile now. The first 3.2 regression. This graph is clear. And where do you think regression is used? It has a huge huge application in oil and gas uh industry. Various types of data collected in oil and gas industry from surface subsurface to understand production sale processes. How much gas is required? How much liquid? How much is the pressure that needs to be created? So linear nonlinear regression model forecast global oil production. Oh my god, the whole world is fighting on oil and uh well we can use our regression models to forecast the global oil production. Right? So the whole world is on a big fight on oil only right. Marketing and marketing linear regression helps to analyze the effectiveness of advertising campaigns. Predict sales based on marketing spend. Segment customer based on demographic data. Right? Then we have retail. Then we have linear regression is utilized in retail for demand forecasting, inventory management, pricing optimization and customer analytics. Health care linear regression is applied in linear health care for predicting the patient outcomes analyzing relationship between the different medical variables and their diseases. Getting my point? And finally we have the real estate that in the real estate industry linear regression predicts the property prices based on the factors such as location size Mntes uh amenities and economic indicators. Clear? So now what are the different types of regression? Regression can be classified into two categories linear and nonlinear. Linear regression finds a straight line relationship between the dependent variable and one or more independent variable. Nonlinear regression uh finds a relationship between the dependent variable and independent variable using a curve or more complex shape. Okay. So getting back to linear regression it is so technically if somebody ask you what is linear regression in in machine learning you will say it's a supervised learning algorithm which is used to please try to understand each and every term is used to predict a continuous target variable please be clear it's not continuous data as you all were saying that's a wrong statement it helps us to predict continuous output variable by modeling its relationship with one or more independent variables through a linear regression. It predicts a continuous dependent variable based on one or more independent variables. So it predicts the continuous dependent variable. Here it says target output dependent completely anything you can say. It uses the le square criteria to estimate the coefficients of the regression equation. What do we mean by the le square criteria? That the value between the actual and the predicted value. The square of the error of this error that is RSS should be the should be minimum that is the residual sum of squares. Getting my point and it can be applied if there is linear relationship between the variables. So in case the dependent variable is continuous and independent variables can be continuous or decre discrete we are not bothered about our input variables. So for predicting the uh you know output we are always considered about the output. So the relationship between a dependent variable Y and one more independent variable X is established using a best fit straight line which is also known as the regression line. And there are two types of linear regression. One is known as the simple linear regression. Other one is known as the multiple linear regression. All right. So simple linear regression is clear to everybody. We have one independent variable and one dependent variable. Do we understand now the terms beta kn and beta 1? This is the intercept and the slope. Yes, everybody is understanding now what is simple linear regression. And as I told you a simple linear relationship is given by one of the input that is TV expenses over here as my input and in the output we have that sales. That means in this particular data set we are trying to establish the relationship that if what is what is going to be the budget of TV and how it is affecting my sales. So what do you conclude? Do you see the red points? These are the predicted values and the blue points are nothing but my actual data points and I'm going to calculate or it automatically does the calculation of the error if once I you know apply the algorithm. Clear? So do you see a kind of positive linear relationship? Can I conclude that this shows a little positive linear relationship? That is the value of TV expenses increase the sales is also increasing. It shows a positive relationship between them. Getting my point? And practically if you look at the real world problems there will not be simple linear regression problem. They will going to be uh you know multiple linear regression problem. There will be more than one input from how we know positive and negative. See positive and negative button that's that's you know shows it very clearly from the graph. If the graph is showing this thing that as my x is increasing y is also increasing. It shows a positive relationship. See over here and if x is decreasing y is also decreasing. But if my graph is like this what does it show? That x increases y decreases and if x decreases y increases. Getting my point? So this is a negative relationship. Inverse relationship between input and output shows negative relationship and uh direct relationship between input and output shows positive relationship. So as I was telling you practically if I talk about uh you know real time data uh you know we have more than one inputs and output. So multiple linear regression models the relationship between the two and more independent variables predictors features and the dependent variable as a straight line. The equation for multiple linear regression is y is equal to beta kn plus beta 1 x1 beta 2 x2 and beta n xn that x1 x2 xn are the predictor variables and beta 1 beta_2 beta n are the coefficients of each predictor. So let's start practically working on this data set that this becomes now if I want to try to find out the relationship between TV expenses, radio expenses and sales. So now my straight line turns into this hyper plane. Do you see the difference how the straight line has got converted into hyper plane? And of course there is one constraint that we cannot view more than 2D or 3D graphs right we cannot view more than 2D or 3D graphs right that correlation covariance is same as uh you know um regression but in regression it's a linear relationship but the positive and negative relationship is the concept same as correlation and covariance that correlation and covariance is founded by Pearson's you know coefficient of correlation but here we are talking in terms of linear equation but if I talk about one input x1 and y how x1 is related to y then that is r that is this + one minus one that's that exactly is the same concept the concept is not very different linear regression is using that particular concept all right so now let's start practice practically working on the data set. So what I want everybody to do is on your uh you know go from data set folder open your Jupyter notebooks and keep the TV marketing CSV data in the same folder where you have kept this 3.2 yes learners wherever you have kept this 3.2 to keep the TV marketing dot CSV copy it from the data set folder and start uploading this data clear and then run this thing. So basically this data set consists of three input variables. This is my x1, x2, x3 and this is my output. All right. So once I have done my head, what does the head command do? What does the head command do learners? What does it prints the top five rows by default and these are my inputs. Right? And what does the info function do? Good rishi. Good anush. What does the info do? It tells me there are no null values in this data set. And of course unnamed is not required over here. And all other my inputs are also in numerical value. And what is my output? My output is sales. And that's a numerical value. And that is why regression will be used. Clear? And that is why regression will be used. Getting my point. Okay. So what is the first thing that we are going to do? Since this data does not require pre-processing, it's a clean data. So nothing required to clean or check on the null values etc. Right? So we are considering the data to be clean. So now when I want to fit any model what is the first step that I will do? I will try to separate the input to the with the output columns. Agreed? The first step is to extract the features of the data set and output. What is this ILC in Python code? Index base filtering. So if I see very closely that 1 2 3 are my input index. This is of actually no relevance. This is of no relevance. These are my input values and this is my output value. Clear? So in this case ilocc the first colon represents it. It talks about all the rows but only the columns which are there. It says one and two. So how many inputs are we talking about? Only the first one. Only the first one is TV. And what is the output? Only the sales. This point is getting clear. What are we taking over here? We are talking about TV and sales. So we are trying to implement linear regression, right? We are trying to do simple linear regression. This X and Y is clear. I've only taken a part of it. First I've taken one input and then I'll take all the three inputs. This point is clear, right? And then we have the from skarn model selection import train test_plit. And now I am taking my input that is only the TV input. Y is my sales test size. 30% of the data is used for testing and random state is taken to be 42. 42 is generally considered from a fiction that it's cons considered to be a you know a good number for the whole universe. Clear? Okay. So now what are the different steps to now fit the model not much of coding that's why Python is a preferred language beautiful language that is from sklearn.linear model import linear regression that is first we will import the library that is skarn is the main library and we are importing linear regression from there. So when I imply linear regression it is lin_re and this is fitting that is the training input and output training data. Getting my point fit a function is used to fit the data or train the data. What do we mean by train the data? finding the parameters over here that is beta not beta 1 beta 2 beta 3 etc. Clear? Are you all there with me till this code? Anybody who still facing any difficulty who's not been able to run this part of the code till here are we good to go? First step is to import the library which contains the linear regression model and then I will create an instance of this linear regression function right that is lin rig it is a instance what do you understand by instance object of this particular function and then fit the training input and output data it is divided by using this train test_split and we get these four outputs input training and testing output training and testing I explained you train test split where we get the four outputs now let's try to find out the linear relationship between the sales and the TV so TV and sales so we generally we want it to be an input. So if you visualize the linear regression model X label we don't want it to be that's I think so that's a little opposite of it over here this is TV change make changes in the code and output is sales how do we delineate contributing and constraining factors and model. What do we mean by that? Delineate. What do we mean by that? Okay. So, what does this graph represent? What does this graph represent? This plot shows a positive linear relationship between sales and TV. And where the blue regression line indicates the model's prediction, the green data points are generally close to the line suggesting the model fits the data reasonably well through those some variability exist. Right? So does it show a positive relationship basant? Right? Do we see that there is a positive relationship between TV and sales? No, it's not the train data. It is the test data. Why test data? Because I'm plotting input as training and predicting the output. The xaxis is the input. The x is input, right? And the output is prediction of the training. Right? So, we are trying to plot over here. Plot over here is the straight line. Right? But why am I saying the test data? Because actual data points are my input and output test data. Now clear, we are trying to plot uh do a scatter plot over here. Scatter plot is between input and output test data. The actual data points are my testing data points. Okay? And the plot the straight line is input training and predicting the output. That is why it say it says the linear regression model for test data set. Now clear. So uh what what do we conclude from this graph? We can conclude that the plot shows a positive linear relationship between sales and TV where the blue regression line indicates the model's predictions. The green data points are generally close to the line suggesting the model fits the data reasonably well though some variability exists. Okay. So this shows a positive relationship. Now the important point is how do I understand is this model underfitting or overfitting. So when we are developing machine model what is the most important point important concept that needs to be taken into account? The concept of overfitting and underfitting. So when developing machine learning models achieving the right balance between the complexity and simplicity is crucial. What are the points of overfitting? Let's quickly go through that. Overfitting occurs when the model learns the noise and details in the training data too well to the extent that it is negatively impacts the performance on unseen data. Right? That means when the variance is high sign high accuracy on the training data but poor accuracy on the test data. So how do I know that my water has overfitted? That I get a good score for training data but a bad score for testing data. And why is it what is the reason? Because the model is too complex. It has got too many parameters. Underfitting happens when the model is too simple to capture the underlying pattern. Poor accuracy on both training and testing data. Underfitting is want a bias algorithm algorithm high variance to be a good fit. This of finding a trying to achieve model from the mathematical intuitive dimensional plane it how do I functions in Python that is concept of that particular should be the option clear and how do I check the error. So if you just look at this is more than the testing is a increasing model complex a good fit on this particular data errors. How do I till here I square error this error? Please try regression. No right there would be several error linear reg about the mean error square of are you understanding this error and when I talk about root mean square error that is when I take the under under root taking the root mean can anybody tell me square error anybody who can standard deviation and variance error will increase square root will get proper value. No. But why do we take that under under root of variance a proper value? No. Have to compare cannot make it at you know can I it's something like me square with cm can I compare a cm squared or a me square a unit of area it becomes at at the same absolute error clear? Now terms of error and if I actual minus the pre whole square is there any difference between that is there any difference? No. So mathematically whatever way there factor of comparison that is understand R square it is a relationship actual points and the right how my points are far from the average right. So SSTO is getting clear to everybody. So I'm doing my comparison is the total same as the RSS actual value and it's diff any no positive SSC or the error sum of square this is like RSS actual minus the predicted value this is my actual regression line if SSC all observation around the oh sorry the predicted or the fitted value on the regression line and the mean of the fitted value that is equal to S in the relation to S's that my to the sum of square and predicted minus the average. Agreed learners? But what is the use of all these? So now there is this another factor another uh unit which helps in comparison of value that is R² that is known as coefficient of determination. That is known as coefficient of determination which is also known as the R square statistics proportion of variance explained the value varies between zero and one and independent of the scale Y right so R square is equal to S STTO minus S SSE upon SST SSR upon S STO so what is the advantage in short let's not get into the mathematics that the R square value will always lie between 0 to 1. So the values which are more close to one are better fit as compared to zero. Yeah. Mean square error and RMSSE relationship is clear but they are not relative errors. I cannot do comparison between the different models that this model is better than the other. They are good to tell the errors are less or more comparison between training. But R square is a relative error. Even absolute error is not a relative error. What do I mean by that? What do I mean by that? Let let me explain you with an example. For example, you scored 70 marks and I have scored 30 marks. Which who scored better? You would say 70 marks. Ma'am, you you scored better than me. But I scored 30 out of 30. You scored 70 out of 200. Getting my point? So there how will I find the relative? If I start taking percentage of this that 30 out of 30 is 100% and 70 out of 200 is somewhere around 40 50%. Then I can do comparisons. Got my point? That is why we are now heading towards ratio between the SSTO the total sum of squares with RSS and SSR taking into all account all the variability of the data points and that's why I now reach on this R squar term now better is it better sum of square error ranges between 0 to 1 which measures the amount of variability that is left unex unexplained after performing the regression. An SST minus SSE measures the amount of variability that is explained or removed after performing the regression R squared and the proportion of variability in Y can be explained using X. So R square is trying to take into account all the errors and variability of the data point. So when R² is equal to 1, SSC is zero. That means it's a perfect fit. It's an ideal situation. Will it ever happen? Hardly. I don't think so. You know it's going to be but something which is close to 1.99 is a good and when I get this horizontal line that you all were getting that means R² is equal to zero error is equal to total that means there is no linear relationship between X and Y. Now clear right but R² suffers from one one limitation right since R² is also known as the if it is greater than one it can never be greater than one that's the assignment that's not possible mathemat can percentage can ever come beyond 100% the formula is such now because we're taking a ratio out of 100 right Simon So mathematically it can never be uh true. Got it? Okay. So R² also known as the coefficient of determination measures the proportion of the variance variation in your dependent variable X and explained by your independent variable X for the linear regression model. But the problem that R² also suffers is that it will always remain or increase as we are adding more number of independent variable. So this problem is solved by adjusted R². What is adjusted R square? That it measures the proportion of the variation explained only those independent variable that really help. So what do we do? We divide the error by degree of freedom. Do you understand the term degree of freedom? Learners might be thinking so many concepts. Degree of freedom is the number of independent the number of independent variables. Right? Suppose if there are 10 10 if suppose if there are 10 input features then the degree of freedom is n -1 that is n clear how many input features are we depending on abinri that is referring to my degree of freedom it's something like now let's understand it over here now if I want to predict the price of the car. What are the factors that are determining the price of the car? Safety is a very very important factor. How many airbags does it have? Does it have a parking sensor or not? The branding. What is the most important factor? The mile, the fuelage consumption, the fuel type in today's world, assistant, infotainment system, is it a luxury car or not? the color, the kind of uh you know the design, the seats, the the the the the wheel, the alloy, there could be several factors. Agreed learners? But do you think the the kind of wheels or the wheel alloys are as important as the engine of the car or the model of the car to determine the price? No. Right. So that is what is taken care by adjusted R² and when we divide it by the number of degree of freedom getting my point see normally what will happen is if I keep on adding the factors in R square that I've added the number of alloys the number of uh you know the seat covers the color of the car the the R square will keep on improving it will give me an illusion Oh, my model is giving so much well answers. But that's not true. Maybe you know adding the kind of wheel alloy is not that concrete uh factor to determine the price of the car. See the these numbers of degree of freedom as I told you this is some example that they are taken. Degree of freedom depends on the number of input features. So that is why adjusted R square is a better better uh you know way of determining the results. So if I get back to the file over here and now look at how I calculate the results. So what am I trying to do? From sklearn metrics import mean square error R square error and then I do the prediction. Prediction is done by the predict function for the training output as well as for the testing output. And then the different metrics can be simply calculated in Python by using the mean square error for training data for testing data and R² and then I can do the comparisons. So if you do you see the training error is 0.573 and testing error is more than that that's more closer. So they all refer that the model is underfitting. Now clear how are we trying to fit the model? How are we trying to uh fit it then predict the output and then calculate the error. So if we talk about linear regression, let's let's look at it again that this is my actual output right and if I talk about a single variable as my input and there are two coefficients beta KN and beta 1 this is the epsylon error. Is this first equation getting clear to everybody? Simple linear regression single variable is getting clear to everybody. Then we have linear regression multiple variables right we have x1 x2 x3 right absylent over here refers to the random error and if we talk about a model evaluation are we clear with these metrics now the mean absolute error the mean square error and the root mean square error right so comparisons between the different models is not possible. Therefore, we have R square error which talks about the ratio and even better than R square is adjusted R² because it is not affected by the number of inputs in the data. Clear? So basically we divide it by degree of freedom mean model. Clear? And we have also understood different scikitlearn objects. Please try to understand fit function. Fit function helps us to train the input parameters or train the parameters for the data. Transform is transforming the input to output. Fit transform is mixing the function of fit as well as transform. And it is the predict function which is used to predict the output of the training data or of the testing data. To understand it better, types of skarn objects are based on transformers. Transformers are nothing which transform the data set. Have you learned feature engineering learners? Do you understand feature engineering where we do feature scaling, standardization and normalization, encoding of the data that we need to transform the data before we actually fit into the model. Let me try explaining you over here. So before the data is actually fed into the model, it needs to be transformed. Why it needs to be transformed? Because if they are missing values that will not fit into the model. If they are categorical features then we have to in involve uh encoding. Feature scaling is a feature which brings the data into one particular range and outlier detection. That is why data science is important because these concepts are covered in detail. How do you deal with data? How do you handle missing values? How do you deal with categorical data? Scaling of the data is all part of the feature engineering process. So in feature engineering, we are not just building models but we are playing match makers for data as well as algorithms. Getting my point everybody? Okay. So getting back for transforming the data set right. So fit learns what does a fit function do? It is an estimator. Please try to understand. Fit is an estimator which estimates the model parameter based on the training data and hyperparameters. And finally we have the predictors that is it predicts the data makes the data set as input and does the prediction score method to measure the quality of the predictions. Getting my point? So here we have the data pre pre-processing training and inference model clear and then we have fit and fit transform right what do we do that when we are trying to train the data what I'm trying to explain let me explain it with this slide that again please try to understand let me rewind things in supervised learning we have label data first First of all, what is label data? That we have the input as well as the output. Now, in the story, have you understood two types of output in supervised learning? That we will have the actual output of the label data and one we will predict the data using the X test. This point is getting clear. what this point which I'm trying to explain regarding the output is getting clear right so this is output actual as well as predicted now we also understand that we divide the training and the testing data right so ultimately we are trying to when we are doing training it is all on the training data by the fit parameter so the model all the training will be done using the fit function in simple terms, right? And the testing and output will be on the predicted. But if I want to transform the data, so I will fit it and then transform it. And can I do the function of fit and transform together? Yes, by using fit transform function. Getting my point? Can I do the function together? Yes. For only training data I can use fit transform and for testing data I will only use the transform function. Now clear okay so now let's once we have understood now let's get back to the code. So now are we able to also understand how when the model is underfitting and overfitting that point is also clear. >> Yes. And if we talk about nonlinear regression, polomial regression, uh polomial regression is a subset of linear regression that includes polomial terms. The relationship between the independent variable x and dependent variable y is modeled as an nth degree. So here we are not trying to do it as a straight line but polomial regression which is a subset of linear regression which includes polomial terms. The relationship between an independent variable X and dependent variable Y is modeled as nth degree polomial. Right? So why polomial uh regression is called a special case of multiple linear regression? Some polomial terms are added to multiple linear regression equation to convert it into polomial regression. So it is a linear model with some modifications made to increase its accuracy. The data set used in polomial regression for training is nonlinear. So what do we what do we observe that in this particular graph this is a straight linear regression and if I change it to a curve it becomes polomial linear equation. Do we see that exponential curve over here? Okay. So I have this uh you know simple data right in this simple data. This is the project statement. Okay. Wait. Okay. Yeah. So a certain spare part manufactured company once a month in lots which vary in sizes. data on lot and size numbers and man of hours that that means the input is the lot size of the people and how many man hours are required right it's numerical right so if I start analyzing the data actually you know it's done through an excel it's simple that I do x - xar yi y - y bar and I calculate this what is this known as rss residual sum of square actual minus minus the output the whole square x i minus x i the whole square. Do you see this? And why am I taking the square value? Because I simply take the difference between y minus ycap. The error is zero which is actually not which is actually not. Therefore we take the square values and I get the answer as 13 six. Okay. So the sample size the number of rows in this data set is 10. Degree of freedom over here is 9. Mean is equal to Xar sigma xi upon n 50. Variance over here is this much. Standard deviation is in this much. So this is where statistics is coming into picture. We have calculated the value. Similarly we can calculate it for the man hours yi. Okay. And if I try to see the relationship, it's coming out to be straight line. Right. Now coming on to the concept of how do we calculate the best fit line right. So this is the le square concept where we are trying to find out the RSS. So over here what am I trying to do again? Y I minus the ycap the whole square not this one. Okay, here it is doing hidden trial method. But ultimately the le square estimators can be found out by trial and error method. But that is not the case. We try to calculate it by the normal equation. This is the equation to for calculating of beta KN and beta 1. So these are already predefined formulas which run at the back end when I run the linear regression function. So this example shows that whatever values you are getting by solving mathematically or through linear regression model I get the same values. Okay. So what does polomial regression on data set work? Again we are using the TV marketing CSV features from the skarn pre-processing we are including the polomial features again first we are splitting the data so now I am creating an object of polomial features right with degree equal to two and now I am using the fit transform function not the fit function itself but the transform function function because I'm transforming the input values as well as fitting but I'm only and only transforming the output. Do I want to train the testing data? No. Why I don't I want to train the testing data because so if I give you what is going to be if in your test if I tell you the questions is it actually a test? No. That calls for paper leakage. If I tell you what questions are going to come for an exam that is known as paper leakage and the same concept is known as data leakage over here. Getting my point learners? So are you understanding the difference between fit transform and transform? The fit transform will only and only work for training data. Transformation can happen will happen for input as well as output will happen for training as well as uh testing but fit will only happen for training data. When I create an object of linear regression I get this output. So basically the polomial features transformer is configured to generate polomial features up to four degree. Then we are only transforming the input to generate a new feature set that includes polomial features and interaction. Then we transform the test data corrected from fit transform. It is used on x test to apply the same transformation and then train the model. The linear regression model is trained using the transform training data. Clear? And finally we are going to predict the output and then create a graph. Right? Do you see the scatter plot is testing data and plot is x range polomial predict the output. Now clear you are getting all the code how the graph is getting created. So what what is the observation and this is exactly you know why Jupiter notebooks are hit in the market because we are able to see the graphs over here directly which VS code lacks right we can see the code as well as I can write my observations also so as you can see the regression line is able to fit majority of the data points you can infer from the above implementation that nonlinear inputs require by a nonlinear model such as polomial model clear so I mean regression linear regression also has capability of dealing with nonlinear data up to degree four but not very high nonlinear data right then we have different variations of the algorithm and if we talk about performance metrics for analysis why different metrics because it calculates the average of the squares of error which is differences between the actual and the predicted. Then we have the root mean square error. Oh, I hope you all are clear. Mean absolute error R square error zero value indicates model explains none of the variance in the dependent variable. The independent variables have no explanatory power for the changes in Y and one represents a perfect fit. The model explains all of the variance in the dependent variable. The change in Y are perfectly captured by the changes in the X. Clear? So now we have to start understanding another technique important technique. Okay. So now let's understand what is cross validation and before we understand cross validation we need to understand few more concepts. Now in machine learning there are two terms that you need to understand. One is parameters and other one are hyperparameters. Okay. So now what is the difference between parameter? A lot of learners are you know get confused because they both think they are the same thing. No again I'm telling you I try to give you exactly you know the cris concepts differences between them. So when I talk about parameters they are values learned by the model during the training. So during the training what the model learns is referred to as parameters and hyperparameters are defined by the user to control the learning process. Purpose over here is it directly impacts the model predictions. So model automatically learns the parameters you know and that is directly impacted in the output. Whereas hyperparameters are controlled by us. For example, how much is going to be the training data and the test data is a hyperparameter test size or train_plit. Right? They the factors or parameters which are controlled by us during the training process or model fitting process that are known as hyperparameters. Whereas when the model learns the parameters on its own using the fit function are parameters. Clear? They are estimated during model training. They are generally uh you know estimated before the training begins. learn from the data using optimization. Whereas the hyperparameters are set they are usually set or tuned using the methods like grid search, random search or basian optimization. Influence on the model affects the output directly affects the speed and the quality of the learning and this is dependent on the data set. These are independent of the data set. So the model parameters are nothing but your coefficients beta KN, beta 1, beta_2 or they can also be represented by weights in general. So generally coefficients become weights. That's the general term used in deep learning. Whereas the test underscore size, the number of iterations and there are several other factors that will be controlled by us. Even random state are all hyperparameters. Is the difference getting clear? Right. So now what is the need of cross validation? Do we cross validate results? Do we want do we like to cross validate views, reviews of doctors, lawyers? What does that mean Adhinatri? And why do we want to do that? What does it mean? And why do we want to do that? So that you know we are more sure of the answer, right? If one of the doctor is say saying that you know you need to be uh you know maybe you know your disease has this and maybe you need to get operated and even the other doctor says that means you're more correct that yes or if the other doctor says no no no you it doesn't need to get operated you know you can cure it through these medicines right so cross validation or also known as rotation estimation or out of sample testing refers to the process of rotating or splitting the data into different subsets. So it helps in uh you know in the process of rotating splitting the data into different sub sets. Okay. So one part we are very very clear that when we take the data set we divide it into training and test and definitely we don't want to mix any of the training and testing data sets to avoid any kind of data leakage. That point is also clear that we want to avoid any kind of data leakage. That points also makes it clear. Now to make our results better, more confident, more accurate, rather than using only one part of the set, if I use only one part or one sample of the set, if I use multiple samples, I can get better results. Right? So this is my mini training data set. So it's always the training data set which gets divided into further training and validation split right. So what is the idea that we initially the data is getting split into training set and test set train and tune tune your models using cross validation. So we will try to find out okay we want to make certain changes this is better not better only through the training set. The test set is not involved. test will we do not touch this until the very end because ultimately that is going to give us whether the model is robust or not. Is this point clear now? So how do we go about cross validation? So model evaluation is this when we fit the model and then we predict the test set. That point is clear that we have one data set that is divided into training and test set. Right? This point this is normally that we do we fit it and then we predict the output and do the comparison. But if I want to do model selection between different comparisons what I will do now the training data set is going to get uh you know divided into train as well as validation data set. What is the use of this model selection? It helps us to explore the hyperparameters grid. Now there are different hyperparameters. How do you how do I know that this is the best hyperparameter for me? I will try to do it through the validation set fit on the train. Evaluate on the validation. Pick the best hyper parameter. Clear? Why are we doing this? because we want to validate our training and results better. It helps us in finding out the different hyperparameters, right? So as we move along more uh you know algorithms, you will understand it more. It's not part of regression but part of supervised learning also. So cross validation when data set is too small for apply hold out strategy then cross validation can be used for evaluation and model selection. So what do we do? Suppose this is my 100 rows in data set. Okay. And now I divide it into 2020 groups. So I have five groups. 1 2 3 4 5. So the one of the part is going to be used for validation other is going to be used for training. Then the next group is use going to be used for validation next. So I will iterate it for five iteration. Got it? Of course it is increasing the complexity but it is giving me a much more validated result. Agreed? So one of the be uh you know u uh techniques or variants of kfold is leave this is known as leave one out cross validation right so we have 1 2 3 and n over here right so if we take one part of the sample and the rest n are taken for training then the next row is taken and the rest for training. So how many time the loop will be executed end times. Is it a good idea? No. Taking each sample for test sorry for validation and others for training is not a good idea because it will increase the number of computation. But if I have kfold that is I divide it into number of groups my number of iterations decrease and I get better results. So how do we go about the performance and the output metrics right? So what will happen? It will so the cross validation will automatically create an array. In the first iteration I will get the first metric. When it takes the second kfold I get the second performance metric. When I take the third iteration I get the third performance metrics. Fourth the fourth one and last one I get the fourth fifth performance metrics. Clear? So to answer that point everybody please understand the test data is not touched. It is the training data and the different folds which are used and final evaluation is done on the test data whether I am getting the correct output. So the concept is same we are doing comparison between training and testing and that will give me the results whether it's overfitting underfitting etc. Clear? Yeah. But when we talk about categorical data, what does categorical data means? That the data is in category in terms of males and female. So the four we have which have been cross validated will be taken up for final testing. Final testing will be on the test data only. Once the parameters have been found out okay the number of folds okay the number of uh you know u k value for this thing is this. So now we will test on this value are we getting the minimum results or not it will help us to calculate the different hyperparameters. Got it? Okay. So when we talk about categorical data dividing it into male and female. So this is my round one. So stratified kfold is being used for categorical class which helps us to keep the ratio of the different categories same. Okay. This we will do it when we do uh classification. Okay. So what is the difference between kfold and stratified kfold? Kfold is random. Stratified kfold helps us to maintain proportions may vary across folds. maintains class uh distribution across FOS imbalanced data set not ideal preferred use cases balance and imbalanced and how do I check the metric that my cross validation is good or wrong first my uh you know parameters for cross validations are the estimators input output scoring and CV equal to five estimator is equal to the model object X is an array of the features values. Y is an array of the target value. CV the number of folds scoring the metric and it returns score an array of scores at each split. So you know the beauty of this particular function is that it returns at you know score of each uh iteration. So you are able to see which one is better but generally we take the average of that. Clear? So now let's get back to the file and do it practically. So this file is quite long. We will be using this file in the next session. Of course we will not be able to complete all the concepts today. So we are on cross validation. After cross validation then we have regularization. after regularization then we have uh hyperparameter tuning that is model optimization and then the pipeline so it's it's a long journey that definitely we are going to continue okay in this file yeah so cross validation technique everybody is there with me 3.7.2 into. So what have you understood? What is cross validation technique? Tell me learners. So cross validation is a machine learning technique that evaluates the model performance on unseen data by dividing the data into multiple folds. In each iteration, one fold is used as a validation set and the remaining as training. That point is clear. Then the process is repeated. So we have to give the number of iterations also. Yes. Is repeated so that each fold serves as a validation set once and the results from all iterations are averaged to provide a robust estimate of the model performance. Some of the common cross validation techniques is kfold cross validation. K refers to the number of equally sized folds that if it is unequal it will make it equal by adding randomly some of the you know randomly the data set only for the last one right and there is no harm also in doing it will not affect it much so the model is trained on k minus one folds and tested on the remaining fold that point is clear so if I have five folds then one of the fold will become the valid for validation test and four will be tested. This process is repeated K times. So is this K also a hyperparameter? Yes, we decide they are going to be fivefold, sixfold, 7fold, 8 folds, not the system. They are not parameters to the algorithm. It is decided by us. So it's a hyperparameter. And with each fold exactly once as the test data set the results are averaged to produce a single performance estimate we generally average it uh them. What is the advantage of kfold cross validation? It provides more accurate estimate of the model performance. Yes, that this is the answer. K is the number of folds. Yes, AJ. And consationally intensive for large data set. Yes, you have to reiterate the training process but it gives definitely better results. Clear? Then we have stratified kfold. Similar to kfold but ensures that each fold has the same proportion of different classes as the original data set. This is especially use useful for imbalanced data set. more reliable performance estimates for imbalanced data set and still computationally expensive. Third is the hold out method simple and fast one uh we we divide the data into training and test and the training data is first further divided into training and validation data set. So the model is trained on training set and evaluated on the test set. It is simple and fast. the evaluation may be noisy and variability in the training. The normal one that we do we divide it into training and test is known as the hold out method and leave one out cross validation is again a special kind of cross validation technique used in kfold. A special case of kfold cross validation where k is equal number to the number of data points in the data set. Each observation is used once as a test set. The model is trained on remaining data points. Clear? So this maximizes the amount of training data used. Cons, extremely computationally expensive especially for the large data set. Clear? Four variants of cross validation techniques. Kfold, stratified, hold out and leave one out cross validation. Okay. Now can we begin with the code? So I hope everybody is familiar with the pandas and the mattplot lab library. We are also familiar with the skarn model selection. These are the different uh techniques and metrics linear regression over here and this metrics to compare mean square error absolute R square etc. Clear? So here we are talking about housing. CSV. Yeah, take it as housing. So what are the pre-processing steps? What are the steps involved in EDA? Tell me how do you perform EDA? You'll do head tail to view the data. Check null values. Do info. Right? So let's look at info. So if you look at info, it has around 20,640 rows and eight columns. Do you see this? All are integer values. Do you see this? Yeah. Now the question is can you tell me which is the what is the output over here? Why it is a regression problem? What are we trying to do in this data set? Can you analyze it through the number of column based on the latitude, longitude, housing, age, total rooms, bedrooms, they all are inputs. I am trying to predict the house value. What will be the value or the price of the house? House price prediction. Now clear all these are my inputs and one output. House values are numerical continuous value. Therefore, this is known as regression. Clear? Okay. Now, let's let's see the observation that price lies uh between 1.1 million to 2.6 million. Houses are generally 18 to 37 years of old. Housing data. Let's check the null values. Yeah. Now, let's check the null values. So total number of bedrooms we have 27 values. How do we deal with null values? Either we will replace it by mean, median or zero. Since null values make up only 1% of the total data, rows and column features with missing values will be removed. So what are we trying to do? We are going to drop now because 207 is hardly 1% of 20,000 rows. So we can drop those data and now my data set is clean right and this is the categorical data which we are not using we don't have that and now can you tell me what does this mean X and Y now I have dropped the median house value access one because that's my output and now this becomes my output this becomes my input Right. So if I say print X and then print Y. Do you see? Yeah. So now do you see this? So X basically has longitude, latitude, house, median, total bedrooms, etc. And Y is only that. So let's remove Y. lift. So I keep making changes in the code. Are you understanding? These are my now the input and my output is median house value and then I use the train test underscore split. Clear? Okay. Now let's perform kfold cross validation. It implements kfold. Number of splits is 10. Divides the data set into 10 10 folds automatically. random state is 42 shuffle is also equal to so I have created an object of this kfold now I initialize the model there with me and then I evaluate the cross validation score based on the model was linear regression training data and the target variable scoring I'm using negative mean absolute absolute error as the performance metric. So this is where the absolute error comes into the picture. CV cross validation is KF. Number of jobs is equal to minus1. What does this mean? It utilizes all the available processes for parallel computation. Do we understand the concept of threading or the number of core processes learners? So this parameter takes care of that. Okay, this parameter takes care of that particular one. Clear? So from statistics import mean, I'll take the mean of the K4 cross validation. K4 folds scores are not defined. So let me run this. Let me you know show you that since the number of folds is 10. So the output of Kfold crow is an array object. See do you see they all are mean negative absolute errors negative does not mean that the error is less it's just the sign okay so we will try to take the average of these right there will be 10 outputs since we have given 10 folds 1 2 3 4 5 6 7 8 9 10 clear it's an array and then I will try to take the absolute ute average of this clear. So we have trained the model and evaluate on the test set. So are we now understanding how are we going about cross validation score learners and now finally after cross validation now we can use the test data. So this is how I have fitted the data predicted it through the test and my MSE test MSE is here. Similarly, my R square is here, right? So, my test MSE and R square scores are here, right? This the look at the error. It's so huge indicating that and on average the squared prediction errors are large. This can be interpreted in context of units of dependent variable which are likely in order since the numbers are very high. That is why the errors is coming out to be large. This can be mitigated by scaling the features. Feature scaling is critical in machine learning to ensure that all features contribute equally. So this is how and then we demonstrate leave one out also locv scores and we get the absolute mean score. Clear? So the mean absolute error is high to improve the model complex more complex model can be considered which will be discussed in further lessons. So the concept of cross validation is clear. Let let's do a quick knowledge check on that. Okay. A quick knowledge check. First question. First question. Which of the following which of the following cross validation versions may not be suitable for very large data set with hundreds of samples? We just now studied that I think so we've all seen it practically you know practical has a major uh impact. We all have just now seen the impact and it is leave one out cross validation. Great. Which of the following is a disadvantage of Kfold? uh cross validation method training algorithm has to return from scratch. Do we understand this? Every time it has to restart and do it again. Next question. Suppose you have picked the parameter for model using 10fold cross validation. Which of the following is the best way to pick a final model to use and estimate its error? Train a new model on the full data set using the parameter you found. Use the average cross validation error as its error estimate. So you can use average cross validation error as its error. Everybody got this? Why C is not correct? Because we will train the model on the full data set. Okay. So what is the idea? The best way to pick why is the answer B correct? The best way to pick a final model is to train a new machine learning model on the full data set using the parameter learned to use the average cross validation error as its error estimate. So please be clear with this particular point that is why I have added this MCQ. We can compare different models using cross validation. Cross validation is mainly used for comparison of different models. For each model you may get average generalization error on the K validation sets. Then you will be able to choose the model with the lowest average. Clear? And cross validation is also used for model checking not model building because it allows to repeatedly train and test on a single set of data set. Let us suppose we have a linear regression model and a neural network. To select the best one among these we can use k-fold cross validation. So to compare between the models also we can use cross validation to select a better performing model. Clear? So when we are now looking at the regression outputs right or the regression analysis some of the outputs are like this. So the output over here that we receive is in these terms of coefficients that this is my beta KN this is my beta 1 beta_2 and this is my beta 3. So this is my north south east and constant term and these are the values right. So what is it showing? This is the constant value positive. East has positive relationship. But south has more positive relationship with the heat flux. Heat flux is the output and north has a negative relationship with the output. Are you now understanding it better? How do we get the output and how are we relating it with the uh you know the equation of linear regression. So this is nothing but like beta kn plus beta 1 x1 the value of east plus beta_2 x2 minus. So a lot of you had question that how do we understand negative it will automatically get this. So this is multiple linear regression. Yes learners this is multiple linear regression and there was lot of confusion regarding ma'am what does multi-olinearity mean? Multi means referring to multiple independent variables multiple inputs with multiple regression. Call means to uh join or together referencing to the linear movement or correlation as I told you tries to find out correlation in terms of minus1 + one occurring within the linear equation and suffix means the idea. So if we look at over here from statistics we understand that p value is a very very important term right that if I have the value lesser than 000.5 then this is accepted this is accepted this has a strong evidence statistically significant but if this is not less than 0.05 05 that means east is 2.12 is not very statistically uh proven or confident that the value is this clear and when I talk about vif how do I calculate multicolinearity it is the variance inflation factor which is coming out to be 1.21 21. What does that mean? This is the correlation map, a heat map that you understand. So, multicolinearity is the phenomenon of high correlation between the predictor variables can create instability and bias in regression model. To identify and address multiolinearity, we use the variance inflation factor. So the VIF is equal to 1 - 1 upon R² and if the value is equal to 1 that means all the input values are independent and if it lies between 1 to 5 it suggests moderate correlation over here we are getting the value between 1 to 5. So there is moderate correlation or we can say independent also and if it is greater than five then it indicates high correlation that means then linear regression cannot be fit uh can be fitted on that input values. So where e vif should should not be used polomial equation dummy variable or a nomial um variable and multi-olinearity reduces the statistical significance of the independent variables. VIF is used to detect these variables. A large variance inflation factor on an independent variable indicates a highly collinear linear relationship to other variables that should be considered or adjusted for structure because multi-olinearity is one of the assumptions that we uh you know uh you know assume when we are uh trying to build the linear regression. model. So that means if the VIF value is greater than five then linear regression should not be used. Clear. I hope these points are clear. Now first concept that we will learn data leakage in machine learning. So now we will understand the concepts of pipeline. Today pipeline is a technique for automating different processes. What the what is the different processes that we do whenever the data is loaded we want to do transformations such as encoding scaling right so we'll try creating a pipeline and what is our main aim that we want to avoid data leakage of course why because if there is data leakage the data gets lost a lot of information is also getting lost so now if we look at the data from the supervised learning perspective. It is divided into two parts. One is known as the training data and the other one is known as the test data. Agreed? Okay. So a scenario when the ML model already has information of a test datas test data in the training data. Do you think that test data should be present in the training data? If I tell you okay in your exam these questions are going to come that going to be very beneficial for you for scoring marks but do you actually learn out of it? No, that's not a good uh way of learning right that you know it can give you good results but you are not going to become you're not going to be a robust model. That means if any other question is asked from that particular topic you will absolutely fail right but this information would be available at the time of prediction called data leakage. So what is the disadvantage and how can we avoid it that it causes high performance while training set but performs poorly in the deployment or the production. Okay. So when the there is an or if I say when there is an overlap of training and test data then it causes data leakage right and basically data leakage happens due to two reasons. First we understand train and test contamination or the target leakage. What do we mean by that? Target leakage occurs when the model is trained on the training data that contains target or the feature information. So we don't want that and that should not be available at the time of prediction. So we have to be very very careful when we are doing this. The other one is the train test contamination is an event where the test data leaks into the training data and the data prep-processing steps for transformation for example scaling encoding are applied before the splitting the data set. So contamination is when the test data leaks into training there and the pre-processing step so that uh the scaling encoding should not be applied before the splitting of the data set to avoid data leakage. Got it? Basically if the there is mixing of training data in the test test test data set or rather test data in the training data set then this causes data leakage and we have to avoid it in every case that is we should not apply any transformation that is scaling or encoding before the splitting of the data set. Now clear. Now another very important concept of regularization. There are two types of regularization available under regression that is ridge and lasso. Preventing overarning. Yes. Normalizing the data. Yes, we can say that. So regularizing thing in in a normal way, right? We want to prevent it from overfitting overarning. Right? So regularization is a technique in machine learning which prevents overfitting of the model. And how do I know that the model has overfitted? How do I know that the model has overfitted? Test error much higher than the training the training error. And why is it that the model is trained so complex than requests high variance it covers every possible train outcome making it yeah so the complexity of the model increase it tries to train on each and every data point and fails on the final test data due to noise due to learning of noise in the training data so to avoid it so as I've been telling you that in machine learning do we suffer more from underfitting or overfitting overfitting right so we need methods so that our model does not overfit and we need certain control parameters to achieve that. So there are two you know algorithms under this that is ridge and lasso. So lasso is known as L1 regularization technique. Can anybody tell me what does this term mean? Anybody? It is summation of the actual value or the true output minus the predicted output the whole square. What is this term known as? It's an error. I want the typical name of this error. It is the residual sum of square error. But rather than only taking this error actual minus the predicted value the whole square I take into account the another hyperparameter lambda which I will use to control or regulate the training process. So do we have a regulator? Do we have a regulator to control the speed of the fran? Similarly lambda we will use the regulator and similarly summation of beta. Can anybody tell me what is this beta over here? What does the term beta mean? It is the coefficients. It could be a single, it could be many multiple variables. So this is known as the regression coefficients. Right? not the intercept or slope that is only in twodimensional case like beta KN plus beta 1 x1 I agree both of you are clear uh correct on those perspective but in general betas are known as regression coefficients or the weights parameter so what is the advantage that we are getting through this L1 regularization that in the error term in the RSS term. Now we have added the penalty lambda which is controlled by us and which controls the value of these regression coefficients. Getting my point and in ridge regression that is known as L2 regularization. So this is now also you know known as my error or it is also known as the cost function. Try to understand in deep learning the error with which we are calculating becomes my cost function. In this case it is represented as L over here. It represents it as my loss function. There is slight difference but actually in machine learning they all mean the same thing. Okay. So here we have the RSSW. Now what is this W over here? Is it the same as beta? Is it the same as beta? Yes. It's the regression coefficient. Is the lambda same? Yes. The lambda in Python is known as alpha. And then we have sum of the square of the weights. And what is this actual value? Predicted value. Now are you understanding whether it's W or beta? They same mean the same. And this is the lambda parameter with the squared of the coefficients. Okay. So what are the advantages and disadvantages? And now then we will practically jump onto the file and start learning from there. So L1 regularization performs feature selection. What do we mean by features independent variables or the inputs? Yes, the inputs. So now we are trying to control. So now what is the use of overfitting? We understand that overfitting generally leads to high variance. Agreed learners and complex models. Agreed? Do we understand this concept everybody? So to reduce the complexity now I will try to control my coefficients regression coefficients beta KN B1 and beta N or in general I can also call them as weights. Now getting my point do not get confused with the terminologies it somewhere it would be written as weights somewhere as beta somewhere as independent variable somewhere as features. So you should be able to understand the concept of it. Clear? So the beauty of L1 regularization is that it performs feature selection by shrinking the less important features weights to zero. So if I have a multivariable data set from beta 1 to beta to beta 7 it will shrink few of the features that means some of the features or the weights will become zero that means they have no significance relationship with the output. For example, if we want to yeah, just try to listen and absorb as much as you can that suppose you know we want to predict the price of the car. Okay. So there are several factors from fuel to design to color of the car to the alloy of the wheels to the security systems to the infotainment system to the sunroof. There can be several re features but maybe the alloy of the car or the wheel of the car might not be important. So it can be reduced to zero that will help us to reduce the complexity and that will try to reduce the overfitting of the model. So very very important concept L1 is used to reduce or shrink the less important features weights to zero. So it helps us in feature selection right and it can also be used for highdimensional data set with many number of columns such as 20 30 50 100 with many irrelevant features and the disadvantage is it is not effective for data set with many important features. Now you might say ma'am how can we remove a feature? my every feature is important for the output. Then we use L2 regularization. Okay. Where the number of the where the value of the coefficients of the weights will not become zero. It provides a smooth solution and improves the generalization performance of the model. So ma'am ridge is going to doing the opposite instead of zero it is reducing the weight. Ridge is trying or uh you know it will it is trying to see the first one is making it zero and the L2 will try to reduce the value. L1 is lasso and the ridge one is making the value of the coefficient small not huge or big but small. So therefore the biggest advantage is that it can handle data sets with many important features. Okay. So now moving ahead to great ain. Now let's get back to the file. Everybody has the file. Everybody's ready in the Jupiter notebooks. Okay. So, regularization in regression in linear regression, regularization encompasses a set of techniques employed to address the issue of overfitting. So, what is regularization? Regularization is the method techniques to achieve the objective by introducing a penalty term. Please try to understand again it's a very very important question from the point of interview introducing penalty term lambda to model's objective function to prevent it from overfitting. This objective function typically measured by mean square error is minimized during the training process. So generally the error of the mean square error or the RSS to be precise is minimized during the training process. What is the advantage of that penalty term or the lambda? The penalty term discourages the model from attaining excessive complexity by penalizing the size of the model coefficients. So model coefficients, regression coefficients beta KN, beta 1, beta_2 or the W1, W2, WN thereby mitigating the overfitting process. Got it learners? All right. So now if we talk about the regularization term alpha, it can be known as alpha as well as lambda. Okay. They all mean the same thing. It's written as lambda or sometimes as alpha. So do not get confused. Okay. So there is a little terminology mishap happening. So everyone uses their own technique but try to grasp the concept. Okay. So high alpha is the hyperparameter that scales the penalty term. It controls the strength of regularization. Higher the alpha, it imposes stronger penalty on the coefficients. That means they tend to become zero leading to greater regularization. This tends to produce a simpler model that may underfeit the training data but often generalizes better to unseen data. If we talk about lower alpha, it imposes a weaker penalty leading to a model that is less restricted by regularization and more complex potentially capturing more details in the data but at the risk of overfitting. So what are we looking at? We are definitely looking at a value which is not very high and very low. So again minimum mid value to find out. Got it? How do we find it? What are the best methods? That we will understand today. That is known as hyperparameter tuning. That's part of this today's session also. Okay. Now, what are the benefits of regularization? It enhance the generalizability of the model by mitigating the overfitting factor. Regularization fosters model that can perform well on unseen data. reduce model complexity. It promotes interpretability and potentially reduces computational cost associated with training complex models. And the two common regularization techniques are L1 and L2. L1 is lasso. Lasso is a full form of least absolute shrinkage and selection operator. Right? So absolute uh term is there as the penalty. So Simon is there any standard to label it as high, alpha or low? Yeah, we'll understand. Of course, low is my zero values or negative values and high values are in thousands and lakhs. Let's do it practically to understand that point better. Okay. So is the full form of lasso clear to everybody? So which is the penalty term? Of course, it is the alpha or the l uh you know the lambda. But what are we trying to add? The absolute value of the regression coefficient. So the least absolute shrinkage and selection operator regression relies upon the linear regression model but additionally performs a so-called L1 regularization which is a process of introducing additional information in order to prevent overfitting. As a consequence we can fit a model containing all possible predictors. What are predictors? I'm I'm purposely reading. You might feel ma'am is ma'am just reads the data but I'm purposely reading it to make you understand line by line. So what is containing all possible predictors? Predicted value is an output. Predictors are input. So we can fit a model containing all possible predictors and use lasso to perform a variable selection by using a technique that regularizes the coefficient. So what is that lambda parameter doing? It is having control on the value of the of the the lambda or the alpha has it's controlling or regulating the values of the coefficients the coefficients the regression coefficients that's up to you how you want to represent it as betas or weights right so it performs variable selection or feature selection it forces some of the coefficient estimates to be exactly equal to zero with the help of large tuning parameter. So more the value of the lambda some of the features coefficient value will turn out to be zero. And that is why this is a technique which is also used in dimensionality reduction. What is dimensionality reduction? Reducing the number of features in the data set so that it reduces the complexity. So L1 again plays a major major role in that. It reduces it helps to reduce learning of more complex data and overfitting. It decreases the variance of the model without increase in the bias. All right. So in minimization objective does not include RSS like the OS regression but also the absolute value term. So this RSS now is clear. This is the residual sum of square and this is how we can expand it and write it. This this point is also clear. What is yi? This is the actual output. And what is this output? This is the predicted output. Are you all understanding it mathematically, conceptually? But in lasso where does the difference come? The RSS is the same but we have added the penalty term alpha or lambda along with the absolute value of the summation of the coefficients. Now clear if the alpha is equal to zero then there is no regularization that will happen because the error term will be same as the RSS. If it is equal to infinity all the coefficients will become zero. And if the alpha is greater than zero lesser than infinity coefficients are between zero that of le square linear regression. Got it? Is the theory part clear? Now let's move on to the practical part. Start with a new data set. So let me share it with you. hitters CV ca dot csv okay here we go please download this data set and be ready I hope different libraries are also clear the numpy pandas then linear model lasso the metrics mean square error r square all these are there okay so basically now let's understand the description of the data set so I I have provided a link over here. So uh this is for like we have like now IPL matches. So here we are trying to predict the salary of the player based on atbat number of times he batted in 1986 hits the number of hits the number of home runs the number of runs the number of runs batted walks number of times a bat during his career. So lots and lots of um parameters to decide the salary of the player that is 1987 annual salary on opening day in thousands of dollars. Got it? And then we have another important column that is new league a factor with a and n indicating the players league at the beginning of 1997. So what is the first step? We've loaded the libraries. We load the data set and the view of the data set with df. head. Okay. What are the other functions that we would perform info? So this data set is quite huge in the sense that it has the number of rows are only 330 uh 322 but the number of columns are many. So this is an object uh data type atbat everything is integer but we have league division as categorical data and then we have salary over here as the output and new lead. So what are the different steps that need to be performed? Can we directly apply the model onto this data set? So these are my columns. So I have used df.trop dot drop unnamed equal to zero. Encoding if any. Do you think encoding is required? Coding is encoding is definitely required for league division for all the columns which are of object data type. Please remember this point learners encoding is definitely required for object data type. Without that you will not load it into the model. duplicate values. If they are then we need to check that. Okay, we are removing unnamed column because that is not required. We are specifying the access. What does in place equal to true mean? Permanent removal of that column from the data set. So now if I see my first column gets removed. So the first processing that I have done. Now let me check the null value. Are there any null values over here? Only the null value is in the output in the salary field. Right? Which is my output field. Agreed learners? Do I need to separate my x and y also? Yes. So the number of missing value in salary is 59. So 59 out of 322 observation with null values correspond to columns salary. Since we will use the lasso algorithm from the scikit, we need to encode our categorical also. Okay. Now, how do we deal with categorical data? What is the use of value counts function? It gives the category along with the frequency along with the count. Right? That's incomplete. It gives the category along with the count. So in league there is A and N with these categories. In division there is W and E western and eastern and again in new league we have A andN the American League or the National League. So if I separate this into a data frame do you see this data set data frame rather? This code is getting clear to everybody. Are we here till here? Okay. So, what are the different ways of encoding data? One is one hot encoding, other one one hot encoding, other one label encoding. What is the difference between the two? One art encoding adds column and then gives the binary output. If that column value is there, then it has one. Else all values are zero. Yes, a label and assign integers to each of the category. Yes. And label encoding is used when we have a ordinal data. One hot encoding is used when we have nominal data. That means there is no order. So over here we are using which function learners? Yes, it is now one hot encoding that now it will have instead of three six columns with zero and one value right so do you see this league a league n division e do you see now so wherever we have value of lee get dummies is one hot encoding and label encoder that is the function so is the output of dummies These dot head is also clear that now if these are the six the three categorical value get converted into six and wherever the value was there this is one or this is zero this is zero and this is one clear okay so what are we doing we are separated the output x numerical value we are dropping the output the league the division and the new league because they all are categorical data and everything gets is converted into float type. So the numerical columns are now clear. The input x numerical and the output y salary. Good chakra. Good. Now do we need to concatenate these columns with the original data? So we'll just take one of them since they it's it has binary category. Please try to understand. Since all of them have binary category, therefore it can be used for league N, division_W and new league N. Getting my point? So now you see all my inputs have become integer values, numerical values and that is what is required before you feed any data into the model. Now once my data is ready now we go in for testing and splitting of my input and output. Test size is 25% rest is training and I get my four outputs. Right? So lasso you know performs best when all the numerical features are centered around zero and have variance in the same order. Homocidasticity needs to be maintained. And if a feature v has a variance that uh that is orders of magnitude larger than others, it might dominate the objective function and make the estimator unable to learn from others. So it should not be that you know one of the columns have very high values and the other one low. This means it is important to standardize our features. We do this by subtracting the mean from our observations and then dividing by standardization. What is this concept known as a feature engineering? Okay, I've given you one hint. This is the concept involved in feature scaling and this refers to standardization that is scaling the factors uh with mean equal to zero and standard deviation equal to 1. Clear? To avoid data leakage, standardization of numerical features should always no be performed after data splitting only for the training data. Please try to understand to avoid data leakage, we will only and only use it after the splitting of the data. We saw that. So what is data leakage? You understood what is data leakage that it occurs when the information from the outside the training data set is used to create the model. This can happen if the data would not be available at the time of prediction is included in the training process. So data leakage can lead to optimistic performance estimates and models that fail to generalize well to new as well as unseen data. Clear? So which is the Python function? Which is the Python function to perform standardization standard scaler which creates an instance of the object fit. Fit trains the X-ray numerical data that is training means to find out the parameter to compute the mean and the standard deviation for each feature in X train list numerical. X train is your for each uh data set and the list numerical is a list of column names corresponding to the numerical features. Got it? What is the use of transform? It applies to the standardization of the training data which transforms each feature in the training data set to have mean zero and standard deviation one. Right? And then we only use the transform function on the test data. Now clear okay standard scalar is clear. It's the function to perform standardization. Standardization is one of the feature scaling technique where we take mean equal to zero and standard deviation equal to one. Anoj that point is getting clear. Fit is always performed on the training data that is numerical data. Till here also this point is getting clear that this refers to the list of numerical data and transform we are doing it on the train data that is how we are going to avoid data leakage now clear got it but when we talk about uh you know uh testing data are we going to apply fit function on it No, we are only going to transform it. So, it applies the same standardization parameters mean and standard deviation computed from the training data to test data and this ensures that the test data is scaled in the same data as training now clear. Okay. Right. So, practically how do we go about it? So we have imported the standard scaler from the skarn library and then we perform this function. So the explanation is given above. Right? So what are we trying to do? We are fitting the standard scala to numerical features so that there is no biasness in the data set. Now uh you know you'll see positive negative value because the sum of all these values should be zero. The mean of all them should be zero. clear to everybody? So the training data has 241 rows and 19 columns. Clear? Similarly, test one will only have 81 rows because of the 25% data and 19 columns. Clear? Now, there were missing values in the output data. How do we go about it? How do we go about it? Let's see. Yeah. So, what are we doing over here? Yes. What are we doing? We are taking the training model as well training data as well as the testing data and filling it with the median value. Wherever there is null value, it will fill it up with median. Not mode over here. Dr. Henry we see it practically that it's getting filled with median. What is the difference between mean and median? Besides that which one is affected by the outliers? Where do we use median when we do not want to get affected by the outliers? So now let's move ahead and understand the lasso model. So whenever we have to create how do we begin with the hyperparameters? If I want to start any of the models, how do we begin with hyperparameters? We will randomly assign any value and then start training on it. And to find out the actual meth value we do hyperparameter tuning. So we apply lasso regression on the training set with regularization parameter that is alpha equal to 1. So we begin with a very simple value alpha equal to 1. This value is commonly used as default and provides a good balance between maintaining model complexity and reducing overfitting. Okay. So by default whenever we have to do thing you can randomly assign any value to the hyperparameter alpha is equal to one maximum iteration is equal to 10,000 and now we are fitting the training data and this is now my new intercept coefficients the regression coefficients beta KN and the other coefficients because since they are how many parameters can you tell me how many parameters are there in this particular question. How many are there? 19. All the numerical values, right? So, we would have 19 coefficients. And do you do you see after applying lasso some of them have become zero? This one, this one, this one, and this one. Getting my point? So if I see the output these are my coefficients and some of my coefficients have become zero. Do you see? Then you might say ma'am what is negative0? It just puts the sign. There is nothing no term in mathematics as0. Clear? So what do we conclude? This is the intercept term of your lasso regression model. It represents the uh expected mean value of the independent variable when all the independent variables are set to zero. In practical terms, it is the baseline prediction when no other information from the variables is provided. So the lasso coefficients represent the relationship between the variable. Positive coefficients. Now what do we interpret out of it? Let's see that a positive coefficient indicates that as independent variable increases, the dependent variable also increases. That's the positive relationship. A negative coefficient indicates that an independent variable increases, the dependent variable decreases. The magnitude of the coefficient shows the strength of the impact. Now clear? So the magnitude of the coefficient shows the strength of the impact. A larger absolute value indicates a stronger effect. Lasso regression is known for its ability to perform feature selection. So ultimately the values have become zero. So now my 19 features have reduced two. So I have reduced these two. Then I have also reduced two above and one in the last. So three features get reduced. So 19 gets reduced to 16. Right? When we have taken lambda equal to 1. All right. Right. But in this model however it seems that none of the coefficients are exactly zero suggesting that all included variables have some impact on the models through some impacts are those some impacts are very very small. So basically four features yeah okay four feature. So now how do I evaluate it? I'll calculate my mean square error for the training data and then the for the testing data. This code is also getting clear that helps me to decide whether it's overfitting or underfitting. So now you tell me is it an overfitting model or an underfitting model testing area is testing error is more than the training. So is it underfitting overfitting? it's overfitted and if I do the R square over here so the value is very very less. So to better understand the role of alpha the regularization parameter. So what is alpha known as the regularization parameter? Plot the lasso coefficients as a function of alpha. Maximum iterate are the maximum number of iterations. So here we have np.linsspace function. What does the np numpy lindspace function do? that the now the values of alpha will range between 1 to five and with 10 equidistance value. So now what are we trying to create lin space function do we understand of numpy? So now if I say now I will start understanding the range of the values range of the values from 0.01 01 right to 500 and the value is still 100 right so over here these are the values or the array created of alpha the maximum iterations are 10,000 and now I will try to run or fit on each of these alphas and then look at this beautiful graph Simon specially for 2. So the value of the alpha goes from 0.01 to 100,000 and we see as the value of the alpha is increasing some of the coefficients actually become zero. You see this? So that is why L1 regularization is also known as feature selection. Agreed? Okay. So, have you understood this graph now? So, we create now the best value of alpha. How do we find out the value of alpha? That now we will use cross validation to find out the best value for alpha. Lasso regression comes with builtin cross validation. So, beauty of lasso is that you can combine lasso with cross validation. Alpha again ranges between 0 to 1,000. CV is equal to 10. Maximum iteration 10,000 and num n jobs is equal to minus1. Can you anybody tell me what is the use of this parameter? Okay, I need to repeat that. Okay, Simon, I'll do that. In the meanwhile, tell me what is the use of n jobs learners? See, once we have checked on on the training and the testing process and we found out that it is an overfitting model, the test error was too high than the others. So now we have to find out the uh you know the best value of alpha. Okay. Right. Let's reduce the maximum course and which is the you know ma best alpha value which has come out over here six comes out to be the best uh regularization. So get the best alpha regularization strength selected by the cross validation clear. So it was not the value of alpha was not one but it will come out to be six. And now I will create and fit the lasso model. Taking this value. Now I will fit it. And now check on my training and testing error. So have they improved? Well, the test error is still higher than the training, right? But it shows a better performance on the tuned parameters. So zero coefficients such as runs, RBI, CAT and all these new leagues have several coefficients shrunk to zero. This indicates the lasso has deemed these features less important, irrelevant for the data set and the nonzero coefficient give us the detail relationship. Clear? Moving on to ridge regression. Ridge regression is another technique for handling uh overfitting of the data that's known as L2 regularization. Here none of the features will become zero rather they will values will become less. Right? So where it is used why where when to use that and when to use the other one get the best alpha selection validation. Now what is this added below library? Okay. So when is ridge to be used? It is useful for handling multi-olinear data where two predictors independent variables are highly correlated to each other. Right? And the second important reason AJ you are absolutely correct Ajay Kana you are absolutely correct but the second reason where we use ridge regression is that it is useful for handling multicolinear data where two predictors independent variables are highly correlated to each other. Now getting my point. Collinearity refers to a situation where two or more predictor variables in multiple regression model are highly correlated meaning they have a linear relationship and this makes it difficult to determine the individual effect of each predictor on the target variable. Right? So that's why the penalty term is added. the regularization terms are added. Getting my point? So now let's see ridge again we have a ridge CV function which can do so on the same data we will first apply the alpha fit function and get the intercept value. Yes reach ridge needed to use if you want to keep all the important features. So I've not been running the code. So I will get error. So sorry for that. I'm not running the whole code. Going back and running it. So now let's move and then evaluate the model. Again testing error is more than the training. So overfitting is there. And do we see the coefficients? Now none of them become zero. Now we can use cross validation again to find out the best value. These are the values. I have used rich cross validation and in this case the value comes out to be 204 right then we train on it and these are my new coefficients these are my new coefficients right all values have reduced if you compare with the original all values have reduced now so the negative error relationship is with errors division W they show the negative relationship. So the coefficients with highest value suggest cumulative career statistics total runs the total hits are most important and showing a positive relationship with salary. Division_W the only feature which is notably high negative coefficient which indicates that being in western division is associated with negative effect on the target variable. So do not be in the uh western division ears assist and new league and these features have smaller. So are you understanding this is how you're practically supposed to run the code and write down the observations over here and if we see it graphically that in this case the values will not become zero but near to zero. Clear? So have you understood the concept of regularization? How it helps in preventing overfitting? Whereas L1 helps in feature selection and U ridge helps in uh reducing uh the values if all are important. And of course this analysis helps in understanding the relative importance and influence of different aspects of a baseball player statistics on the predicted target. Clear? Now moving on to the next concepts that is optimization. Right? Model optimization. Now what does the term optimization mean? What is the meaning of model optimization? Optimizing resources. We want minimum number of uh code less number of time to giving the best. Yes. How the model performs efficiently? Absolutely. So if we talk about the journey of data science or analysis see every thing uh on if you if you are doing a normal Excel maximum minimum drawing graph that's also analysis and even when we are doing machine learning that's also analysis but where is the difference are you able to get this um graph that I'm showing where is the difference can you tell me If we already have a data set. So this is the side for that we are going to describe you know about the uh historical data that we have that is simple finding out hindsight in the data. When we try to find out why that happened that becomes diagnostic analysis that is insight. And if we move further what is going to be predicted sorry or what will be the output that becomes predictive analytics and similarly why that you know we are predicting that output whether that is going to be good or bad that is known as prescriptive analytics. So this is where we start from inside and now we want to move to the foresight. How does it happen? When we start optimization of the model. Are you understanding this point? Very very important graph from one analysis to another to explain you in more lame language. Suppose you have cold and cuff right and you go to the doctor. Doctor sees okay you know your nose is running you have cold and cuff he says you know um you take this normal predict uh you know medicine and you know do a lot of steaming and take this medicine you'll be fine but that was his an you know analysis which he done to just by seeing you so then you come back you are not fine then you go back after a week to him you say no I have fever also I have other thing then he thinks that the fever he's not getting well so maybe he has some kind of an infection then he prescribes you certain blood test. So based on the blood test he would give you certain medicines but you are still not fine your further analysis is needed. Maybe you have a chest infection now or something and maybe you want to go in for an MRI. So that's how the level of analysis increases. All right. So basically what do we want to do and what do we want to optimize in this whole machine learning process that if we have this input data that is the training data including the target output. Of course if it is supervised learning we would also have the target output. This is my model right and output prediction calculated by model. So when we try to compare the predicted output with the actual output, therefore we are able to calculate the error and the loss function. Are you all there with me? Then we are able to calculate the error and the loss function. And using this error and all loss function we can use optimization method to sorry further reduce the values. How we can further reduce its value. Getting my point right? So that's how we are trying to improve on the error part of it. And then another way to optimize the model is through hyperparameter tuning versus model training. What is hyperparameter tuning? That to find the best hyperparameters which give us the most efficient results. And this is a very very important slide because it gives the whole crux of the data uh of all the points that we have studied till now. Please look here. This is my data set. Feature engineering is performed before. What are the different aspects of feature engineering? And why is feature engineering required? Tell me what are the different aspects of feature engineering and why is it required to prepare the data to fed into the model not tune we will say to prepare the data prep-process the data to fed into the model and what are the different types of feature engineering so what are the different process encoding scaling transformations Right. Dealing with the outliers. They all come under feature engineering. First point, first step is clear. Then what is happening in this particular step? Can anybody tell me what is this step known as? Splitting. But what what are we splitting and why are we splitting? It is splitting of the data set. Why are we splitting it into three parts? Normally it is training and testing. This is this step is known as cross validation that it divides the training data into training and validation and a separate test data. Second step is splitting of the data. Yes, but we generally keep using it for cross validation by taking multiple samples. Got it? By taking multiple samples. Now this is an iterative process. This keeps on getting repeated. So the training data we build models, train the results, right? And finally once the training is done then on the validation uh thing we do the training results do hyperparameter tuning and keep on repeating this iterative process. Finally, the best model which is selected is then used for testing the output and compare the results. Do not get confused. See, every concept that we study has its own role and concept. Right? So, where will the regularization fit over here? If I compare the result, my testing result is much higher than the training result. Then it is an overfitting model. Right? We always start with the basic mega then we will use the model over here as ridge or lasso and then do the comparison. Now getting my point everybody. All right. And the two types of hypertuning methods are grid search and random search. Grid search is the one in which we it's an exhaustive re search technique where we have different hyperparameters one such as a b and c it's not necessary that you know we have understood that regularization as one hyperparameter we have other models like decision tree which have many hyperparameters hyperparameter to xyz value and then we try to take all the combinations and then see which one works best for a model. Disadvantage of this method is it's an exhaustive method uh computationally and timewise it is expensive. Clear? And random search that randomly we will select any hyperparameters and do training on it. Clear now and we have seen so cross validation is a very very important technique. Why? Because it helps to split the data into training, validation and test set. Please make this point clear. Cross validation technique will almost be used in supervised learning because it will divide the data set into three parts that is training, validation and testing set. Useful if you want to have a metric on how well your model is performing. Clear? So once the grid CV search returns with tuned parameters, build the model using this set with the tuned parameters and test the new model with the test set. Clear? And this is another uh you know um slide which I always uh show it to my learners even in my feature engineering class. Now are you able to interpret this particular slide everybody? Yes. Right. So we again have a data set, retrieve the data set, perform pre-processing, wrangling, feature extraction, feature engineering, train the model, then model evaluation, hyperparameter tuning and reiterate the process. But this slide gives a much better picture of each step. Got it learners? So getting back to the file. So this file is pretty long. You know last week also we have done and I think so this session also we would be using. So model optimization means hyperparameter tuning is the process of finding the best settings for the parameters in the machine learning model. Hyperparameters are settings that are not learned during the training but are set before the training process begins. Hyperparameter tuning involves trying different combinations of the hyperparameters and evaluating the model's performance using validation techniques. So the two techniques that we can use is grid search. So how does it work? How do we do the grid search? It systematically works through the multiple combinations of hyperparameters. It performs an exhaustive search on a specified parameter grid. So how do we go about it? Please try to understand the different steps. The first step is we define a parameter grid that is defined with the help of dictionaries because they can be more than one hyperparameters in the model. Combination evaluation. The algorithm evaluates all possible combinations of these hyperparameters. Model training. For each combination, the model is trained, evaluated using cross validation. And then we have the optimal parameters combining yielding the best performance. Highest accuracy is chosen as the optimal set. All right. And how does the random search work? It defines the parameter distribution. Specify the distributions and ranges of the hyperparameter. Random sampling, random sample combinations of hyperparameters from these distributions. Model training for each sample combination and optimal parameters. All right. So now on the same hitters CV, we are running the code. I might get a lot of error. It will let me run the x train. Okay. So this is the grid search cross validation. Here it we are creating a cross validation object with repeated kffold function. Number of splits is equal to 10. Number of repeats is three. Random state is equal to 1. Are you now there with me? Everybody other code else code is running then we create a grid or a dictionary and then try to find out the value. So still it's giving error. Is it running for everybody else? Oh well I've tried to do a shortcut but shortcuts never work in life. Value y input consist of nan. Okay. Yeah. Are you getting it? I'm not running the code. So this point is clear. So the best parameter grid of alpha now it's coming out to be 0.9. Is it the same with everybody? So then we create and fit the ridge regression model to avoid training data with the optimal alpha. So here we have the ridge function results into best parameters alpha. fit them through the training uh input as well as output and find the mean square value that comes out to be 341 and R square comes out to be 0.38 which suggests a moderate fit the model captures some of the variability in the data but not a large portion and now if I see the coefficient this is how the coefficient values are now let's move on to the next concept last concept by creating pipelines. So skarn pipelines are nothing but an automation of the model fitting and data transformation steps for training and data set. So what are we doing at the moment when we upload the data we separately use fit and transform do encoding do scale standardization. Now we will try to implement it with the help of a single pipeline. So what do we do? If we have test data, test labels, we would perform feature scaling standard scaler initially. Then if feature selection or extraction, dimensionality reduction is required, that can also be done. And then the model data can be fed to the different models. So how do we go about it? We are very clear in supervised learning. We have a training set and a test set with class label. Right? So in the first step we will try to fit the training data that is missing in the test set because we do want to avoid data leakage. Right? The fit will ne fit function will never ever work for the test data. Right? And therefore we have the fit transform function over here. This is the pipeline right? So over here you know we u do the transformation such as for for scaling dimensionality reduction and then do the predictive model and over here it is fit and transform for the test training data but for the test data it is only the transform function. clear and the predict will only be for the test data. Now let's see how do we practically implement it. So why sklearn pipelines? Pipelines provide an organized approach to managing your data prep-processing modeling the code. They combine the pre-processing and the modeling steps into a single streamline process. Cleaner code pipelines eliminate the need to manually manage uh training and validation data at each pre-processing step reducing the clutter and the complexity. Fewer bugs by bundling steps together. So bugs are few. It gives a much more cleaner uh code and easier to productionize. that is they simplify the transition from a prototype to a model to a scalable deployable solution. Clear? So we have skarn is a beautiful library. We just need to uh you know create a class or we do import the pipeline function and we will create an object of pipeline class whatever steps that we want to perform. Do we want to use any kind of memory or verbose clear? So a pipeline is a sequence of data transformers that can include a final predictor also final output also. It lets you apply reprocessing steps to your data in order and u optionally end with a predictor for modeling. Each intermediate step in the pipeline must have fit and transform methods while final step only needs the fit function. Memory stands that you can use cache memory for these transformations to make the processing fast using the memory argument. The pipeline main goal is to combine multiple steps that can be cross validated together and have their parameters adjusted. You can set parameters for any step by using it name followed by a underscore. You can set any of the parameters by default followed by a underscore name and the parameter name and you can replace any steps estimator with another estimator or remove transformer by setting it or pass through or none. So now over here they are using a different data set that is the one with ocean proximity that is housing one I think. So I gave you for testing also. So it is little different from original because one of the parameter is categorical ocean pro uh you know object. So how do I check that? What are the different uh uh you know categories of this particular function? How do I check that? So we have this categorical data. So what are the different encoding techniques? either we use one hot encoding that is better because there is no order in this. So we'll not use label encoding. We'll use get dummies function to perform it. Value counts function is a function in Python which gives the categories along with their frequency count. Very very important function. What is happening in this particular code of line number 46? Come on learners. Y is getting the output value and then we are separating the X. Yes, we are defining input and out. What is happening in the next step? Line number 48. What is happening in the next step? Splitting of the data. Very very important. And then we see the info and sum over here. Right? So where are the null values? Where are the null values? Total bedrooms has 10. 62 null values data is huge 14,448 right so definitely cleaning is required removing of the null values encoding is required right so rather than now performing each of these steps separately can I use the concept of sklearn pipelines first I will feed in my for training data performance feature scaling, feature extraction and ML algorithms and then work on the test data getting my point learners. Okay. So what are the steps that we want to perform before building the model? Feature engineering steps. So feature engineering part is used as skarn. Even the other parts can be combined as pipeline. First missing value treatment imputation 162 missing values in the total bedrooms uh numeric data column. Then we have the dummy variable creation for categorical data and finally standardization of the numerical value. So how do we go about it? These are my different libraries that I will impute import and most importantly is the pipeline library. getting my point? A very important concept in skarn is the column transformer that importing column transformer class to apply different pre-processing steps to different subsets of the features. So column transformer has the capability that encoding is applied only on categorical data not on numerical data. Getting my point? So it will only be applied on categorical. So let me explain this. Okay. So this is how the fit transformation happens. But if these are my features which is a combination obviously of numerical, categorical and others. So column transformer has the beauty that for numerical columns only scaling will happen. For categorical one hot encoding and maybe others can simply pass through. It's not necessary that some operation needs to be performed on it. And finally we can get a transform data in very one go and it can be combined together. Got it? Column transformer concept is clear to everybody. So here we go about implementing it. So now when you read about column transformer you will understand that it allows you to apply different pre-processing steps to different subsets of the features in your data set. This is particularly useful when you have a mix of numerical and categorical data that require different types of pre-processing. Column transformer ensures that each column or group of columns gets appropriate transformation before combining the results for further processing or modeling. Okay. So now step one, let's store the names of the numeric and object type variables differently. So how are we going to do? And now we have splitted the training and the testing data and we are exclusively selecting the data types include object exclude object and result. Are you understanding this code learners? So housing_cat will only have one since this data set has only one categorical data that is ocean uh proximity and the rest are all numerical in nature. Clear. Now the next step says to set up skarn pipelines for numeric variables we need to perform missing value imputation and then standardization. So how do we go about it? Now we will create a pipeline object using this pipeline function num pipeline. of a numerical data. We will import uh you know do the missing value imputation and standard scaling. Is this point