Overfitting and Regularization in Deep Learning

Hello and welcome to my channel. In this video, we are learning about overfitting and regularization in deep learning. So, without a further ado, let's start the video. So, what is overfitting? When a neural network learns the training data so well, including all the noise, details, and random variations, which causes the poor performance on new data, is called an overfitting. For example, let's imagine a student preparing for an exam. Let's say student one understands the concepts and is able to solve new questions. Whereas, student two memorizes all the previous exam questions, but does not understand the concept behind it. Therefore, in an exam, if there are new questions, the student one performs well because he or she understands the concept to solve new question. Whereas, student two performs poorly because he or she lacks in understanding the concept. Therefore, here student two is similar to an overfitted model. Characteristics of overfitting. We can get high training accuracy because the model performs extremely well on training data. There is poor generalization as the model cannot adapt new examples. There is low test accuracy as the model it struggles with the unseen data. Why does overfitting happen? Excessive training can lead to memorization. Small training data set. If there is small training data set, it is obvious that the model has limited examples. Therefore, the model it starts memorizing all of them. So, due to these reasons, overfitting happen. Now, let's see what is regularization. It is a method that is used to prevent overfitting by decor racing the neural network from becoming too vague. Regularization helps the model learn patterns instead of memorizing the data. Importance of regularization. It improves generalization as it performs better on unseen data. It prevents overfitting because it avoids memorization of training data. It produces robust models because it is less sensitive to noise and outliers. Now, let's summarize overfitting versus regularization. So, overfitting memorizes the training data. But in regularization, it learns the general patterns. In an overfitting, there is a poor test performance. Whereas in regularization, there is a better test performance. In an overfitting, there is a complex model behavior. Whereas in regularization, there is a controlled model complexity. There is weak generalization in overfitting. Whereas in regularization, there is a strong generalization. For an overfitting, it required high training accuracy. Whereas in regularization, there is balanced accuracy. So, these are the difference between overfitting and the regularization. Regularization techniques. Regularization techniques help to prevent overfitting by reducing the complexity of a neural network and improving its ability to generalize to unseen data. Regularization techniques are L1 regularization, which is also called lasso regularization. And next is L2 regularization, that is ridge. And next is elastic net regularization. Now, let's discuss each of them. L1 regularization or the lasso regularization. L1 regularization adds the absolute values of the weights to the loss function. So, mathematically, loss equals to original loss plus lambda into summation of absolute values of the weights. So here, lambda is the regularization parameter and W is the weights. For example, if we consider three weights, that is W1 equals to 5, W2 equals to 2, W3 equals to 0.01. And during training, W1, it goes like this, 4 to 3 to 2. Whereas W3, it goes to 0. That means during the training, L1 regularization adds a penalty to the loss function. As a result, the optimizer gradually reduces the weights. Large weights become smaller, while very small weights may eventually become exactly zero. When a weight becomes zero, the corresponding features no longer contributes to the prediction, effectively removing it from the model. Hence, before regularization, W1 equals to 5 and W2 equals to 2, W3 equals to 0.1. But after L1 regularization, W1 it becomes three, W2 it becomes one, and W3 equals to zero. That means feature three is removed. So many weights become exactly zero. So this is the L1 regularization process. Now, let's see the advantages of L1 regularization. It creates sparse models. It performs feature selection automatically. It reduces the model complexity. And the disadvantages are, it is able to remove useful features if lambda is too large. L2 regularization or ridge regularization. This technique uses squared weights instead of absolute values. So, mathematically, loss equals to original loss plus lambda into summation of w square. Here, lambda is regularization parameter and w is the weight. For example, if weight one equals to five, weight two equals to two, weight three equals to one. Then with L2 penalty, we do like 5 square plus 2 square plus 1 square. That is 25 plus 4 plus 1, which is equal to 30. So, large weights receive much larger penalty. So, how it works, let's see. So, five it will be changed to four, three, and then two. So, L2 typically shrinks the weights, but does not make them exactly zero. So, this is how the regularization process happens. The advantages of L2 regularization are it produces a stable models, it reduces overfitting effectively, it is most widely used techniques. The disadvantage is it does not perform feature selection. Now, let's summarize L1 versus L2 regularization. In L1, there is In L1, we use absolute values of weights. Whereas in L2, it uses w square. In L1, there is a feature selection. Whereas in L2, there is a weight shrinking. In L1, many weights become zero. Whereas in L2, weights become smaller. In L1, there is a sparse model. Whereas in L2, there is a dense model. So, these are the difference between L2 and L1. Now, let's see what is elastic net regularization. It is a regularization technique that combines both L1 regularization and L2 regularization into a single loss function. It helps to prevent overfitting by combining feature selection capability of L1 regularization, weight shrinking capability of L2 regularization. So, why do we need elastic net regularization? Suppose we are building a model to predict house prices by using 50 features. For example, size of the house, parking space, number of rooms, number of bathrooms, age of the house, distance from the city, garden area, nearby stations, or many other features. Here, not all the features may contribute to the prediction. Because some features are very important, some are moderately important, and some are not so important. The L1 regularization problem is L1 tend to remove features by making some weights exactly zero. Therefore, 50 features, it may change to only 10 features. Hence, with L1 regularization, sometimes useful features may also be removed. The problem with L2 regularization is L2 keeps all the features, but reduces their weights. Therefore, 50 features, this is still present. Hence, with L2 regularization, unnecessary features may also remain in the model. Therefore, the solution is elastic net. The elastic net combines the strength of both methods. So, for the 50 features, they remove the unimportant features, and it reduces the remaining weight. Therefore, we get a balanced model. Mathematically, general elastic net loss function can be written by this equation. Here, original loss equals to prediction error. Lambda is the overall regularization strength. Alpha, it controls the balance between L1 and L2. W is the model weights. Now, let's see how elastic net works. Suppose a model has the following weights. W1 equals to eight, W2 equals to five, W3 equals to 0.3, W4 equals to 0.01. With L1 effect, L1 pushes very small weights towards zero. That means W4 is 0.01. It leads to zero. So, feature effectively removed. Whereas, with L2 effect, L2 reduces the large weights. That means W1 has eight. It will become five. And W2 has the five. And it will become three. So, weights become smaller and more balanced. The final results are large weights. They are reduced. And the tiny weights, they are removed from the model. Hence, this results a simple and more generalized model. Advantages of elastic net. Elastic net combines the benefits of L1 and L2 regularization. It combines both feature selection and weight shrinking. It improves generalization as the model performs better on unseen data. It reduces overfitting because it prevents the model from memorizing the training data. It handles high dimensional data as it works well if there are many input features. So, these are the advantages of elastic net regularization. Disadvantages of elastic net. There are more hyper parameters that increases the model complexity. It has higher computational cost compared to compared to using only L1 or L2. Summary. Okay, this is all for overfitting and regularization. We understand that overfitting occurs when a neural network memorizes training data and fails to generalize to new data. Regularization is a technique that prevents overfitting by controlling model complexity. The ultimate goal of regularization is to improve the model's ability to perform well on unseen data. L1 regularization encourages sparse models by driving some weights to zero. L2 regularization reduces large weights and improve stability. Elastic net is a hybrid regularization technique that combines L1 and L2 regularization. Okay, this is all for today's episode. I hope you enjoyed this video. If you do, please do not forget to like, share, and subscribe to my channel. I will see you in the next video. Thank you for watching. >> [music]