Why we should use Momentum Based Gradient Descent | Accelerate Deep Learning Training Process

Hello and welcome. I hope you are having a great day. In this video, we'll understand an optimization technique called momentum based gradient descent. So this is used to accelerate our optimization process in Okay, so this is provide acceleration to the optimization process in deep learning. How does it do? What is the meaning of momentum? And how are we doing it mathematically? We'll understand all of this in this video. Let's begin. So first of all, let's revise what are the problems with non-convex loss. So we have flat regions. This is the main issue. Say for example, we have We first of all, we know that in deep learning, we have non-convex real loss loss function surface, which was not the case in like machine learning. Non-convex means which is which has many local minima. I'll show you an example. So here the flat region means say for example, if you have uh like this and then there's one local minima like this. So there's a flat surface here. The gradient in this flat region will be very very small. So if the gradient is small, my step size is is is dependent on the gradient. So my convergence towards this local minima will be very very slow because of this region, this flat region. So how do I move around move across this fast region this flat region faster? That's the whole idea of momentum. In momentum, what do we do that we will see the history of gradients that we have seen and we'll use that history to build our momentum. How do we do that? We'll see that. But before that, let's understand the non-convex. I'll just do a revision. We have discussed this in the optimization techniques first video. But I'll just show you again. So this is this is the main problem, the passing through the flat region. If you have a flat region, the gradient is small, how do we move past that? So where the gradient is near zero offering minimal guidance to the optimizer. And then we have ravine-like structure also, which is like steep on one side. So, multiple dimensions are we talking multiple dimensions? So, in one direction it is steep. In one direction it is kind of flat. So, when it is moving, so it is moving the gradient has to weight has to weight has to move in all the directions. So, one side it will just move very fast. And one side it is flat it will move very slow. So, because I have to cover the flat region also because of this until unless it crosses this flat region, till that time it will just keep on in the because of the ravine it will move fast in one direction and move slow in one direction. So, it just keep on oscillating back and forth around the even if it is reaching the local minima. One side it will move fast because of the ravine and one side it will move slow. So, it will until unless it has covered all the flat region, it will keep on oscillating in that region. Okay? I hope that is that makes sense. Some it gives you some view of what is the problem with the ravine-like structure. And then we have saddle points. Saddle point is like maximum with respect to one direction and minimum with respect to one other dimension. So, it is not a local optima, a local minima or local maxima. It is maximum with respect to one minima or with respect to other point. Other dimension. We have discussed this earlier, too. So, I'll show you what a non-convex surface looks like. So, this is a non-convex surface. So, what I'm talking about is this say for example you have this flat region. You start from here. This region here it is very flat. The gradient is like almost zero. It is like a flat region. And then here you will just go down and it will go you will go to the global minima here, whatever is the Here also you see these are the flat regions. If I start from here I move very slowly and then I will come down. So, these flat regions are where where I move very slow. Can I accelerate this? Can I accelerate this movement? That's the whole idea of momentum. Can I use the history that if I'm moving very slowly in one direction only? Means I'm moving in one direction only. So, can I take bigger steps? That's the idea of momentum. And here you can see there are so many local minima. Here there's one local minima here, there's one local minima here, one local minima here. Depending upon where you start, you will reach there, okay? Here also you can see one local minima is here. One local minima is here, here, here, here. So many are there here. There's one global here also. So, depending upon where you will start, you will reach that. So, for example, if I start from here, I start from here, I'll reach this. I start from here, I may reach this point. And say, "Okay, wherever you are reaching the local minima, the surface will become flat." You know what happens? This is a flat surface. Let's say, "Okay, the gradient is become almost zero. I have reached the minima. I cannot go beyond that." See? So, that's the you can get stuck in a sub-optimal. This is a sub-optimal. This is not This may not give you a optimal. So, you have to at least somewhere here, somewhere like here, to get the optimal solution if you're not reaching the global minimum. So, that's the whole idea. That the optimal which I'm reaching, it should not be a sub-optimal. It should not Wherever I reach, it should give me a good generalization. The model should be able to generalize. Okay? So, a revision of saddle point. So, you see here, this is the maxima with respect to Y axis and respect to the X axis here. This is the minima. This is the curve is like this. You can see this direction. This is a saddle point because it's like a you're sitting on a horse. Let's say, for example, there's a person here like this. Okay? And they're sitting like this on a horse. And here there is a horse like this. And he And he's having this. This is because you're riding a kind of horse here. That's why it is called saddle. So, your sitting point is here and this is one point is on the horse. Your legs are on both the sides. So that's why it's called saddle point. So it is maximum with respect to Y axis and minimum with respect to where the sitting will be down. So minimum with respect to one axis. So it has not reached it so the gradient will be zero. But it is not a local minima. Okay, and we get stuck in such regions. How do we And the momentum will let us pass through. But the momentum will pass through this saddle point. That's the whole idea. So we'll see how the momentum is built. Momentum So momentum introduces velocity. So momentum is the term has been taken from physics. What is momentum? Momentum is mass into velocity. Okay, so the I'm taking the velocity term. So we'll do it the mass is assuming mass is constant. So the velocity will be equal to my momentum. In which direction I'm moving? If I'm moving in the same direction, I can take larger steps, I can move fast. So that's the analogy it has taken from physics. Not a direct analogy, but that's where it is like I can build a So if I'm moving in the same direction means I'm building momentum. I can move faster now. In that direction. Okay? So it accumulates velocity in directions of consistent gradients. If I'm moving in the same direction, consistent gradients, then I can take larger steps and I can move faster. And cancel the direction which fluctuate. If I'm moving once like this once like once I'm moving like this another time I'm moving like this then I'm moving like this. So basically I cancel out. I'm not doing making much change here. Okay? So it just cancels out. It won't build a momentum. But if I'm moving in the same direction, in the same direction, so I can take a larger step. That's the whole idea. I told you like in a flat region. So if I'm moving in the same direction for many steps, so I can take a larger step. That's the whole idea of momentum. So first what we were doing, instead of moving purely by gradient that whatever is the gradient I'll just take randomly one stochastic I'm doing like I'll just take the gradient of the mini batch and I'll just move in that direction, right? But what am I doing? What we will do that instead of doing that we will do something we'll accumulate the previous gradients that is velocity based. So we have speed plus direction also now we'll take earlier the gradient give us only the speed now we'll use the direction also. I'm moving in the same direction then I'll move faster. So it's getting given by this formula. Okay. So what is the formula here? That VT is equal to gamma times V of T minus 1 plus eta times G of T. So this is my old gradient here. This is my old gradient. That is remains same but we are using the past velocity V of T minus 1 and I'm multiplying that with gamma and I'm using that also. Earlier only this term was there I have added one more term. With this gamma so this is the historical velocity what I have and this is my current gradient. Okay and we'll give some weight to this gamma gamma is generally less than one so this is between zero and one. So say for example gamma is 0.9. So to the history in which direction I have been moving till now what has been accumulated so I'll give 0.9 times the history and plus eta times whatever is eta is the gradient. So I'm giving 0.9 weightage to the history that I have. And plus the gradient. Okay and this is called the momentum coefficient typically 0.9. Okay and then we'll update the parameter with the older weight minus the velocity. This velocity is also having the gradient. The accumulated gradient over a period of time with the velocity in which direction I'm moving. Okay. So, let's expand this and then you will get a sense how it is adding up. Mhm? So, the size of my step depends on how large and how aligned are sequence of gradients. So, if I'm moving in the same direction, means I can take a larger step. How? Let's see. Say, for example, let we initial velocity zero. So, what will my first velocity? So, eta times gamma times V0 plus eta times G0. So, this is zero, so this becomes zero. So, just eta times G0, my initial gradient. Then, come V2 will be gamma times V1 plus eta times G1. That will be gamma times then I will replace G1. What are the value of G1? Eta times G0 plus eta times G1 as it is. Then, what will be V3? Gamma times V2. V2 V2 is gamma times gamma times eta times G0 plus eta times G1. So, I'll just replace this gamma times V2 plus eta times G2. So, if you expand this, gamma into gamma will be there. Gamma times gamma times and then this will be there. So, if you do the math, you'll see that if I take eta outside, it will be like gamma square G0 plus gamma times G1 plus G2. So, as we So, the current one will have the highest one. This gamma is 0.9, gamma square will be 0.81, like that. So, the older one will have less weightage, but it will be that will be the momentum the So, older gradients I'll give less weightage, but it will have the recent one will have higher weightage. But, each one will keep on adding and that way we'll see that if all of them are adding in the if all of them are same, for example, so this is my generic gamma K minus one for the Kth velocity. And then this is your generalized. Say, for example, all my gradients if I'm telling if I'm moving in the same direction, all my gradients are in the same direction. G1 equal to G2 equal to G0 equal to all of them to GK. And my K tends to infinity. So, it will it become eta times G. So, I'll take all the G same. These all Gs are same same same. I'll take that also out. What remains inside? 1 + gamma + gamma square + like that till infinity. So, what is this series? This kind of a geometric progression. GP. You're multiplying each term with gamma. And here gamma is less than one. So, what is the sum of this series? So, if I tell eta into G as it is, what will be the sum of this series? So, generally in if the sum of the series if if the multiplication coefficient if it that is less than one, in geometric progression how do we get? That is A by 1 minus R. That is your multiplication coefficient R. Here we have gamma. So, this will be A by 1 minus gamma. And here A is What is the A? A is the first term. So, this will be like 1 by 1 minus gamma. So, this will be eta into G. That was the earlier. Anyways, we if all the gradients are same, we'll take this step eta into G. Okay, eta into G G step and then this this will move only eta into my gradient. That was my old term. Wait. But, what is the benefit of accumulating all the gradients? Okay, what is benefit? So, the momentum algorithm always observes gradient G. If that is the case, that's what I was discussing in that direction. So, the terminal velocity will be reached by this is this is what I told. This is the magnitude of this where gamma is less than one. And if gamma was 0.9, so 1 minus 0.9 will be 0.1. So, if you divide something by 0.1, that is like multiplying something with 10. That's what. So, if we are multiplying this with 10 means I'm taking 10 times eta will become like 10 times. I'm taking 10 times larger step now. Because of the and this is then it moves multiply the maximum city by 10 relative to the gradient descent algorithm. So earlier if I'm taking just beta times and if all my gradients are going in the same direction, this will give me a 10 times step size. And that's what momentum means. I'm using the momentum to take a larger step. I hope this is clear. Okay. Simple geometric progression. That is what is doing the the momentum. That is what is building the momentum. Okay. This is an example. So for example, here I have started from this is the non-convex surface base function generally used for testing on convex gradient descent how the gradients functions in in a non-convex surface. This is a very common function for testing. So this red one is my stochastic gradient descent. You see this is this this is stuck this is stuck here. And the blue one this blue one here here you see it has reached this is the global optimum. Global minimum. It has gone randomly because I'm doing stochastic gradient descent plus momentum. So somehow it has reached the after fluctuating here. It has reached the the stochastic one was stuck here itself. It didn't go beyond a point. And this is what because it might be stuck in some flat region after it was not able to build only. And then the learning stopped after iterations. If I continued more iterations or epochs, then it might have done but whatever epochs I have run this in those many epochs only my momentum was able to do. So basically I have accelerated by the time stochastic gradient reached here my momentum base was able to reach here. So it took same number of iterations and stochastic gradient was not able to reach the global minima. So here it is able to reach the global minima but it can reach the local minima also. So that's how I'm accelerating, taking larger steps. So, what are some drawbacks or limitations or limitations of this? So, it can be like a heavy ball. So, it is like taking momentum. It is If it is going down a steep like this here. So, it can build a momentum and it can overshoot the minimum. This is a minimum. I can overshoot this. I can go here. Next step can be here direct. Because it is a momentum. It has It can take larger steps because of it is moving in the same direction. For a long time it has been moving like this like this like this. So, it can overshoot the minima. Okay. So, this step can go from here to here. So, that is one problem. That is a limitation of momentum based. And another another issue is that it does not adjust the learning rate based on the importance of specific features. As I mentioned earlier, so there can be features which can be sparse. And there can and there are normal features which will be like dense or frequent. As in the housing price example. So, housing if I want to predict the housing prices and you say if I have a feature like proximity to mall, if there is a mall nearby the house. So, very few houses will have this feature. Rest of them it will be turned off. It will be zero only. So, very few houses will have some mall nearby. So, these features they need special handling because the number of times they will the gradient will be updated. Those many entries will be very less. So, in this dimension the gradient will not move. So, how do we handle this? So, this momentum based does not handle this specially. And that's what will be done in the next one which is called AdaGrad. Where I will handle the this sparse features in a special way. How do we do that? That we will see in the next video which is adaptive gradient. How do we handle sparse features? Okay. So, that's all for this video. Thanks for watching this video. Have a great day ahead and bye for now.