Pytorch Quick Tip: Using a Learning Rate Scheduler

Key Takeaways

[Music] welcome back for another pie torch video in this short video I want to show you how to use a learning rate scheduler to make learning boat easier and better so there are a bunch of different schedulers and I think all of them can be used I'm gonna show you one in this video that I think is pretty cool and so first of all I have some code here some pretty basic code that's just loading a pre train model and doing some fine tuning by setting yeah so freezing some layers and we're only gonna train a last layer a linear layer if you're unsure about how fine-tuning works you can check out my video on that but it doesn't really matter how the code looks like let's you say you got some code that's working to train a neural network so what we want to do is we want to go down to where we've created our loss and the optimizer in this case cross-entropy loss and atom optimizer we're gonna write after we've defined optimizer we're gonna create the scheduler we're gonna do optimal learning rate scheduler and then the one we're going to use in this specific case is reduce learning rate on Plateau and we're gonna send in the optimizer that we've already defined before in this case in the line above and we're gonna send in it takes another argument patience equals and we're gonna set it to 5 and then verbose equals true so what what this scheduler does reduced learning rate on Plateau it it looks if the loss has not decreased for a specific number of epochs in this case we have set it to 5 so if the loss doesn't decrease for 5 epochs then we're gonna lower the learning rate and this verbose equals true it's just it's gonna print and say we've changed a learning rate when when when the scheduler decreases the learning rate so then what we're gonna do is in this specific case it needs to take as input the loss for that specific epoch so what I've done here is I've I've calculated the mean loss for that epoch and we're gonna do scheduler dot step and we're gonna send in as argument the mean loss okay so pretty simple I'm gonna run the code I'm gonna pause the video until it's finished and we can see how it looks like I've trained the model for about 30 bucks and we can try to see how it works one thing I wanted to add here is that we can input another argument here which is factor which is default 0.1 so this wouldn't change anything in the code but the factor is that if it hasn't decreased for the number of epochs that we set yeah so if it hasn't decreased for that number of epochs then we're gonna decrease the learning rate by the factor times the original learning rate so it's gonna be one tenth of the learning rate yeah so let's see so in the beginning it trained for about eleven epochs and we can see here that so here the came to 258 and then it didn't decrease didn't decrease so then decrease again that's three four five and then that's when it shows let's reduce the learning rate and then continue training and yeah this is kind of hard to tell but I guess it's five times again and then yeah so that's how it works one thing to keep in mind is that it looks at the minimum so like the global minimum not the if it hasn't changed the global minimum for five a pox then it changes it updates the learning rate by by the factor times the learning rate yeah so that's it for this video if you have any question leave them in the comment below thank you so much for watching the video and I hope to see you in the next one