Basic Recipe for Machine Learning (C2W1L03)

Key Takeaways

in a previous video you saw how looking at training error and Deborah can help you diagnose whether your algorithm has a bias or variance problem or maybe both it turns out that this information that lets you much more systematically using what I call a basic recipe for machine learning that lets you much more systematically go about improving your algorithms performance let's take a look when training your network which is a basic recipe I will use after having trained an initial model I will first ask does your algorithm have high bias and so the child evaluate if this high bias so you should look at really the training set of the training data performance right and so if it does have high bias it's not even fitting the training set to that well some things you could try would be to try to canet work such as what layer's or more hidden units or you could trim it longer you know maybe run gains and longer or try some more advanced optimization algorithms which we'll talk about later in this course or you can also try this kind of a maybe a work maybe it won't but we'll see later that there are a lot of different neural network architectures and maybe even find in your network architecture that's better suited for this problem putting this in parentheses because all those things that you know you just have to try maybe to make work maybe not whereas getting a bigger Network almost always selves and training longer well doesn't always help it associate never hurts but so when training learning other I would try these things until I can at least get rid of the bias problems as if go back after I've tried this until and she's doing that until I can fit at least fit the training set pretty well and usually if you have a big enough network you can you should usually be able to fit between training day so well so long this is a problem that is possible for someone to generate an image is very observing it may be impossible to fit it but if at least a human can do on the tall so do you think Bayes error is not too hard and by training a bigger than network you should be able to hopefully do well at least on the training set so these fit to over 30 trainees once you've reduced buyers to a acceptable amount I would then ask do you have a variance problem and so to evaluate that I would look at the guest set performance are you able to generalize from a pretty good training set performance to having a pretty good job set performance and if you have high variance well best way to solve the high variance problem is get more data if you can get it this node can only help but sometimes you can't get more data or you could try regularization which we'll talk about in the next video to try to reduce over 15 and then also again this is a sometimes adjusted to try it but if you can find a more appropriate neural network architecture sometimes that can reduce your variance problem as well as well as reduce earlier your bias problem but how to do that is harder to be totally systematic how you do that but start writing a savings in collective going back and so hopefully you find something with both global buyers and low variance whereupon you would be done so a couple points to notice first is that depending on whether you have high bias or high variance the set of things you should try could be quite different so I'll usually use the training depth set to try to diagnose if you have a bias or variance problem and then use that to select the appropriate substantive things to try so for example if you actually have a high bias problem getting more training data is actually not going to help where this is not the most efficient thing to do so being clear on how much of a bias problem or variance problem or both can help you focus on selecting the most useful things to try second in the earlier error of machine learning there used to be a lot of discussion on what's called the bias variance trade-off and the reason for that was that for a lot of the things you could try you could increase buyers and reduce variance or reduce bodies and increase variance but back in the pre deep learning era we didn't have many tools we didn't have as many tools that just reduced bias or that just reduce variance hurting the other one but in the modern deep learning baked into error so long as you can keep trading a bigger network and so long as you keep getting more data which isn't always the case of either of these but if that's the case then getting a bigger network almost always just reduces your bias without nest recruiting iberians so long as a you regularize appropriately and getting more data pretty much always reduces your variance and doesn't hurt your bias much so what's really happened is that with these two steps the ability to train bigger network or get more data we now have tools to drive down bias and just drive down bias or drive down variance and just drive down there and without really hurting the other thing that much and I think this has been one of the big reasons that deep learning has been so useful for supervised learning that there's much less of this trade-off where you have to carefully balance bias and variance but sometimes you just have more options for reducing bias and reduce or reducing variance without necessarily increasing the other one and in fact so as you have a well regularize network will talk about regularization starting learning video training a bigger network almost never hurts and the main calls to training in your network that's to base is just computational time so long as your regularizing so I hope this gives you a sense of the basic structure of how to organize your machine learning problems to diagnose bias and variance and I try to select the right operation for you to make progress on your problem one of the things I mentioned several times in the video is regularization is a very useful technique for reducing Berens there is a little bit of advisor and trade-off when use regularization it might increase the bias a little bit although often not too much if you have a huge enough network but let's dive into more details in the next video so going to understand how to apply regularization to on your network