Gradient Checking (C2W1L13)

Key Takeaways

gradient checking this technique this helped me save tons of time and help me find bugs in my implementation of backpropagation many times let's see how you can use it to to debug or to verify that your implementation of that top is correct so your new network will have some set of parameters W 1 V 1 and so on up to WL PL so to implement gradient checking the first thing you should do is take all your parameters and reshape them into a giant vector theta so so what you should do is take W which is a matrix and reshape it into a vector you'll take all of these WS and reshape them into vectors and then concatenate all of these things so that you have a giant vector theta giant vector parameters theta so instead of a cost function J being a function of you know the W's and B's you will now have the cost function J being just a function of theta mixed with W and be ordered the same way you can also take DW 1 DB 1 and so on and reshape them into Big John vector D theta of the same dimension as theta right so same as before reshape DW 1 into the matrix EB ones already vector reshape dwl all the DWS which are matrices remember DW 1 has the same dimension as W 1 DB 1 is sin which the B 1 so the same sort of reshaping and concatenation operation you can then reshape all of these derivatives into a giant vector D theta which is same dimension as theta so the question is now is D theta the gradient or the slope of the cost function J so just how you implement gradient checking and often abbreviate gradient checking into grande check so first remember that J is now a function of the giant parameter theta right so you can also expand without the J is a function of theta 1 theta 2 theta 3 and so on right however whatever the dimension of this giant parameter vector theta so the implement drag check what you're going to do is implement a loop so that for each I so each component of data let's compute D theta aprox I to be and we're going to take a two sided difference so I take J of theta theta 1 theta 2 up to theta I and we're going to nudge theta I to add epsilon to this so just you know increase play the I by epsilon and keep everything all the same and because it taking a two-sided difference we're going to do the same on the other side will stay to open now minus epsilon and then all of the other elements of theta are left alone and then we'll take this and we divided by 2 theta and what we saw on from the previous video is that this should be approximately equal to D theta I which which is supposed to be the partial derivative of G with respect to I guess data I if D theta I is you know the derivative of the cost function J so what you're going to do is you're going to compute this for every value of I and at the end you now end up with two vectors you end up with this D theta aprox and this is going to be the same dimension as D theta and both of these are in turn the same dimension as theta and what you want to do is check of these vectors are approximately equal to each other so in detail well how do you define whether or not two vectors are really reasonably close to each other what I do is the following on I would compute the Euclidean distance between these two vectors D theta aprox minus D theta so just - OH - normal this notice there's no square on top so this is the sum of squares of elements of the differences and then you take a square root to get the Euclidean distance and then just to normalize by the lengths of these vectors are divided by D theta aprox plus D theta just take the Euclidean length or from the defectors the role for the denominators just in case any of these vectors in really small a really large you know the denominator turns this formula interracial so when you implement this in practice I use epsilon equals maybe 10 to the minus 7 so you know one year minus seven and with this range of epsilon if you find that this formula gives you a value like ten to the minus seven or smaller then that's great it means that your derivative approximation is very likely correct now this is just very small value it is maybe on the range of ten to the minus five I would take a careful look maybe this is okay but I might double check the components of this vector and make sure that none of the components are to launch and some of the components of this difference are very large there may be other bug somewhere and if this formula on the left is you know on the order of ten to the minus three then I would worry it would be much more concerned that maybe there's a bug somewhere but you should really be getting values much smaller than 10 minus vo and if there's any bigger than 10 to minus C then I would be quite concerned I was seriously worried about whether or not there's about and I would then you should then look at the individual components of theta to see you know if there's a specific value of I for which D theta cross I is very different from D theta I and use that to try to track down whether or not some of your derivative computations might be incorrect and after some amount of debugging if finally it ends up being this kind of very small value then you probably have a correct implementation so when implementing your network will often happen as results in plan for problems and back prop and then I might find that this graduate gives a relatively big value in and I was suspected it must be a bug go in debug debug debug and after debugging for a while if I find it a partial strand check with a small value then you know you can be much more confident that is them correct so you now know how a gradient checking works doesn't help me find lots of both in my implementations at neural nets and I hope will help you too in the next video I want to share you some tips or some notes on how to actually implement gradient checking let's go onto the next video