Neural Network from scratch - Part 4 (With Python)

Key Takeaways

finally finally we get to the point where we can implement the neural network in numpy so we've gone through all the theory that is needed for this part centrally there will be some small parts that we haven't but we take those as we go through it so first of all we need to import numpy as NP and also I need to tell you that we were using a data set called M nest which is a handwritten image data set so it looks something like this we have an image which is 32 by 32 pixels so 784 pictures in total and for each of them we have a label the correct label and also the image of course and the data set can be found if you go to this website or if you're just google M this data said it will be the first one I believe and download all of these four and put them in a folder just I've done it already so in a folder called data or something and what you do is that you have all these files and you unzip them for example I unzip one of them so this one and the only thing you need to do here is to change this dot to a it's one of these and yes that's all you need to do do that for all four and you should be fine so to be we are only focusing focusing on implementing the action or network so I'm I'm just going to use this too to be able to read the data so from M&S import m nest and if you have it you can pip install python /m nest so all we need to do that now is load data so m and data is M nest of data and here you need to write a folder where you put all of those the data set that we downloaded from this website so what we do then is that we split the data into we use mmm and data that load training to split it into X train Y train so the extreme will be all of the images and the boy train will be all of the labels corresponding the correct label for each of the images similarly for lo testing will load the test set and don't thing we need to do here is that we need to convert all of them to numpy arrays so n p del rey and x train etc now we also what I did here is that I the data set was a bit slow to run it works but it was a bit slow to train since yeah we're just we're loading 60,000 images simultaneously and just training on all of them at the same time that costs a lot when doing it on on the CPU so I just lowered it to make training a bit faster so we have 5000 training examples and 500 test examples and yeah so if we print the size we have extreme has shape 5,000 images each image has 784 pixels and also that the test set has 500 examples 784 pixels for each and we have a label for each of the images okay that's that's the data let's go to the actual implementation so we make a class for our neural network we defined in it which takes self x and y so we send in images and also the correct labels for all of the images and we define self dot m to be the amount of training examples that we have self dot and for the number of features in our case this will be 784 and remember the hidden layer we choose how many nodes do we want in our hidden layer this you can choose whatever here you can choose 1025 I just chose 25 in this case and the number of nodes in the output and we need to have 10 because we have 10 different digits that we can classify and also selves our learning rate 0.001 I just picked that it works you can pick I think a little bit larger a bit smaller but in general one needs to be quite careful with learning rate because if you take a to large learning rate the cost might diverge which causes problems so good learning rate is important I tried some and this one seemed to work and also what we have to do is we have to initialize to wait and one idea is that we could initialize all the weights at start to zero but that would the a better way is to initialize them randomly which is small small variation so for example we do Empire and Empire at random R and n to be normal normal random distribution with mean 0 and ya then we just then we just with with variance 1 and then we multiply by 0.01 to make the the weights very small and the biases are ok to initialize as zeros ok so then we come to the for prop propagation of our network so and we send in the input X the parameters so W 2 W 1 B 1 B 2 and B 1 which is stored in a dictionary so a 0 will be our input then we compute Z 1 to be a matrix multiplication a 0 comma W 1 and we add the biases we've seen this before so no surprise a 1 we take MP dot max of 0 comma Z 1 remember this is applied element wise and lastly we do Z 2 to be the MP dot a 1 comma W 2 and devices now we're at the last layer so we do the softmax and the scores are Z 2 then we do the exponential scores we take the exponent of all of the scores and then we take the probabilities to be the exponential scores divided by the sum of all of the nodes scores this is to make it normalized so that they are actually probabilities where each of the output in the last layer the output layer each node will have a value of 0 2 2 1 then we cache all of them to for the next which is we use later on so we just store a 0 probabilities in a 1 and the last part is that we compute the cost so we take we need our parameters W 2 W 1 actually I don't think that's new this is needed no I don't think this is needed yeah but so we can just delete this I guess what we do is that we take the loss which will be the minus NP log of the probabilities I remember from the back row the back propagation we want to take no sorry the forward propagation we want to take the minus MP log of the probabilities of only the computer probability of the correct label so we also send in our our labels for the correct ones so for all all examples we want to just take the minus MP log of the ones that are actually correct then we compute an average loss by taking the sum of because remember this is the loss for all training examples so we sent in 5000 so we get a loss of 5000 here all of them so we sum them all together and we divide by the total amount of examples that we have to get an average loss then we return to average loss for the backward propagation we need all of our parameters again w2 our weights and biases we also unpack the output of so our input our output from layer 1 and our probabilities now we start at the end remember so we had probabilities which is at the end of the for propagation and what we need to do is that we take equal to the probabilities from softmax as we have derived earlier on and we take - we take that equal to - and we just take that equal to itself minus 1 but only minus 1 for the cases where we have that the node is in fact the the node which corresponds to the correct label so that's what we do here for every example look at the node which is actually there the one that we want the one that's correct and then we take that equal to minus one then we divide here by by because if we took a loss that was the average so we divide by all of our training examples now for the weights to w2 we take a MP dot of a 1 comma D Z the Z - this is one thing that we had to figure out while we're actually doing it now because we found the derivative of Z 2 / with respect to w2 is a 1 and what we have to do when calculating the total gradient is that we have to take it multiplied by the gradients from above so the first gradient that we calculated which is DZ - the only thing that's tricky here is that we need to take this transpose and the only reason we do it is because for these dimensions - to match so really just print a 1 dot shape print DZ dot shape look how can we multiply here how can we multiply them together to actually get the shape that we in this case we want w to to be a shape of 25 comma 10 right because the previous layer had 25 the output layer is 10 so just just look at these two because we need we know we need to multiplicate multiply them together from from the derivation but we just don't know exactly if it's the transpose or not and we we look at by looking at the shapes and seeing how do they match together so that's how we get to the transpose here and for the biases remember that derivative was just 1 except that we had gradients incoming from several examples from all 5,000 examples and when you have a gradient coming to a node from several different positions so for each example then the correct way to calculate the gradient is to sum them all together and that's what we do here and we sum 0 because the examples are at the rows never just keep dimensions so that it doesn't become anything other than what we expect really and then the last one is that we take DZ 1 to be the MP dot of the previous comma W 2 transpose I believe this is what we expect to from that it's this transpose from matrix calculus and also here this is the derivative with respect to the reloj and this is a element-wise multiplication as we discussed and the reason why we so this is essentially if the value was greater than 0 then it's 1 so this is just a boolean value if it's not then it's set to zero because the Darrell you didn't let the value the Z value the set value passed when we computed it and then similarly with your wdw one exact same as this one except we have one instead of one layer previous and the same for devices and then we just store all of those gradients into dictionary and return the dictionary and update parameters this is from gradient descent so we unpack first we call the learning rate which we decided in the beginning we store all of our we unpack rather all of our our parameters and then we do gradient descent so W 2 is equal to W 2 minus the learning rate times the derivative that we computed in backprop for W 2 so V for derivative for W 2 and we do that for all and then we we store the parameters again in a dictionary and return the dictionary so this is the the parameters here or after there they have been updated once from with gradient descent ok so now for actual main function we first need to initialize all of our parameters that we want B 1 W 2 B 2 we pack all the parameters into dictionary in parameters and so we want to ask how many gradient descent update do we want - how many forward passes followed by a backward pass or backward propagate do we want our neural network to do so we decide that when we send it into this main function the number of iterations and so what we first do it we do a forward move for propagation we calculate loss and sometimes we print loss just to get an understanding how are we doing is it doing is it the cost decreasing which is what we want or is it increasing so that we can easily debug and after following the forward pass as well as calculating the cost we do a back propagation and lastly we update parameters and we redo that number of iterations number of times so that hopefully our network has then learned from this data and can generalize this to unseen data so what we do I've already run this for 1,000 iterations well what we do is that we first call the class neural network with our data extreme white rain so this will be the 5,000 examples and then we call the main function so we call with with with our data extreme white rain and 1000 iterations and this will be so what when you get return here other parameters the weights and the biases for this network after it's been trained for 1000 iterations we can see the loss starts quite high and then decreases to a very low number which is so this means that okay it has actually learned something from from from our neural network or the loss has decreased so and then we have W 2 so we want to this train parameters will be a dictionary containing all of the parameters we unpack them from the dictionary and what we want to do now is we want to check well how does it do on the training the test set we've trained on 5000 examples how accurate is it on the 500 examples on the test set that we haven't seen before so what we do is that we we run just a forward propagation so we don't run the entire neural network we just run the for propagation part and remember the for propagation calculates the the probabilities at the end so what we send in here is the X test and we also send in the Train parameters that we we've computed or that that's the result of having trained previously so we get return here are the probabilities and so we look essentially what for each of the probabilities which one R is the largest because that's the one we'll pick if one node says yeah 99% sure it's a is a two then we would say okay it's probably a two so we look at this MP dot org sort it's sort the matrix and and it sources from lowest to highest value so what we have to do for each example we pick the one to the furthest to the right and also MPL arcs or returns the index for which for the position so it it gives us the node corresponding to the actual like the digit that we believe this this image is and after doing that we we compute an accuracy so we look how does our predictions that we computed equal to the actual labels and we take them equal so this is a boolean value 0 or 1 and then we sum them together and we divide by by the amount of divided yeah so we divide by the amount of images that we checked so let's say that we have 500 then we divide by 500 because if we take this sum so let's say 400 of them are correct divided by 500 so we know it's about a percent correct in this case we get an accuracy of 91% which is it's okay it's pretty decent because we have let's see we have a very small hidden layer just 25 nodes we trained on only 5,000 examples and we check on 500 so yeah you can get this a lot higher to I think by training it longer and lowering the learning rate etc and so I also have a few other ideas on how we can improve so we can train longer and on the entire training data we can use a better optimizer now we actually don't use an optimizer at all so we can use Adam for example we can initialize the weight better instead of just doing the random normally distributed weight with you there are other ways to do it and there's something called an initialization that's actually very good and we can also train using mini batteries instead which would make the training a lot faster one more thing we could do is make a larger network with more hidden layers and more nodes this would also a large network it's a better network in general so it's one thing we could do and we can also introduce l2 regularization or dropout to make sure that our our neural network doesn't over fit to the training data so that's the implementation of a neural network from scratch if you have any questions or anything post them below and thank you so much for watching