Neural Network from scratch - Part 1 (Standard Notation)

Key Takeaways

alright hello everybody so I remember being quite confused on how to implement in your network from scratch so perhaps these couple of videos will be of some help the the idea is that we in this video go through some intuition behind the calculations for your network and important notation that we'll use in later videos where we'll go through forward propagation as well as backward propagation and lastly we'll do an implementation of a neural network from scratch in python using numpy so these videos assume that you know calculus linear algebra and also that you know the basics of in your network so for example you know what forward propagation is and backward propagation is at least an an idea of what it is you don't have to know the calculations for it but just the idea of what it is and as well as then you know that what softmax is so these videos will be more practical on how we actually implement it and so I assume that you know the basics if you've never heard of neural networks before but still want to learn I will post some good videos by three blue one brown that will that will help you build an intuition and understanding of what neural networks are and how they work so let's see started so first of all we have I've written a quite simple under network here so we have an input some type of data that we send to our network we have some type of hidden layer that does computation and we have some type of output from our neural network and the what we usually call is that the first one is called layer zero similarly the second layer is called layer one and the last one will be layer three oh sorry there too okay and well what's important here is and while I try to be quite specific and try to explain as best that I can is the notation that we'll be using so the the output from a particular layer I will denote as a okay that's the output from a layer and the thing here is that each layer has several nodes right so from this hidden layer for example we have several outputs we have five nodes so to try to be specific in which node I'm referring to I will write a J to refer to the J note where this one is the first node second node third fourth and fifth node and to say which layer I'm referring to we write in brackets L now let's examine take a take an example that we have handwritten digits so we have ten images of handwritten digits from 0 to 9 and let's say that the we have sent it into a network and the calculations for each of the images are different right because the pixels the input that we send in are different pixels so one can imagine that it does different computation depending on the input so what we have to do also is the way you need to refer to the eye training example or the the specific image that we're standing in now I will I won't be referring to a specific training example I will just assume that we're talking about a specific training example and just write the elf layer and the JT node so let's take an example of a a let's say 1 and node 2 so we know that we're in layer one and we also know that we're in node two so what we have is that we know that we're talking about this node and let's say we're talking about layer first layer I mean layer zero and let's say we're talking about node one well we first go to layer zero we look which is the first node well this one now one thing we have in between of the layers are weight and we have them from layer zero to layer one as well as to layer 1 to layer two and we need to have some type of a notation to refer to these weights as well so I will write W of a specific layer now we need to be careful because each node here each node has weights associated to every other node so we need to specify from which node J 2 to node K so specifically which edge are we referring to so some examples might make this more clear let's say that we have W of 1 and so what we refer to when we say W of layer 1 we mean the weight that connect layer 0 to layer 1 so we refer to these weights here and also if you write let's say J equals to 1 and k equals to 2 then what we're referring to specifically is the know the weight or the edge from node 1 to node 2 so this one okay so now that we have some idea of what notation we'll be using let's do a quick example so let's say that let's say that we take this portion so this part of this neural network and I'm just going to redraw it so we have we have one node another node and there note and all of them are connected see all of them are connected to a node so as we this is the purple that I draw earlier on the other image that we had so we have wait so let's say that so these nodes this is the node from layer zero and the first node this is the node from layer zero and the second node as well as this one to be layer 0 and the third node now this weight here will be weight connecting layer 0 to layer 1 so we will say that it's layer 1 node and it's the one connecting one two let's say that this is the first node so 1 comma 1 similarly this would be 1 and 2 comma 1 W of 1 3 comma 1 so well first of all I need to draw this little bit bigger so see a lot bigger so let's see it's still one node so it won't make sense just bear with me so all of them are connected to one node and this node we can view as being in split into two parts of its comput so we can view each node as doing two parts of computation now the first competition will be that it will take each weight and each weight will just have a number each weight is just associated to to some some number so let's say it's 0.1 or it can be any number so we take this weight and we multiply it by the value of the node that's associated with that specific weight now we take this weight and we do the same thing with that node we multiply them and lastly the third weight with the third node now what we do is that we call the result Z which will be the sum of these multiplications so we call it the sum J equals 1 2 3 because we have three nodes incoming and three weights and coming to this specific node so we take W of layer 1 and it will be 1 and now let's see it will be J comma 1 times a of 0 comma J now there's one more thing in this part of the computation and we will have right here it will be a bias term so it would be B of layer 1 and for the specific node which we will call 1 so the reason why I write the node here I write the arrow here to denote the bias in coming to this specific node is because the bias term can be viewed as local to the specific node so it's always associated to this specific node and the only thing that we do to the bias term is after doing this multiplication in summation we added at the end so we take this bias term remember the bias term is just a constant now one thing one can see here is that well this is quite close to the form y equals MX plus B right this is this is quite linear this is a linear function and the problem with this is that we want in our network to be able to to classify quite complex patterns quite complex data and a linear classifier won't work so what we have to do is some type of non-linearity to our linear network and what it's called normally is an activation function and there are many different types of activation functions for example sigmoid tan H and relu and perhaps relative being the most common that's the one that we'll use so what we do in the second part of the computation for this specific node is that we so just to be very specific here this is Z of layer 1 for node 1 ok just to be consistent with their notation so the last computation of this node will be a of layer 1 for this specific one which will be a real loop so the activation function value of Z and I'll skip writing the layer and the node but exactly the same as this one so what we'll do is that they're the rel affection is a quite simple function actually it's the maximum of 0 comma Z so what is what does what does it do well it takes if Z is greater than 0 then it will just picks it if it's let if said is negative anything less less than or equal to 0 will be set to 0 so literally it's just let everything passed through if it's not negative if it's positive it lets pass if it's negative then it sets it to zero so now that we know the notation and some basic intuition behind the calculations we can get started with the more complex in the next video for forward propagation as well as backward propagation