Pytorch ResNet implementation from Scratch

Key Takeaways

hello ladies and gentlemen welcome back for another PI torch video in this video we want to challenge ourselves and implement resonant architecture from scratch so this might be a bit more difficult than the previous architectures that we've done but let's go through it step by step first of all let's look at some of the ideas from the paper and the general overview the resna model 1 the image classification competition in 2015 and the general idea of the model is that when training deep neural networks the accuracy should always go down as we increase the depth of the network which in theory holds true but since it can learn more complex features or mappings but in practice it actually becomes worse after after some layers and this is the problem that the resonant architecture wants to solve here we can see kind of the an image of the resonant architecture in this case a specific one resonate 34 so to the left of it we see a 34 layer plane which is just a normal and guess calm layer this is just we're gonna take so the next the next layer is just based on the previous one computed and then if we look to the right we see that there's some weird connection in between this is what we're gonna call a skip connection so we see that for a particular layer in the one that I'm right here it takes as input from the one before right the one that it previously computed but it also takes it takes as input what it computed a couple layers ago in this case two layers ago but as we'll see the resonant 50 and 100 and 150 - it's actually gonna be three layers in between so in this case it's 2 in between but the one we're gonna implement it's gonna be three layers in between either way the way here this works is that so it can learn more complex things in these two layers but it so it can learn Newcomb more complex features but it's also going to take this what we're gonna call the skip connection the identity mapping from what it previously computed so the network can kind of choose what it wants to you have either a combination of what it has learned and also this identity mapping so that it can remember what it learned before and the argument here is that well it can learn new things but it's at least never gonna forget what it learned before so in theory it should never become worse as we increase the depth of the network and in fact that's exactly what the resonant shows it as we increase the depth it almost never I don't believe it ever ever versions the performance here are there sort of implementation details that we're gonna focus on for this video and we're particularly gonna focus on the resonant 50 resonant 101 and 152 and first of all we can see that in the beginning it does a convolution with seven by seven so the seven by seven here is the kernel size and then it's the number of channels and destroyed and then it does a max pooling and yeah so for the 7x7 kernel it doesn't say the paddling but given the output size we can infer that the padding should be three and yeah then what it does is it has these in this case it has four different what we're gonna call resonate layers so if we're looking at the first resonant layer it has a block here which is first it's a one by one kernel and then the second value is the number of channels in this case 64 and it performs this this block three times so it's going to do a three three by three kernels so in this resonant layers there are nine calm players for example in the the second resonant layer it's going to be twelve convolutional layers and the resonant 50 is just if we sum all of those the complex in this case as well with the last fully connected that's where we get the number 50 from and one thing to keep in mind here is that if we look at the output size from this one it's going to be 56 and essentially all of these are what we're gonna is same convolutions meaning that it did none of them change the the the size of the input but looking at the second resonant layer we can see that it's decreased the output size from 56 to 28 so it's half the size of the input and then we can infer that astride of one of these 3x3 kernels should be two and similarly we can see that for the resonant layer three and Resnik layer four it has also decreased the input by half and one more thing to note is that if we look at the input channels here for their first resonant layer it's 64 and then the output at the and the number of channels at the end of the first resonant layer is 256 for resonant layer 2 it's 128 and in the end it's 512 present layer 3 256 to 1024 and so it follows the pattern that the input channels if it's 60 428 2056 it's always going to be 4 times the input channels at the end of that particular resident layer so we could sort of see it that as the number of channels gets it gets an expansion by 4 times the original that's sort of the overview of the of the resonant architecture if you want to go and look at the more specifics then I recommend you read the paper I'm gonna link it in the description but now let's go back to the code so the ResNet architecture uses these blocks multiple times in a resonant layer so a good idea might be to first start with creating the block and then try to build the resonant architecture so we're going to do class block and yeah I just want to say that this code is for me at least was a bit trickier than the previous architectures and the code is kind of complicated I'll try to go through it step by step so that we understand it and the first thing we're gonna do is the block that we're gonna utilize later on so all of these variables might not be very clear yet but we're gonna talk about them in more detail as we go on so we're gonna send an inch or the initialization it's gonna take in channels and out channels and then it's also going to take some identity down sample which we're gonna set the default none and stride which we're gonna set the default one to remember the identity down sample in this case is going to be a comm layer which we might need to do depending on if we've changed the input size or if we've changed the number of channels so we need to adapt the identity so that we can add it later on when we've done that few comm layers first of all we're going to do super block elf in it yes we're gonna call super and we're gonna do self that expansion equals four so remember the number of channels after a block is always four times what it was when it entered so we're gonna first do self con one come to D within channels and out channels and then the kernel size will be one the stride is going to be one the padding is going to be zero in between then snake it next comp layer we're gonna have a bachelor with our channels and then we're gonna have essentially the same thing I'm gonna copy those two lines we're gonna do comm to batch 2 and then we're gonna do here stride is stride padding is gonna be 1 remember it's right here is what we're gonna input to the init and the kernel is going to be 3 and we're gonna have the same number of so our channels will be the number of in channels kind of confusing and then our channels will be the out Channel so it's just gonna have the same number of channels another call and then come to deal with our channels and then the output from this one is going to be our channels times self that expansion so it's going to be four times the number of channels then the kernel size is gonna be three I'm sorry size is gonna be one stride is gonna be one padding is going to be zero and then lastly another bachelor with our channels times four or septet expansion we're also going to define a rather layer really and we're gonna do self that identity downsampled and we're just gonna call it identity downsample and again this will be a calm layer that we're gonna do to the identity mapping so that it's of the same shape later on in the layers so we're going to define forward and we're gonna first say identity equals x and then we're gonna do a few computations so we're gonna go through each layer that we've defined so X is self comm 1 X is self that batch 1 1 of X yeah and and similarly for the rest I'm gonna copy in these values here essentially comm - mirela combat controller column bathroom and here at the end of those three we're gonna do so after a block that's when we're gonna add the identity so if itself that identity down sample is not none so we use the identity down sample layer if we need to change the shape in some way so if it's not none then we're gonna run it through this identity down sample and that's gonna be our identity then we're gonna do X plus equals identity and X equals self that value of x and then we're just gonna return X hey this is the block that we're gonna reuse multiple times later on in our resident implementation so now we're ready to move on to the to the ResNet from any module we're going to find in it and here we're gonna send him first block which in this so it's just going to be this class I'm actually not sure we might not even have to send this in but let's just do it in this case and then we're guys sitting in layers and image channels and the number of classes and the first thing we're going to do is call super ResNet and in it yeah so the block here is as we said it's going to be this class layers in this case is going to be a list which is gonna tell us how many times we want to reuse or rather we want to use this block right here so for example I think in our case if we look at the resident 50 I think it was 3 4 6 actually my checking yeah 3 4 6 3 so essentially we're gonna use in the first resonant layer we're gonna use 3 the block three times and then the second resonant layer the block 4 times etc the image channels is just the ink the number of channels of the input so I mean either we have normally RGB so we have three input channels or if we have like em notice we would have one image Channel so we just you just have this to be able to make a resonate implementation general in the way that we can input any type of images yeah and the number of classes depending on what we have in the data so first we're just going to do in channels is 64 and we're gonna do self.com one is and then come 2d image channels 64 output that's always gonna be the case then we're gonna do kernel size 7 stride of 2 padding of 3 so this is just the initial layers we haven't done any resonant layers yet and then you just a batch norm and then a value and remember also we're not in we're only initializing the modules we haven't actually called anything yet so here we're just in it we just defined the layers and then we're going to use them later on then we want the max bool max bow to thee strike is going to be to having is going to be one and then here we're gonna do their resonate retinas layers so we're gonna do self that layer one equals self layer two but to be able to make this simple we're gonna call a another function that we're gonna do to define make layer and what this so this is gonna create the layer so we're gonna send in the first the number of times it's gonna use the block right and then we want to send in like how many out channels it should be after that one so let's do the just input for this first we're gonna send in block again and the block meaning the class we define first then we're going to send in a number of residual blocks that's just what we're gonna call it that's the number of times it's gonna use the blocks and then we're gonna do out channels which is the so the number of channels it's gonna be when when we're done with that layer and then we're gonna send in a stride because as we saw previously so there's gonna be one of those one of those in this case four times that we call the block one of those is gonna have a stride of two and similarly for this one and this one except for the first one it's gonna have a stride of one so what we're gonna do is we're gonna send in the stride that's that's either going to be one or two first we're going to do identity downsample is none and then we're gonna define layers to just be empty and we're gonna add the layers as we go on and then the first thing is that well we want to know when are we actually going to to do a identity downsample in this case like when are we going to have the calm layer to to change the identity well either either we change the input size so we have the input size right if the stride is 2 then we're gonna for example if we have 56 or like make it easy 28 by 28 it's gonna be 14 by 14 then we need to change that identity so if the stride is not one or if self-thought in channels is not equal to our channels times four yeah so this is gonna make more sense later on but if essentially if we've changed a number of channels so if we if we have multiplied number of channels by four but we but the identity still has one so a quarter of what it has in the future then we're gonna have to change the number of channels so that it can be added like we did here right so then if that's the case then we're gonna do identity downsample is n dot sequential and then come to D and it's gonna be out channels times four and since we only want to change the the colonel in the channels we're we're not gonna do anything different with the colonel we're just gonna do a colonel size of one and then this stride will just be stride then so let's see then we're going to do just a bachelor and that's it for the identity downsample and so let's see now what we want to do also is then we're gonna do layers that append and what we're gonna do in in in the first one is that we're gonna we're going to change the number of channels so this is a tricky part we're gonna the in channels is yourself that each channels and then we're gonna do our channels which is gonna be so we send in the outer channels right here so for example I think in the first case it's going to be 64 input channels and the output will be 256 and then we're gonna do identity down sample that we just created with that specific stride essentially it this is the layer that changes the number of channels so after this block which is the first block that we append to layers is gonna change the number of channels in this case if we look at the first resonant layer it's gonna change it to 256 then what we're gonna do is we're going to update self 30 channels to be out channels times 4 yeah sorry so I think I did a mistake so the out channels in this case is gonna be 64 but remember as we define the block it's gonna be the output from this is gonna be our channels times 4 so it's gonna be 256 just that our channels in this case is not gonna be 60 256 it's gonna be it's gonna be 64 but it's gonna be multiplied by 4 in the block so it's going to be 256 at the end of this block right here so that means that we need to change number of enchants the in channels is not gonna is now going to be 256 so it's going to be in this case 64 times 4 then we're gonna do for I in range of the number of residual blocks that we're gonna compute in this case minus 1 because if we computed already one residual block right here that changed the number of channels and then we're just gonna do layers that append block self-thought in channels and out channels and then in the end we're going to return an adult sequential of layers so this this will unpack the list and it will do so that my torch knows that each of them will come after each other so I think I want to add a few comments because literally these are the most difficult parts of the the implementation I believe so first of all here the number of in channels in this case it's gonna be 256 and the out channels is gonna be 64 so in the end of the first block that we did here we the output is gonna be 256 but the out channels in this case is 64 right because we need to map 256 back to 64 and then in the end of the of that block it's gonna be output is going to be 64 times 4 so it's gonna be 256 again so essentially it's gonna be input 256 and then it's gonna it's gonna map it 256 again going through intermediate complex where it goes from 256 to 64 and then back to 256 and destroyed in all of those case is gonna be one and number of channels is the same right 256 256 is the same number of channels so we're not going to use the identity down sample in this case the only case where we change the number of I guess yeah the number of channels and the destroy the input is in this first block then the other ones are going to be shred one as we define as default so this means that we're just gonna do this one without doing the identity down sample that's only for the first block yeah and then and then we can just do this loop and in the absolute end of the loop it's still going to be 256 output yeah all right so then what we want to do is now we're ready to define the resonant layers and then we're gonna do self dot make layer I'm gonna send them block meaning the class that we defined we're gonna st. sending layers 0 so in this case yeah this layer 0 is gonna be this list here it's just gonna be the three actually remove this and then our channels from this going to be 64 and remember it's going to be the out channels so the name is a little bit confusing it's it's our channels times four at the end yeah and then we're gonna do a stride of one for the first resonant layer then let's copy this and use the layer two and changes to one and then in the out channel is going to be 128 and then in the end it's gonna be 128 times four and then for the rest layers we're going to use a stride of two so let's again copy this layer three we're gonna do block and layers two and here's going to be 256 again stride of two and then for the fort resonant layer layers three and the out channels is gonna be 512 so 512 times four it's gonna be 2048 channels at the end and again it's tried of two then we're gonna do self dot average pool and we're gonna use n n dot adaptive average pool which essentially is that we define the output size and it kind of does an average pool that depending on the input that's gonna fix it to that particular size in this case we want the one by one at the end and then we're gonna do a fully connected layer at the end which is gonna be 512 times four right so the number of channels here times the expansion which is four and then we're just gonna map it to the number of classes all right and then what we're gonna do is one more thing we're just going to do forward on all of those so X is self.com one of X set that bathroom one of X and then relu and max pool and then we're ready to send in through the the redness layers so layer one and I think just for a moment like appreciate how much goes into that particular layer we're calling the make layer which calls the block multiple times and the block by itself does a lot of calm like three calm layers so there's a lot to unpack in just one line that we're gonna do in the forward and then layer three of X or and then we're gonna send it through the average pooling to make sure it's of the correct shape or yeah to make it output one by one and then we're gonna reshape it so that it's we can send it into the fully connected layer and then silly self a fully connected of X and then we're just going to return X okay now what we want to do is I want to define resonate 50 we're just gonna take his input number of in channels and yeah so we're gonna do in in channels and let's say number of classes which we're gonna set to default to 1000 and also in channels 3 and then we're going to return ResNet block in the number as of the layers three four six three in this case and then we're gonna do in channel and the number of classes and really so all that we need for the resonate is this one right that that's what defines the resonant fifty we can do the same thing for for the rest for resident 101 except all we're going to do is we're going to change this in this case this the six becomes twenty-three and again so really now we notice that there's not that much difference between the different resonates except that they're larger so here we can actually change this to eight and we're gonna change this to 36 and then to test that we've actually implemented it in the right way we're gonna define test we're gonna do X is torched at random and we're gonna do let's say we have three in let's see if we have two images just to make it fast no no this would be 16 or 32 or 64 something like that but yeah this network is quite come like expensive to run so let's just do two let's do 3 - 24 - 24 so the number of in channels in this case is 3 and then we're going to do y equals net of X and never let's do two CUDA so we're gonna see we need to do net equals ResNet 50 resonance 50 so then so we've defined a network which in this is written at 50 we're gonna we have some random input just to check that the shapes are correct then we call the net and then we're gonna do print y dot shape and final let's call test though this is the moment of truth let's see if all the implementation we've done actually works up the interactive Python just one this okay so in channels yeah this should be in channels let's around again yeah so this is our channel it should be out channels all right and perfect it works so let's actually try all of them let's just see that they do the right thing and Brezhnev 152 all right that's great this code is like I sat for a couple of hours just to really understand the code and it's kind of difficult to understand all the steps hopefully this was useful and if you have any questions and leave them in a comment and thank you so much for watching the video