Pytorch Text Generator with character level LSTM

Key Takeaways

ladies and gentlemen welcome back for another PI torch video now normally we just dive right into the code and then start coding but for this one we need some context so my brother is gonna have a baby and the case is this he is absolutely terrible at names and so am I so I can't really help him but maybe we can create a neural network and more specifically a character level RNN that can generate names so without further ado let's now get started with the code so let's start with all of our imports and I'm just gonna copy them in I guess the only one that you really haven't seen before I guess is is there this one you need decode and we're gonna use this just to make sure that every character that we so we import a large text file which are names we're gonna make sure that everything is in ASCII format and yeah I'm let's see I'm gonna copying some things that I guess aren't that important so just device we're gonna run CUDA if it's available and then we're gonna do so we're gonna get characters from string dot printable and so let's see if we run this and we do all characters yeah we sort of see that these are the characters we're gonna use and so yeah we have all the numbers I'm going to have a to Z and then capital letters A to Z then we have some other other characters there and so let me show you let's see let me show you the data set so all I did here is I just downloaded a very large file of names so I guess it's like million names or something like that and I've just put them all in a single text file and we're gonna load this text file so I'm just gonna copy that in as well let's code the important parts like the RNN and the in the actual code but this is not really that important so we open that text file we read it and then we call uni decode that unit decode on on that to make sure that everything is in ASCII format and yeah that's it so let's now do our RNN so we're gonna do class R and N and then module and then we're gonna just let me do a pass on this right now and we're gonna have another class here with color generator and so the RNN is gonna be our recurrent neural network and the generator is gonna do pretty much it's gonna that's the one that's gonna train Durnin and also it's gonna it's gonna have some function to sort of generate some text from randomly so yeah let's do the Arnon first and then let's do two generator it's gonna use the Arnon so we're going to define in it and if you watch my last video this is going to be very familiar to you we can do a hidden sized number of layers output size we're gonna do super RNN self and then dot in it self dot hidden size is hidden size and then sub type num layers is non layers and the first thing we're gonna do is we're gonna create an embedding for each word I guess character that we send in and so yeah we're going to set that embed and then dot embedding input size and hidden size and this is this is going to be learned by the RN n sort of the embedding for each character so this is gonna be a long vector and I guess since we have this what we have right here if we do n characters we have 100 characters and so the input size is going to be 100 and then we're gonna have some word sort of the vector embedding of each character which is going to be some hidden size that we're gonna create later on but that's gonna be so that the embedding vector is gonna be learned by the Arnon now we're gonna do sublet LCM and lsdm we're going to hit in size two hidden size and then number of layers batch first equals true and yeah that's all and then self dot fully connected it's going to be a nonlinear to from hidden size sort of the output from the LCM and then to output size like this then we're gonna have define forward let me just put this in center we're gonna add forward we're gonna have self x hidden cell and first we're gonna just do out is self that embed of X and out and then we're gonna have hidden comma cell itself that LS TM and yeah so we're gonna do out dot on squeeze one this is what we're gonna just make the shapes match we're gonna add another dimension for the first axis and we're then also going to send in the hidden and the cell and then out will be self dot fully connected of out a tree shape out shape 0 and then minus one sort of unrolling everything except the first first one which is the number of examples that we send in then we're going to return out and hidden comma sell and define define in it hidden we're gonna do batch size sending the batch size and then we're gonna do hidden is torched zeros and then num layers flash size self that hidden size and then duct to device and we're gonna do the same here for this cell like this and then we're gonna return hidden comma cell okay this is our RNN so quite basic right we haven't really defined anything about the the RNN right now since we haven't said what the hidden size should be the number of layers etc just a general class so we're gonna do everything in a generator class and so we're going to define in it and what we're gonna define first of all is self-taught Cheung length alright let's just do a chunk Lin to be let's say 250 so what this is is how many characters it's gonna take from the text file that we showed you previously how many characters it's gonna take at a time okay so it's gonna take 250 random at a time so yeah I guess let me show you again this one let's say I don't know let's say this is 200 cats gonna take down one sometimes it might take like like this like half of just the beginning of a name right here but it's not really gonna make a difference in the new year so in the general way that RNA is going to learn so we're just going to set it to 250 and then let's do self dot naam epochs - like 5000 or something and let's do self dot Bosch size and one thing that I don't really understand but there must be some kind of explanation for it is when when trying different batch sizes I it doesn't really become better if I have a batch size greater than 1 and in some cases actually becomes worse so just keep the back but we're just going to keep the batch size to 1 and then we're gonna do print every 50 bucks let's define the hidden size to be 256 and stepped-up num layers to be to set that learning rate let's say 0.03 I found to be a good one so then let's just say we're gonna do let's rewrite the skeleton code first we're gonna char tensor self comma string so this function is gonna do is it's gonna take a character and it's gonna map it to a a vector of size 100 then we're gonna define get random batch so it's gonna do pretty self-explanatory its get just gonna get a random batch of 250 characters for us and then we're gonna do define generate we're gonna write this later on but this is gonna just generate some names for us we're gonna have a train yeah so this is gonna train the ironin and yeah so those those are it let's do the chart answer first so we're just gonna define tensor to be torched zeroes of the length of the string and then we're gonna do it to to format long then we're just gonna go through for character in the length of the string we're gonna do tensor of of that specific I guess for each character we're gonna do all characters dot index string C so this is just gonna return the the index for that character in this in this I guess in this string right here so it's just gonna return the index of words where it's at and so I guess we could for example do our characters dot index of a you shouldn't like that and we get back the index for it so that would be the tenth index and yeah so we do that for the entire string remember the string is gonna be 215 length that we decide here and then for our get random batch we're gonna do start index it's gonna be random dot Rand int of 0 comma length of the file - self dot chunk Len so this is kind of make sure is that it's always gonna be 215 linked and it can't sort of pick from from from the last I guess characters and then we're gonna get an end index which is gonna be start index start and yeah let's do and index it's gonna be the start index plus the self dot chunk the length plus one just because Python is zero index yeah we have to yeah so it we're just gonna add a 1 here and and so remember that we're gonna send in the first character of the string that we send in but the mapping so the corresponding correct character is going to be the prediction of the character right so sort of the first character in the text input should map to I guess the second in detecting in our string that we send in since the RNN is gonna do a prediction for each character we we obtain a character and we're gonna output our guess for the next character that's gonna be in the sequence so when we train these curricula vernon's we're gonna have an offset by one from the input to the target and then India we're just gonna return text input and we're going to type torch that long I guess dot long and text target that long I think that should work and then we're gonna do let's see we're gonna do the do the training first so we're gonna do self dot RNN it's gonna be iron in of n characters that's gonna be the let's see what it is that's going to be the input size right here let's just copy this make sure we know what it is so we're going to have the number of characters characters No and characters and characters and we're gonna have self dot hidden size self that num layers and itself and then end characters again and we're just gonna do dot two device then we're gonna see we're gonna do optimizer to be torched Upton Adam self that RN n dot parameters and learn rate is she's going to be self dot learning rate the criterion is going to be cross entropy loss and we're gonna do writer is summary writer runs names zero let's call it so this is just for for for tensor board we're gonna print the loss and then yeah let's see we can do yeah we can do print starting training and there can be a for epoch in range one cow myself that Nami bugs plus one and so we're gonna do is we're gonna run this one and get random batch we don't have to input anything and this is gonna be our input and target for this for this epoch remember those are going to be 250 characters in a vector or I guess in a tensor and then we're gonna do hidden comma cell it's gonna be self that are n n dot in it hidden and a batch eyes is just going to be self of batch size and let's see here next thing we're gonna do we're gonna do cept at iron 10.0 grad and then loss is equal to zero and are we gonna do is in is m dot to device target is target to device just to make sure it's on cuda and then we're gonna so go through each character we're gonna send in a character by character to earn n so for c in range of self dot chunk len we're going to do output hidden and self it's just gonna be safe that are none of of the input for all batches at the same time she's gonna be one and for a specific character c then we're gonna send in hidden and cell right the one that we had here and then we're gonna use loss plus equals criterion of output become a target of all comma C and I guess yeah now it makes sense as well the why we need to offset this by one because this is gonna predict the next character and so it should match their first of the target vector so then so we're adding all of those losses which is gonna be 250 losses and this might be an explanation why using mini batches won't work since it's just gonna be it's gonna be yeah I guess adding 250 is enough I'm not really sure about that but then we're just gonna do lost our backward I'm going to optimize your diet step and our loss let's say lost that item and let's just divided by the chunk length so get sort of an average of the loss for each character and then I'm good actually we think using scheduler so we write that and then we're gonna do if I park self dot print every is zero so I guess if Deepak is modulus 50 then we're gonna we're gonna print loss and it's gonna be loss and what we're gonna call here as well is self generate we're gonna do so we're not gonna write we haven't done this yet but we'll come back to it and then we're going to write add scalar training loss loss global step this equals epochs this is just gonna to that sensor board and so we're almost done when you just need to write to the generate here and we're gonna have a initial string so let's just do I or something this is sort of what we're gonna have as our initial string for the generation so I guess we could have something like a here which means that every the first name that it's gonna generate is always going to start at a and then we can send we're gonna send in the prediction link and you can set this to anything let's just do 100 so it's gonna generate I guess I don't know 10 names or something like that and another thing we're gonna send in is temperature and the temperature is guess it a little bit complicated but we don't really need to go into it in this video what temperature is it's really sort of how risky is should the iron and be if it's if it's a low value it's gonna sort of be very conservative and let's say you have a not I text a text file of names but let's say you would have sort of I guess written just written sentences you would have a lot of and sort of a lot of or etc and and if you had a very low value here it would just predict and and and and something like that if you have a higher value here and it's gonna be be between zero and one if you have a higher value it's gonna be more risky and make more sort of random looking things so more cool things I guess more fun things so what we're gonna do is hit then comma cell it's gonna be so let's do same as we did here I'm just gonna initialize this hidden and the cell state then we're gonna do prime yeah prime I guess any show input it's going to be self touch our tensor of initial string we're just gonna convert it into a tensor and then we're gonna do predicted to be initial string then what we're going to do first of all risk we're gonna build up the hidden and the cell state so we're going to for P in range of length of initial string and then we're gonna do so this is gonna be one we're gonna do - minus one on this so we're just gonna sort of build up - hidden in the cell state up to the last character not including the last character so I guess in this case we won't even run this loop well let's say that we have a B or something like that then it would run one time just to prime it for this initial character so what we're gonna do here is we're gonna we don't care about output from this but we're gonna do the we're gonna take the hidden in the cell we're gonna run set Arnon on initial initial I guess in show initial input of P I'm going to do that view one and dark two device and then comma hidden comma cell so really all we're doing here is we're just sort of sending in the input right and all this is gonna do is it's gonna send it it's gonna predict something random but we don't really care about that all we're gonna do is we're just gonna build up the hidden in the cell state and up to the the last character which is what we're gonna send in to start making our actual predictions so we're gonna do the last character is going to be the initial input of -1 and now is when we're actually going to start doing our prediction so we're gonna do 4p in range of predict length we're going to pretty much the same as here but we're gonna send in last character and yeah I wrote dot view here is just we're gonna add an additional access here just to make the dimensions work when we send it into the RNN this is these are just details really so I won't focus on them but yeah so after doing that and this is gonna be the output we're we're gonna do and these are sort of the details to temperature but what we're gonna do is we're gonna do output distance and we're gonna do output data dot view and we're gonna sort of so unroll that entire thing and we're gonna divided by temperature and then we're gonna take the exponent of that and then we're gonna do the top characters is gonna be torched multinomial up output this comma one then we're just gonna take the first character so so what we do here is we do this thing right here we divide by temperature as I said previously to make it make more risky predictions just make more get more fun names and then we're gonna take sort of the the highest probabilities but we're gonna do it sort of a little bit randomized with this torch that multinomial I I don't want to go too in depth on this one you can read more about this if you want but we're just gonna take a the top character which is the highest the one that it predicts with the highest probability but we're gonna have some random is randomness to it so that it might actually pick some things that are not on the highest probability and then we're gonna do predicted character it's gonna be all characters and top character and predicted is gonna be plus equals predicted character and in the last character it's going to be self dot chart answer of predicted character and and then the loop continues right then it's gonna be the last character it's just gonna output another one and in the end we want to return predicted now let's see get random batch itself I'll get random batch and so we want to do here self to generate and yeah I don't know let's copy I guess we don't have to actually run I guess you can just run it like this and [Music] that should be pretty much it I think now let's just actually see that this works right so let's do name engine names our generator and then let's do Jen names train and hopefully this works now let's see okay so so I had this two name's not baby names so let's just run it now and see if it works so it sort of seems to be starting training I'm gonna let this run for a while not too long but just enough so we can start seeing that it generates some names yeah so it's been training for a little bit now I guess not too long just a couple of epochs and or well maybe these are a couple of hundred eighty bucks and we can sort of see that it's generating some name like so I mean some of these are definitely names like Maria and I guess this is a name that's in the training set as well so that wouldn't be too interesting to look at we would only look at names that aren't actually in our training set but yeah we can sort of see that there are some de mean and Avery and let's hear it and I guess and matter Trina on me I'm not that and yeah so there's some some names and I guess some of them look pretty cool and let me bring up so I took out a few some example names that it generated that are pretty cool and this was after training I don't know I guess two to three thousand ones and none of these are in your training data so these are all just ones that it generated so Neela Elia and I thought it was pretty funny Lynette so this is uh I've done that in a previous video if you want to check that out and lay era I guess Bernando Jellico oh yeah so some of them are gets pretty cool names and so that that this learned tab from the training data it's pretty cool yes we see her Elia again and that that's not in the in the training data and yeah so one thing I notice here is a road to not not limited to just Shakespeare and guess not limited just names but actually first I sent in so you can send in any large text file here and in the beginning I sent in a large text file of Shakespeare see if I can find that yeah so I had a large text file here of a lot of Shakespeare texts and yeah so if you train for on this it's gonna generate some Shakespeare like text and really you can send in I guess anything you want it's just write it all in a text file send it in and then you can see some pretty remarkable results but that's that's all for this video hopefully you learned something and it was relatively easy to follow and thanks so much for watching and I hope to see you in the next video [Music]