How do transformers work? (Attention is all you need)

Key Takeaways

what's up folks so uh over the last uh month uh i've been reading and learning about transformers so i read a bunch of research papers uh i started implementing the original transformer paper from scratch just a week ago so i thought well it's still fresh in my memory i should go ahead and do a small deep dive into the attention is all you need paper so without further ado let's jump into it so let me open the paper attention is all you need so actually before i go and do a high level overview of this of this paper i strongly recommend you go and read these three blogs which i already briefly mentioned in my previous video which i'll link somewhere here so it's from jail jay elmer and uh they are super useful uh if you're if you're new to transformers and to your attention mechanisms and stuff like that so this one is really nice uh introduces you to the concept of word to back to like virgin betting's in general and then there is this sect to sec uh tutorial which is also really nice and finally the most relevant one is this uh illustrated transformer which is really really nice so take your time uh read through those and then um you'll have the the required uh prerequisite knowledge for understanding this video um although you can try and do it without reading the blogs i just strongly recommend you go through the box uh also so yeah i already said this so like a week ago i started implementing the original transformer paper and you can see it here my coding history so hopefully all of those nitty gritty details will be like fresh in my ram so without further ado let's let's jump into the paper okay so let's first do a short high level overview of the whole paper so in 2017 a team from google uh published this paper called attention is all you need and they basically introduced this novel architecture called the transformer uh the main idea there was that they start stopped using rnns and cnns and instead used a simple architecture but they used attention in a really clever way that enabled them to have models uh transformers which are much more parallelizable and thus more compute optimal than previous architectures and they basically show here uh that yeah they basically say hey this is this is better than lstms and better than gru's in modeling long range dependencies among words and also much more compute optimal you can see the architecture here and how it functions on the higher level is you basically take the input sentence you convert it into embeddings you add something called positional encodings which i'll show you right now because i've been coding that up from scratch on my own so here's my implementation and just visualize it because it helped me also understand better how these look like and a bit what you basically see here is that every single row is something that gets added to the token representation and using this information the model knows where the all of the tokens came from otherwise the the word order would get lost so this is really important and they didn't use the the learned representation for this positional encodings they just use this this these are basically sine and cosine functions uh whose frequencies form a geometric progression and it turned out to be as good as learn learnable uh positional encodings which they showed in ablation studies later in the paper okay if enough of that let me go back to the paper so then what you do is you pass those representations into multi-hat attention which i'll go into a bit more depth a bit later and they then apply something called 0.5 speed forward network finally the output encoder representations are fed into the decoder which is super similar to encoder part except that it has this middle multi-head attention which actually attempts also to encoder uh representations and not only to uh previous representations from the same stack so that's that's the difference and uh this bottom uh multi-head attention also use uses something called like uh masking basically uh you the the the like the car and token can only see representations that came before it and that makes sense because transformer was used to do some machine translation so translating from english to german and english to french and thus it doesn't make any sense to see the like future words because you're trying to predict the future words that's why the mask is used and i can show you that here just a second so this is how the the mask looks like you basically put some really big negative numbers uh into the before the soft max indian attention module and that causes uh masking of token representations okay so that was brief overview of the architecture uh yeah finally you have the linear layer and softmax which creates a distribution and you're just trying to minimize uh the distribution either using cross-entropy loss or kl divergence in the case of soft labeling uh which i'll explain a bit later and hopefully i i know this probably sounds gibberish at the moment but just stick with me and you'll get to kind of slowly learn all of those details uh in time okay so attention i already mentioned it they are using something called scaled dot product attention i'll go i'll explain that a bit later and uh point wise feed forward networks i mentioned those so basically what they do is you you have a single network and it processes a a single token representation and that's it you just take us you just train a single network and you just do something like a for loop over all of the token representations embeddings and softmax positional encoding i showed you how this looks like so basically they they use this sine and cosine functions with frequencies which form geometric progression and then they argue why self-attention is good so why it's good is it's computationally really a really really nice uh thing so basically if you compare uh the second row recurrent uh network with self-attention uh you have that the maximum path length is o of one for uh for transformer whereas it's so fan for rnns and similarly complexity of a layer is n to the n squared times d where n is the number of input tokens which is usually smaller than d which is the dimension of the model which was 512 in the baseline model so that's that's why it's so basically attention plus highly parallelizable that's what the what's the secret sauce of transformers um okay they then just they train this model for uh translation tasks as i already mentioned and they achieved sodas so state of the art on both english to german and english to french translation tasks they also use some some really interesting uh optimizing uh learning rate schedule and i can show you that briefly in this super nice blog called the uh let me find see if i can find it the annotated transformer and i'll link the link uh down in this description so basically this is how it looks like you basically have you linearly increase the learning rate initially for for the warm-up number of steps and then you start decaying them uh proportional to proportionally to this uh inverse uh square root of the number of steps so that's how the learning rate schedule looks like for their transformer and they use a simple like a pretty popular optimizer like first order optimizer called atom finally they did some regularizations dropouts label smoothing which simply said you you usually want to match like a one hot vector what label smoothing does is instead of putting one on the like ground truth word you put 0.9 and just spread out the rest of the mass across uh the rest of the vocabulary and that's that's it in a nutshell so i mentioned machine translation that did ablation study this part is interesting so they figured out that bigger models are better and like fast forward to 2020 uh you you know you you know this and you have gpt family of models where gpt3 has 175 billion parameters and you have bird and bird derivatives and all of those models are huge compared to this one i think it was the original transformer had around 100 million parameters so compare that to 175 billion today in 2020 and finally they conclude uh saying uh they have uh ideas of using this same architecture also in vision and speech recognition and again fast forward to 2020 you have this visual transformer which performs really good really nice with images and that's super exciting although only in the big data regime but still finally they have some nice visualizations here showing how different multi-hat attention modules attend uh different words in the previous and the previous layers so you can see here that the masking this word masking oh sorry making uh is focusing a lot on more difficult which makes sense if you look at this semantically and yeah they're just nice interesting emerging patterns that that appear uh when you train these transformers and networks so with that being said uh let's go a bit into more depth about how the multi-head attention actually works okay let's let me try to explain how multi-hat retention works and i'll use two methods here i'll first use uh jl and mrs visualizations because they're really nice and i'll use my own code i've been developing over the last couple of days so let's see how how it works so basically uh you have uh so let me let me open the paper you you we said you convert the sentence into into a sequence of embeddings right and you're basically here and now let me show you how the multi-head attention works so these are the embeddings so you have say thinking machines and in in real world they you won't actually have a single vector for a single word there is something called tokenization and concretely a transformer use something called byte pairing coding meaning thinking will actually be split maybe into two vectors like one uh which which actually uh corresponds to t-h-i-n i don't know or and the second one for the uh k-i and the third one for n-g whatever so but basically at the end you have a sequence of vectors and what you do now is you apply three neural networks uh one will produce something called queries the second one will produce something called keys and the third one will produce values and now you basically will use the same network for every single uh token vector to produce queries so there are only three networks in one multi-head uh attention and there is one more uh like at the output but i'll mention the one a bit later so what you do now is you want to find how similar a query from a particular word say query one is similar to uh two keys of every single token in a sentence so you do a dot product so that's more generally called a scoring function but the the transformer paper use something called scale dot product scoring function so you do a dot product between k query one and key one and you get number say 14 and then you do a dot product between query one and key two and you get twelve and those are scores and now you do soft max which turns the which makes the sum goes to 1 and you get 0 8 8 and 0 12 and basically you take those numbers you multiply the value vectors value 1 value 2 and j nicely visualize it here so because this is 0 12 basically this one will be dimmer and then you sum them up and you get that one so that's how the attention works but now the only difference with multi-head attention is basically you split this query one in the baseline transform model it has 512 dimensions once you split them into eight heads that's the number used in the paper you get so one query one will actually have 64 dimensions and then you do the same procedure i just described but independently for all of those eight hats and once you get z ones for all of those eight hats you just concatenate them together and you do you apply another you do another feed forward through a neural network and you get the final result and that's what gets that's that's when you get to this part here before uh addition with this residual connection so that's how it works uh in a nutshell now i mentioned the scoring functions i mentioned dot product why do you do a dot product between query one and v1 and v2 whatever uh the the reason is that gives you a similarity between those vectors so what i mean by that let me let me use google for this so this is a 2d case so basically i told you so those vectors are like uh so query one and key one or key two uh they are they have 512 dimensions but if you kind of maybe uh project them into two dimensions uh and you get something like this and now the smaller the angle the the the more similar the vectors are and basically if uh so so the smaller the angle you you'll get higher score here and basically that means uh you'll attend more to that particular representation so i hope this didn't confuse you too much uh because that wasn't the point but yeah let me know if this helped or it did not just reiterating once more you take queries and you take keys and you want to see how similar they are and if if if certain query is really similar to a certain key then you wanna take into account uh that vector with a high score and basically if vectors are orthogonal the score will be zero if vectors are uh like even on the opposite side then you have a negative score so that's basic basic uh cosine distance basically jumping into code i want to show you how it looks like uh like in flash so as i said here so you have these uh query key and value nets you uh apply them separately on all of those tokens and you get you get a query key and value batches and once you do that you pass that into the attention oops you pass that into the attention function which does some matrix multiplication and that way you basically uh apply attention to all of the tokens and throughout through through all of the batches in a really optimal way but it's harder to understand than by looking at a single like single batch and examples that i showed you in the blog then we have potentially some masking operation and as i mentioned uh in in case you want to ignore a certain representation you just put negative infinity oops this is should be negative here uh it's it's still up it's still a prototype code so forgive me for this um and then you apply softmax to scoring functions you get the attention weights and also one more one more detail is that you you sometimes wanna drop certain attentions uh attention weights which means certain um attention well we get uh pulled to zero and that's one one more nitty gritty detail of implementation and then you do once more matrix multiplication between attention weights and between those values and here is where you get those um so here is where you get those uh let me just open this up uh these vectors and now you want to concatenate them and do one more pass through a neural network to get the final result and that's exactly what i do here so intermediate representations and then we change the view of those and finally we do another projection and you get the final results so i hope this wasn't too uh too hard to follow along if it was please let me know in the comments and i'll be open sourcing this code in a week or two so you'll be able to go through the code at your own pace and read through the blogs at your own pace so hopefully this will all of this together will help you grasp the how all of this works and fits together uh finally i just want to give you an like a feeling of how the actual machine translation training for the original transformer looked like so you basically have two core processes of text you have say german and you have english and those corpuses are synced meaning you have one sentence in german and there is a corresponding sentence in english so for example vicenzi einstein that was a bit of german for you and you basically have a corresponding sentence like uh here on the other side of the document if i open it up you know one of the intense pleasures of travel blah blah blah so having those two uh synced so that's the so basically if you're translating from english to german the german sentence is something called like the gold translation so somebody like a professional translator translated the sentence okay so once you have those two sentences that correspond to each other what you do is the following so you have the the transformer you take the english sentence you tokenize it you embed it you you do a forward pass through the encoder module here and you get representations on the other side you take a german sentence you again tokenize it um but now what you do is you set you you put the as a prefix you put this uh start of scene a sentence token uh so that's here and then you do uh forward forward pass here and finally uh the golden the gold translation uh just you convert that into soft labeled uh distributions and you just do kl divergence and then doing back prop will train and tune the weights of the whole transformer model and that's a single uh training step uh the important part is that you want to use those masks so that a certain token like say a token that's here cannot see tokens that are coming after afterwards that being said uh if you found this video useful uh consider subscribing to this channel and hit that bell icon to get notified when i upload a new video and until next time keep learning deep you