Fastformer: Additive Attention Can Be All You Need | Paper Explained

Key Takeaways

what's cracking guys in this video i'm covering fast former additive attention can be all you need so yet another uh all you need uh paper and yet another uh like attempt to create a transformer that's more efficient that's hopefully linear uh like complexity with respect to the input sequence token length uh so the reason i'm covering this paper is because i actually like it i think like the idea is fairly simple so what really resonated with me was the simplicity so i guess that's rich sutton uh school of thought and uh aside from that they have awesome results on various data sets and they actually showed both theoretically and experimentally that they have a better uh like efficiency compared to previous work so previous work even though they sometimes have uh allegedly linear complexity they have a huge constant that's hidden behind the big o notation so in practice they are not that useful but i'll dig into those details a bit later so before i do that let me just quickly contrast this work with previous uh attempts to make attention uh the the attention module of the transformer uh more efficient so one group of papers we're focusing on how to make the uh the dense attention pattern of the original vespani transformer more sparse uh but still uh like keeping the the performance that the original transformer has and so you can see in the long former paper they they proposed a bunch of various different uh like attention patterns you can see they are fairly complex and intricate and uh in my honest opinion when i see something that's so complex and hacky i just don't think it's gonna be uh there in a couple of years it's just the current way to boost the the results and i i i still think it's it's useful for the community but i don't think it's gonna work in the long in the long run uh same thing with with uh like reformer here you can see they used uh locality sensitive hashing you can see just the the number of steps you have to do here so lsh bucketing sore by the lsh bucket chunk sorted sequence to parallelize blah blah blah ten within the same bucket so it's fairly complex you can see just looking at the graph it's very complex it doesn't seem like the way to go that's that's one of my main problems i had with these papers uh second thing is the thing i don't like about these uh transformer papers which are trying to make it more efficient is that they are all using different data sets so it's kind of hard to compare apples to apples here so i'd encourage the community to start using like a dedicated set of data sets and try and evaluate all of them on the same suit of tasks that's just my two cents okay so all of these papers uh are basically trying to modify this multi-head retention module so this one here uh so same for reformers say same for a long former same for many of them uh that that was the basic pain point that's where the the quadratic complexity lies in uh and again i won't be digging into how transformer works that's so that's like uh by now a building block of deep learning so go go check out my video on transformer if you want to learn how it actually works so yeah that's that's the motivation we want to reduce the quadratic complexity and now let's see how fast former actually managed to achieve that i think the idea is fairly neat and fairly simple so let's dig into it okay so first things first we have input sequence which are uh embedded into these vectors we have so we have m vectors which represent the n tokens of the input sequence uh we have the usual query transform and key transform and value transform and now the interesting thing happens here so they first uh take these uh query vectors and they uh do additive attention i'm going to explain in a second what that means uh they uh doing additive attention they form these alpha coefficients they multiply the the vectors with the alpha coefficients and they sum them up to to achieve this uh this global query vector which serves as a initial step in in creating this global context of this whole sequence uh so let's see quickly how how the the the additive attention works so we have a vector let's let's take like q1 whatever the additive attention is super simple you just take the input vector so q1 and what they're going to do is they're going to just feed it through a fully connected network so every single dimension in this vector every single feature is going to have associated uh like a link here with a weight and you basically form that's how you form the alpha coefficient so alpha i in the general case we basically have the operations in here where these the number of dimensions of the vector so you can see it's fairly simple and computationally super cheap to form these alpha coefficients which as i said then you just use them to do a scalar product of the vector and you form a novel representation which is here and then you just add them up you add add up all the vectors you form the q vector uh that's it that's how you form the q vector then you what you do is you take the q vector and you modify the key vectors by doing element-wise so this this symbol here is just element-wise uh product uh they did some ablation studies we'll see that in a couple of minutes but basically the concatenation of this global q vector or the addition uh did not work as good as as doing this uh like uh uh element-wise product basically uh once you do that you form these p vectors and then you again do the additive attention so the same thing the same procedure as here and then you add them up and you form the cube the the the k vector so that's the key global vector finally uh the the final step of this of this module is uh you basically have value vectors uh what you're going to do is again do the element uh product you're gonna form the u vectors and this time instead of doing additive attention uh they did something similar to the original transformer i.e they did simple linear transformation here so this block is just uh like a set of linear layers or or mlps i think i'm not sure where they're using single layer or mmp but that doesn't matter so basically you have a simple transformation which is learnable you form these r vectors and then finally you just add them up with the query vectors which we had here to form the final output and that's it so it's much more simple compared to in my opinion uh compared to long former compared to limformer compare which is the lower rank approximation of the matrix compared to every other paper pretty much uh yeah so that's a big plus in my book at least okay so everything i just explained is uh written down in these formulas uh the only thing i forgot to mention is they have the same as in the original transformer they have this uh division by the square root of the dimensionality which uh serves to keep the things stable and so this is just a scale dot product uh and that's how i formed the alpha coefficients and then as i said you just do weighted uh like uh some of these query vectors to form the global query vector and that's it you do the same thing for for the key for the global key vector as i already mentioned it here so that's this one and finally i explained the third step and that's this linear transformation part i explained okay so um a couple of details which are interesting they did some ablations again uh they mentioned here we shared the value in query transformation parameters to reduce the memory cost in addition we shared the parameters across different fast formal layers to further reduce the parameter size and mitigate the risk of overfitting so this is nothing new we had a bunch of papers prior to this one which were experimenting with this weight sharing and so nothing new there but still um that's this is additionally going to save up some parameters uh and they also didn't have any uh slowdowns doing this so that's that's cool so if you followed along the explanation of how this thing works it's easy to notice that the whole complexity will be n times d so that means it's linear with respect to the number of tokens and d is just the dimensionality of the vectors which are being passed through the transformer okay so funny thing i just kind of had to notice this this must be the most unlucky place to put the uh footnote but yeah i guess they did it on purpose hopefully they did it on purpose okay uh so having said that let me just show you the complexity compared to the other previous papers so we had transformer the original transformer which was obviously squared compared to the input sequence length and that was the problem that uh that make the yields transformers um not good enough to to do inference on long sequences then we had a bunch of various papers so i mentioned long former i mentioned uh i don't see reformer here but yeah uh linformer and other papers so as i said there is always uh all of these papers pretty much had some hidden caveat inside so for example long former had this k and if i remember correctly you had to push k uh pretty high in order to get decent performance boost so maybe something like square root of n would be optimal and so you you don't you you kind of have linear complexity but you're not quite there uh same with other papers you can see some some have uh square dependency on the dimensionality some have a huge constants in front of them which you don't see because the big o notation just eats it up so i think fast former is the first paper to actually have decent linear complexity okay let's finally see the results so they have two sections first one is they show that the fast former is effective and then they show that it's actually efficient as well so they had i think five data sets here to show on amazon imdb in mind uh so sentiment and topic classification tasks and you can see that first former uh pretty much outperforms all of the previous baselines on all of these tasks um except for this one where i guess the difference is fairly negligible and the standard deviation here is even bigger so yeah uh but like as i mentioned the thing i don't like about everybody's using their own data sets it'd be nice to have like an agreement of which data sets we should use when we are evaluating these uh efficient transformers uh just putting it out there maybe maybe somebody uh starts implementing that idea okay so that's the uh being effective and uh here are some other other data sets they they used so they have the i think recommendation task here and they show again then fastformer is better compared like across these various different metrics and i don't don't want to even get into what these metrics are but basically it's better and they have this combination with this plm and our uh baseline plus and sample disaster just signifies uh ensemble and they have even better results they achieve even better results by doing that but yeah i guess this this row is what is actually uh important for us additional results on text summarization again fast former is fairly better compared to all of the previous uh baselines across rows one rows to row gel metrics so different metrics again not that important for for this paper uh before i show you how efficient this fast former is let me just kind of dig into this part so in our experiments we use glove embeddings to initialize token embedding matrix um i'm not sure why they're using globe here so this is like a alternative to where to back embeddings i'm not completely sure why they did not learn the whole thing from scratch if anybody knows what's going on here let me know down in the comments because i'm this part confused me to be honest okay let's see how efficient this thing is so looking at inference looking at training graphs here so we can see that as the sequence length increases somewhere around 512 the transformer starts exploding so the gpu memory uh is not large enough i think they were using v100 gpus nvidia and uh so basically that means you cannot actually even evaluate the transformer on these larger sequences on this longer sequences um then you can see that this dark blue uh like uh line is the fast former and it's way better compared to the other base lines so this pulling former if i'm not wrong let me just see the theoretical complexity they they reported so pulling former n times d times w uh where w is the window size and you can see that even though this looks linear in practice it's not it's not like optimal because they have huge constant and i guess the w also contributes to that constant exploding uh so yeah in practice it's does doesn't seem to be as good as as as all of the other baselines forgot to mention uh we have here on the y-axis inference time per layer and here we have training time per layer i guess it's correlates pretty much these two charts that basically means it's better it's more efficient for both training and in france compared to the other baselines and i guess that's nice results finally here are some ablations i mentioned uh the element-wise product they were using to construct those global vectors you can see that compared to adding up those vectors instead of doing element wise or concatenating them to the key vectors uh is worse uh looking at the accuracy on different data sets so the it's the trend is kind of similar and sticks uh throughout all of these different data sets uh finally uh i mentioned weight sharing they did ablation studies there so without sharing you can see uh decent results but i guess the number of parameters explodes um with cue query value sharing uh some improvements and then uh when you do the headwise sharing i think that's that's important we have a significant drop here and the same trend goes for every single data set and the whole point is if you read the original transformer paper everything and if you ever saw those uh attention visualizations every single head is learning different sub function so uh sharing the weights there kind of hurts the performance and that would be some hand-wavy explanation of why that happens so every single one needs to learn a dedicated function uh for example if you have a sentence uh one of those uh like heads where maybe will maybe focus on attending the nouns one will be attending like verbs i don't know whatever so different semantics all in all so that will be a quick uh explanation of this uh fast former paper so again as i said what i like about this paper is it's much more simpler uh like implementation wise and just understanding how it works compared to previous uh like papers i've covered and read as well as they obviously have both theoretical and experimental guarantees that they're more efficient and they're also effective uh like across a bunch of different data sets as we saw so yeah that's pretty much it um and finally if you missed it i just created a brand new discord community so do join it i'll link the link in the description so basically you'll be able to engage with others there ask questions related to ml i'm actually working on bringing startups to offer ml jobs so that's something that's interesting to you do join the community and also subscribe to my monthly ai newsletter which is going to be super valuable hopefully uh having said that subscribe to this channel hit that bell notification and until next time bye bye [Music]