Breaking down the OG GPT Paper by Alec Radford

Key Takeaways

Okay. Sure. So everyone, uh my name is Amgad. I'm a machine learning engineer. Here, I generally do ML consulting services to startups. I help them like shape AI powered products, especially in the field of NLP and speech-to-text applications. And I run a pro- uh blog where I like publish posts about ML stuff. So feel free to check it out. I've done some posts about Whisper. Uh yeah. So with that out of the way, let's get directly to what we want to discuss today, which is like the GPT-1 paper by the folks at OpenAI. So uh the paper is titled Improving Language Understanding by Generative Pre-training. It was published in June 2018. And these are like the authors. Let me switch to presentation mode. Yeah. And these are like the authors of the paper. So uh mainly Alec Radford and Ilya Sutskever. Very well-known publishers in the ML field. So let's get started. Uh back in 2018, like uh deep learning was was was becoming very popular. But the main uh thing with deep learning is like it is very very data-hungry. And there are like good news and bad news. The good news is like there is a ton of data available everywhere on the internet and just online. We have like tons and tons of data. The bad news is is this data is not annotated and it's not curated and it's basically very very messy. And if you want to train machine learning models back then, the only way to do it is just to annotate data yourself or hire like data annotators. And these tend to be very very expensive and difficult to scale on hire. Like if you think GPUs are expensive, you have not worked with data annotators. That's what I like to say. Uh so this this makes like deep learning very restricted. You cannot use it everywhere. Like you're only restricted to fields that have good high-quality annotated data sets. Uh and this is a very big bottleneck. So, people back then in 2018 are trying to solve this problem, like how do we how do we get like over the fact that we need labeled data? And one potential solution to this problem is like unsupervised learning. So, the question is what if you can leverage like just the linguistic information from the unlabeled data? So, we have like a bunch of text, uh like novels, books, articles. How how can we leverage like the linguistic information from this? And the answer to this could be using unsupervised learning. And if you can do this successfully, this alleviates the need for large amounts of labeled data. Uh because basically you can utilize Wikipedia, which is very big, or even uh all the papers published on archive and so on. And even if you have like lots of labeled data, using unsupervised learning as a step one or step zero is going to make your model actually perform better than just training on this labeled data because of transfer learning, where you can transfer the things that your model has learned during pre-training into your actually objective. And a good evidence of this is back in 2017 and 20 or even a bit a bit before 2017 is like people have been using pre-trained word embeddings, so things like word to vec, glove, fast text to achieve very good performance on many tasks, like classification and machine learning, uh sorry, machine translation. So, this this is a good evidence that actually unsupervised learning works and it's a very good approach. So, uh this is the premise of the the paper, actually. Like unsupervised learning is very very promising. So, let's talk a bit about word embedding. So, the idea behind word embedding is like you want to project words, so just text into an n-dimensional space. And n is usually 300 or 1,000 back then. And this space has a very special property that words that are similar in meaning have very similar products. Sorry, very similar vectors. And by by by similar vectors, I mean with like you can measure similarity by things like cosine similarity dot product or like L2 distance. Uh so, we can capture sim- similarity between words that look like each other, but have similar meanings. For example, the word booking and reservation. Just from the syntax, they are very different words, but they have very similar meanings. We're just booking something. And similarly, the word Adam and SGD. These are like completely different words, but they are both actually like optimizers and used in machine learning. So, they are very similar and their vectors are going to be uh probably similar. The most common implementation of word embedding is like Wave2Vec by Google. Uh this is what popularized popularized the usage of word embedding. And then Glove by Stanford and FastText by Facebook. And the way these these word embedding models are trained is like uh leveraging co-occurrence between words that have like similar context. For example, similar words tend to be like tend to occur in similar context. Like you would generally find the word Adam associated with learning rate or machine learning. Similarly, SGD is is associated with learning rate as well. So, we can conclude that these two words are kind of similar. And the way these word embedding are used is that they are like utilized by training a head. Like for example, if you want to classify a word as being positive or negative, having a positive sentiment or a negative sentiment, you can train a classification head on top of the frozen embeddings. So, you use the word embeddings like as a fixed uh feature input input features. Uh, just frozen things. Without training the word embeddings embedding themselves. And this has like a significant drawback or a bunch of significant drawbacks. The first thing is it doesn't utilize the context of the of the text. So you're just using the word itself. And some some words even the same word can have very different meanings depending on the context. For example, the river bank like the Amazon bank or the Nile bank is is different from the HSBC bank or like the JP Morgan bank. Even though these are the exact same words. So if you're going to use just wave to back these two words will have the same vector even though they are like a very different context. And and even beyond this like natural language has like nuances that cannot just be captured by using words like the way you write things like grammar three when and skill issue and open your eye the way you write it this way like you have a very specific intonation compared just to writing open eye in the normal way. Uh, so are there any questions about word embeddings so far? Uh, someone says in the chat nope. Um, I cannot read that chat. So if just someone can say this in the microphone that would be great. any questions. I think Sean said no. But if anyone has any questions you can maybe drop it in the chat. And then I can just have them ask me when. Uh, but it seems like everyone is okay right now. Yeah. I'll keep an eye on the chat for you. Yeah, that's great. Yeah, thank you. That's appreciated. So this this was word embedding. The the main limitation or drawback is like you cannot use context. So can we go beyond word embedding? Like word embeddings are very like too local. We need something that's more global that can capture the higher level semantics. But this also has as its complications like How do you leverage more than word level information from unlabeled text? You're going to have like some some questions that questions that you need to answer. First is like which objective should you use while training? Do you want to use language modeling or machine translation or like discourse coherence or something else? And like in in 2024 right now we definitely know the answer like language modeling works very well very well because this is what we have been using for the past 3 years. But back then in 2017 and 2018 this was not very clear. Like for example this paper came out before BERT and even the the original machine learning the original transformer paper attention is all you need used machine translation. So back then this definitely was not very obvious to people. The second question is how should we transfer these embeddings like to the target task? Uh there are like bunch of ways to do this. Do you want to you like do you want to modify the model architecture? Uh very like each task is going to require each specific modification to the the model itself. This is one approach and this like requires very deep knowledge of model architecture and like being just a wizard to modify the architecture. The second approach is like to use a specific recipe or schema to do transfer learning and a very popular example of this is ULN fit uh by Jeremy Howard and Sebastian Ruder and we're going to cover this in the next two slides I think. The third option is also you can add like an auxiliary learning objective during pre-training. So while you're pre-training on language modeling you can have an an auxiliary learning objective like machine translation or discourse coherence. Uh so these are like some of the approaches you might want to take when you are deciding on the target task. So these these all these questions may just going beyond word embedding not not so straightforward. And they made it like difficult to utilize semi-supervised learning or or unsupervised pre-training. So, let's take a look at ULMFiT and how they did this. So, this is this is a paper titled Universal Language Model Fine-tuning for text classification. So, their their objective is text classification, but they want also build a universal model. And this is a very seminal work in NLP. It's a very well-known paper and it has a big impact. So, the question they are trying to answer is or the question they are raising is instead of just utilizing the word embeddings uh like you're going to have a classifier, you're going to have an embedding layer and a classification layer. And the old way of doing this, you're going to use the embedding layer from word to vec and you are going to randomly initialize the classification head. So, they are asking why not just have a good initialization point for all the layers, not just the embedding layer. And their answer to this question is an approach called ULMFiT and it is a three-step recipe for like state-of-the-art text classification back then. So, we have like three steps. First step is like to train the language model on general domain data. And we call this pre-training on large on large corpus these days. So, you just train your language model on a very big corpus like WikiText and this was big back then, but it's probably small now. You can do pre-training on like 15 trillion tokens of data if you are like a big organization like Meta, for example, or even more. Uh so, this is like the first step. The second step is you do fine-tune the language model on your target data. Uh and you you keep doing language modeling in this step. So, this is kind of like similar to what we call continued pre-training on target domain. So, you just take your llama 270B base and you just do language modeling on, let's say, financial books to try to make your model like Bloomberg GPT, for example. Uh but you're still doing language modeling. You're not doing any task-specific training. And the third and final step is to train a classifier on your labeled data. And this is like the fine-tuning step that we are all familiar with. So, if you have If you want to like, let's say you want to classify Amazon reviews as being positive, neutral, or negative, you're just going to get maybe a thousand reviews and the the labels for this for each review and just train the model on in a supervised fashion. And this was a very good paper and was very influential and uh made a big like a big buzz in the in the ML field, but this paper was was released in 2018, but it did not mention the word transformer at all. Like, they did not even reference the paper. The architecture of the model used was an RNN-based model. I think it was an LSTM. So, no transformers at all. Uh and this is like this is kind of a big big gap in this work that GPT folks are going to fill. Otherwise, this paper would have been like very very good paper. It still is a good paper. So, that was ULMFiT. Uh so, let let's talk a bit about GPT, the thing that we want to talk about today. So, GPT stands for generative pre-trained transformer. Uh and the keyword there is transformer. It was developed by OpenAI by these awesome folks. And it was actually one of the first things that made OpenAI like a popular organization in the ML field. And the whole premise is to use semi-supervised a a approach for N L U tasks, natural language understanding tasks. Uh so, the goal is to learn a universal representation that transfers transfers very well to other downstream tasks. Uh so, basically have a good starting point instead of like starting from scratch every single time. And their approach is basically two-step. First step is to do unsupervised pre-training and then supervised fine-tuning. And this is kind of like where the word semi-supervised is. It comes from like you have a bunch a mixture of unsupervised training and supervised training. And their architecture is transformers, of course, because uh we have GPUs and GPUs love transformers. So, that's a good thing. Uh So, to do this approach, you you're going to need two things. First thing is like a very big corpus of unlabeled text. And then the second thing is like a data set that is annotated, that is like ready to use for supervised fine-tuning. Or you could have multiple data sets if you're going to train your model for several tasks. But the good news is your target tasks do not need to be in the same domain as like the unlabeled text. So, for example, if let's say you want to train on financial task like you're going to give the model some information about a stock and ask it about how it performs, uh whether or whether you should buy or sell. So, like a financial classifier. Actually, you can pre-train on just like normal general data. Like you can pre-train on a corpus from the internet and then just fine-tune on the on your desired task. Like your your your your unlabeled corpus does not need to be in the same domain as the uh your objective. And this is good news because we have a lot of general purpose text that you can use for pre-training. While obtaining very domain-specific corpus is more involved. And a very minor note here is like the name can be misleading in this work like the word generative here mainly refers to like the pre-training step. The actual tasks that they had in mind are more discriminative so like classification, question answering, and semantic similarity. Uh that is natural language understanding tasks. So they did not work like they did they did not discuss machine translation or just being a a chatbot in this work. And they actually released their blog post under a different and I think more fitting title called improving language understanding with unsupervised train learning. And this is like the key This is the key idea here like unsupervised learning. And this is a very nice blog post on their on their blog. So now we have discussed ULMFiT and GPT. Let's let's also discuss some of the other related work in this domain. So let's first start with semi-supervised learning. Uh the work GPT falls under this domain. And back then this was becoming very popular like sequence labeling task text classification tasks. People were doing semi-supervised learning for these. And you can you have like different levels for this approach for so like the first and basic level is just using uh statistics language statistics to do to get the features like use bag of words TF-IDF and all of these cool classical machine learning stuff. You can use them as input features and just train a classifier on top of this. Uh but that's not very helpful because like two words that are different in syntax but are similar in in semantics are going to have very different features. Uh so makes it a bit uh the model is a bit limited. The second step is to use the word embedding like we discussed previously. And this approach allows you to capture the semantics, but it's also very limited in that it's a based on words. We we're not capturing the higher level semantics. The third level is like a sequence embeddings. So, instead of just using the the word to get the embedding, you you're going to utilize the entire sequence. Uh so, like the sentence of the paragraph get the embedding. And this actually allows us to utilize the context to understand like the high-level semantics of the text. And this is like level three is is what where GPT falls. Like we are using the the entire sequence to generate embeddings that we can use for classification or any other task. So, this is regarding semi-supervised learning. The second field and the more specific field is like unsupervised learning. So, it's a special case of semi-supervised learning and these terminology these terminologies can be confusing, I know. But, the goal is like to find a good initial starting point instead of just And instead of just going directly to do an supervised learning objective. Uh this is a typo, probably. So, the early works explored the the early works in unsupervised learning was actually used in in vision in image classification. So, for example, you can use a ResNet that was trained to classify ImageNet. You can take the backbone the ResNet as a backbone and then do like segmentation or just try to detect pneumonia in chest x-rays. And this is a very good starting point. Even though ResNet was was even though ImageNet is just classify classifying images as being cats or dogs or or humans or horses. So, it's more of a general purpose data set that you you learn a very good representation that can uh, be used in your downstream task. And in the field of vision, this actually proved to be quite well. Uh, because people have found out that pre-training acts as a regularization scheme. Your model tends to have better generalization if you pre-train it on a very large corpus. And the authors mentioned the paper that the closest line of work to their work is actually what we discovered what we covered so far, the ULMFit by Howard and Ruder, and also another, work by Dai et al. But I did not go into I did not go go in detail into this work. However, the the main the main drawback of these two works and everything else, almost everything else back then, is like they are using RNNs, and we know that by 2018, like Transformer reigns supreme because we have GPUs. Uh, so we can train Transformers more efficiently, and also RNNs have like they are limited in their ability to process large context uh, because of like the gradient vanishing problems and so on. So, this is like the most common approach back then. Another approach is also to use the hidden representation from a pre-trained language model or machine translation model as auxiliary features while training your model. So, you can have let's say you have a, a machine learning model. You you're going to get the representations or like the the hidden state of this this model, and then just uh, give it to your classifier as like, uh, additional features. And as you can imagine, this is like this involves like a a substantial amount of work and new parameters for each task. Uh, so this is not very universal. So, any questions so far before we actually go in detail into the approach? Um, I think there was a question from Six. Uh did AlexNet use GPUs? And were the Transformers the first ones to use a GPUs? I think he just put it in the chat. Yeah, but I added yeah. Were Transformers like someone else Yeah, AlexNet AlexNet, yeah. Yeah, yeah, AlexNet did definitely use uh GPUs, but back then I think they were writing uh CUDA code. I think Alex Krizhevsky was the person doing the GPU programming stuff uh himself. Uh so yeah, I think AlexNet did definitely use GPUs. If I remember correctly. Yes. Okay. Okay, cool. Oh, seems like they did. Okay. I would just ask if anyone else has any other questions, but I think we should be good for now. Yeah. Someone say according to papers with code data, yeah, I think they did use GPUs. So, let's let's go into details about the the GPT approach. So, we have two steps. The first step is unsupervised pre-training. So, basically, the goal is to train a high-capacity language model on a large corpus of text. Uh the training objective here is language modeling, that is given a sequence of tokens, try to correctly predict the next token. Uh so, let's say the cat sat on the you're trying to predict the next token which which I think should be uh the mat. So, this is basically language modeling. Uh the loss function they used is negative log likelihood. Uh it's also called cross-entropy. Uh basically, uh this equation, the negative log likelihood of the correct label. And you do this over a very very important note here is you do this over the entire sequence. So, if you have a sentence that said the cat sat on the mat, you do this over every single token in this sentence, and this is very important. Uh so, you are training on your entire input to your entire corpus. Uh, not just the last token. So, the architecture is like a transformer. Multi-layer transformer, but they are using the decoder only. They are not using the encoder. So, and the difference between the encoder and the decoder, I think, can be summarized to like how you do attention. So, in in in encoders, uh, every token has access to every other token, but in decoders, every token has access only to the tokens that came before it. So, you only have one-directional attention. I think it's also called, uh, left-to-right attention or right-to-left or like you only have attention in one direction. And basically, the the transformer architecture applies like, uh, multi-headed masked self-attention operation over the input tokens, and and then this is followed by a position-wise feedforward layer. And you keep doing this for like n, where n is the number of transformer layers, and then you just, uh, generate an output distribution over the vocabulary. So, let's take this into a more detail. You have your input text that has been tokenized, uh, into tokens. And these tokens have been encoded. So, you have like the token integer or token ID. So, it's a number. So, you you take this number and you pass it through the embedding layer, the token embedding layer, also called the the semantic embedding layer. And you get an a vector for this, uh, token. And you you also get the positional embedding for this token. You you sum these up, uh, like just vector addition, and you get the your input to the to the first transformer block, which is h0. So, just the positional uh, the positional embedding and the token embedding added together, and you get your input h It's zero. And then you take H0 and and pass it to each block each transformer block. And the output of the first block is going to be the input of the the second block and so on. And this is what this equation says. Uh and at the end, once you've done going through all these blocks, we're going to go to the the output layer, which is actually a reverse embedding. So, you have a vector, but you want to go back to a token or or like, I should say a probability distribution over all the tokens. And once you have this distribution, you just pass it through a softmax to get uh actual probabilities that sum up to one instead of just logits or scores. And I think we've covered the Transformer paper, so I think people are familiar with this, but if you have any questions, please go ahead. A small remark about this is that we use tied input and output token embeddings, so the embedding layer in this step is the same as the one used at the final step. Oh. Yeah, someone says causal masking. Yes, exactly, causal language modeling. So, the second step is supervised fine-tuning. So, the goal of this of this step is to adapt the parameters of the pre-trained model to your supervised target task. And for this, we need a labeled data set where each instance is a pair of like you have the sequence of input tokens and the label. So, for example, an input sequence could be this product is very bad and your label could be like a negative sentiment. Uh so, the inputs are passed through the pre-trained model to obtain the final Transformer blocks activation. So, if you go back a bit, you take your input sequence and pass it to through this same Transformer and get the output of the final token. And then you compare this to your label. Uh so you're going to get the hidden representation of the last encoder layer at the M token where M is like the final token the input sequence. And you just pass this through your classification head or whatever head you're using. So for example in classification we're going to use a softmax on top of a linear layer to get your output and compare it with with the label. Uh and you are using roughly the same loss function which is negative log likelihood estimation or cross-entropy. And a key distinction here between this and the previous step is you are only calculating the loss over only the output token not the entire sequence. So the loss is only on Y not on X1 or XM and so on. So the only extra parameters you need for this is is the your classification head. So for example WY in in if you're trying to do classification the uh parameter metrics of the the metrics of the parameters of the output layer. And also embeddings if you're adding new tokens. And we're going to see this in a bit. Uh something they also experimented with is auxiliary training objective. So they also use language modeling as an auxiliary objective in So not just this uh not just this classification but also language modeling. And they say this helped them uh by improving the generalization of the supervised model and accelerating convergence. They also say this is in line with prior work also observed better performance uh when when using it auxiliary objective. And the way you do this is your loss function is now a sum of multiple loss functions where one loss function is this one, the classification loss function, and also you have the language modeling loss function uh with a certain weight like lambda here is like a weight and lambda could be for example to 0.5 or 0.3. So, you have like uh a summation of multiple losses. And a small note from myself here, I'm not sure if auxiliary language like auxiliary objectives are popular today. I think people just do uh supervised fine-tuning without an auxiliary objective. That's just my take. Uh so, any questions so far? I think you I think the chat seems not have any questions. So, maybe you want to just you can just continue for now. Sure. So, now we have discovered uh we have covered the approach, the basic two steps. You will now get a very good let's say a classifier because you have done unsupervised pre-training and supervised fine-tuning on classifiers. But, GPT-1 is actually trying to be more It's trying to be a more of a universal model than just a classifier. So, they are trying to handle multiple tasks like classification, entailment, semantic similarity, and answering multiple-choice questions. And these are all as you can see discriminative tasks rather than generative tasks as we discussed before. So, for tasks like classification, this is very easy. You can do what we have covered so far, just add the head on top and do the classification. But, other tasks have like different structured inputs and outputs. So, for example, text entailment has ordered sentence pairs. MCQs have a question with multiple answers. Uh and so on. So, each task each task has its own specific structure. And the way people have dealt with this previously is like just learn a specific architecture for each task on top of your model. And this this like this defeats this defeats the whole purpose of the GPT work. Like we're trying to do something that's global, general, and a general-purpose model rather than having multiple task-specific architectures. So, instead of using this approach, they opted to use a different approach where we can they convert the the structured inputs into like just tokens. So, they're trying to create a multitask format. And this is similar to like what people have used in future work like T5 and Whisper, where basically you're trying to uh, model different tasks just using tokens and special tokens. These input transformations allows us to use like the same architecture for different tasks. So, you you don't need to do a lot of modification. Uh, and we're going to go into this into details in the next slide. So, for example, let's take textual entailment. This task, it involves reading a pair of sentences and judging the relationship between them. So, the relationship could be one of entailment, contradiction, or neutral. Uh, and a small notice that this task is is still challenging because you you need your model needs to have good understanding and reasoning of the language. Uh, because it can be confusing sometimes. So, you have your premise, and you have your hypothesis, and you're trying to classify or try to predict the relationship as being entailment, contradiction, or neutral. So, the way to do this is just to concatenate the premise and the hypothesis. So, this could be a sentence, and this could be a sentence. You just concatenate them and add a special delimiter token in between them. And obviously, you add your start token at the beginning and the your end token at the end. And just try to train a classifier on top of this. And your classifier should classify this uh this sequence of input input tokens as one of entailment, contradiction, or neutral. So, this has become just a classification task uh by just doing transformation. Uh the second task they cover is semantic similarity, and I think this is very popular nowadays because of retrieval augmented generation generation and and embeddings in general. So, all this cool rag stuff. Uh so, this task is about predicting how semantically similar two sentences are. And semantic similarity just means how close in meaning they are. Do they talk about the same thing? Do they mean Do they mean the same thing? Do they have similar meaning or not? Uh And again, this can be challenging because you can have uh two paragraphs that have very different usage of words, but they convey the same meaning. So, this can be challenging if your model is not smart enough. And one note about this task is there is no inherent ordering of the sentences. So, you can just uh there is no sentence A and sentence B. You just have two sentences. Unlike, for example, in the entailment, you have a very specific order. Like, you have a distinction between the premise and the hypothesis. But here, you have just two random sentences. And the way they approached this is using a a Siamese architecture where we have where we generate two input sequences and pass them through through the transformer and compare between them. So, basically, Siamese architecture is a fancy way of saying that we are using the same model twice or two identical versions of the model. So, you get your sentence, first sentence, and second sentence, and then concatenate them and add your special tokens. And you pass them through the transformer, and you get some vector at the end. And also, you do the same thing, but you reverse the order of the sentences. Uh So, you get the second input sequence, pass it to the transformer, and you you get your vector. So, basically, at the end, you can have two vectors. You just add them to vector addition on top of them, and then you just add a a head on top of the the output vector. Uh so, you just multiply the vector by by some uh layer, a linear layer, for example, that has parameters W, and you get your output. And for example, if you're doing uh like if you just have if you're just interested in knowing whether these sentences are similar or not similar, so you have only two labels, you can train a classifier. If maybe you're interested in having like more of a like a scale of similarity, like from zero to 10, you can train a regressor on top of this. So, that's very cool. Uh we're still using this same architecture, by the way, just the transformer here. We're not modifying it. We're just uh being smart about how to approach this task. Uh you can extend this to do question answering. So, let's say you have a question, and you have four choices. Or let's say you have a document, you have a question about the document, and you have four potential answers to this document, and you want your model to pick one answer of these. So, the way to do this is you take your document or context, and add the question, and then add the the first answer, and then you get your first input sequence. And you pass this through the transformer. And similarly, you get your context or document, and add the question, and add the second answer, and make this into a sequence, and give it to the transformer. And you do this for all the answers. And then you just compare between uh the scores the model gives to each of these potential answers. So, for example, we have a Wikipedia article and we have a question and we have answer A. We concatenate all of these with some special tokens and we give we pass them to the transformer. And to the final linear layer and we get the score. So, score for answer A and score for answer B and score for answer C. I'll just do a softmax to get a proper probability distribution. And that's it. You got your model to do question answering. And you can do this also for other tasks like common sense reasoning. So, any questions about this uh this like these transformations for each task? Uh I think there was a question about modifying the input sequence to both possible orderings, at least when it comes to semantics and that. Um I think someone had a question about that. So, if Sure, uh So, you just said that he doesn't really understand what it means to modify the input sequence to both possible orderings. Yeah, sure. So, if you're doing textual entailment, you have very like you have distinction between the premise and the hypothesis. Like you have a premise and the hypothesis. Uh so, you cannot just mix them up. You have a very specific premise and a very specific hypothesis. And the ordering here matters. Like you should put the premise first and then the special token and then the hypothesis. You cannot put the hypothesis first and then the delimiter and then the premise. So, this is what they mean by the ordering here matters. But for semantic similarity, you just have two two sentences. You can there's no inherent ordering for the semantic similarity task. Uh I hope this answers the question. Uh I think it did. Uh he said, "I see. Thanks." So, I think probably that's about it. We could I don't see any other questions. Oh, he said that does it also combat things like positional biases in transformers by ordering by switching the order of the sentences itself? Any validity to the point he Yeah, yeah, exactly. I think yeah, I think this is like one of the motivations they did this. Like, do you want to say there's no inherent order for this task, so maybe the transformer will just have a bias. So, let's try both ways. Like, let's give it the first sentence and then the second, and let's give it the other way around. Uh to like get over the the positional bias because I think for some models, uh they they pay more attention to the last few tokens in the input, and they disregard the earlier tokens. This This tends to happen sometimes. Yes. I think yeah, I think this this uh the good way to think about this. Awesome. I think uh I don't see any other questions in the chat, so I can just surface them as a one-minute call. So, I think we should be good for now, by the looks of it. Sure. Sure. So, that was like the whole approach. We can now discuss some details about the training and like then OpenAI actually did release info about their training and models. They don't do this now, unfortunately. Uh-huh. But anyways, so the data set they used for training is Book Corpus for training the language model. So, this is in step one, which is unsupervised training. Book Corpus. And back then this was huge. It has 7,000 unique unpublished books from different genres. So, like the variety here helps as well. They have adventure, fantasy, and romance. So, kind of like a very good big diverse data set. And And the main the main advantage of this data set and the reason the reason they chose it is because it has long stretches of text. So, if you have a a book or a novel, you have like a paragraph that's maybe 10 lines or more. And this helps the model to learn how to handle long long-range information and how to handle long context. For example, there's also another data set that's called Word Benchmark, which is also big and diverse, but it doesn't have this long context. It's just a bunch of small sentences, I would say. Uh uh and this way your your model not learn how to handle long context. And uh a side note here is like ELMo is also a very similar seminal work in NLP and it falls under the same domain of like unsupervised pre-training and having good embeddings. So, it's one of the uh the fundamental papers and works in NLP. So, they say their model achieved a very low token level perplexity uh perplexity of 18.4 on this corpus. But, I don't think this is actually low today like I think perplexity of 18 is a bit high. Uh but, I'm not sure. And their model is about uh their data set is about 5 GB in size, so not quite big by today's standard, of course. Uh their their architecture, as we mentioned, they have a transformer model. They use a tokenizer. They use byte pair encoding uh back then, which is also I think what's used now right now. Uh so, this is still not changed from today. They have a surprisingly big vocab size by their times standard like they have 40,000 tokens. I think Llama 2 had also something that's in the same range like 40,000, 50,000 tokens. So, back then this was actually quite big, I would say. And they use the FTFY library to the raw text and then do some standardization using spaCy tokenizer. Uh so, this is good work uh in the tokenization area, I would say. And their model is just uh a typical transformer model like typical by today's standard, just a transformer decoder only transformer with like 12 layers and masked self-attention. And it has a very big size of 117 millions, and this was actually big back then. Although this is trivial now, of course. Uh their embedding they used learned positional embeddings compared to the sign sinusoidal embedding in the original transformer paper. So, this was actually much simpler to implement, and the model just has to learn the the embeddings in training. They have a context size of 512. Again, back then that was very big. And they also used tied input and output token embeddings as I mentioned. Uh for the attention block, they have 12 attention heads. Each head Each head has 64 Uh each head has a dimension of 64 for a total of 768 dimension for the for the entire attention block. And after the attention, you have the MLP layer, also known as position-wise feed-forward network. And the size of the the inner state of this network is 3,072. Uh the size 3,072. And this means that actually this is a expansion and then contraction uh network. So, you go from 768 to 3K, and then you go back from 3K to 768. And all this is happening inside this MLP. And the activation layer they used it is the GELU activation layer. Uh the optimizer the optimizer they used Adam, which was becoming very popular back then. I think Adam was It developed in 2014-2015. So, it was getting a lot of traction back then. And the maximum learning rate is this learning rate, which is also very popular nowadays. So, I think this was used in one of the Lama 2 models. So, this is very familiar. And the warm-up, they do warm-up as well from zero to maximum to the maximum learning rate over 2K step 2K steps, and then they just cosine annealing to they do cosine annealing from the maximum learning rate to zero. So, just learning rate decay. They compute They used one machine that has 8x B600 GPU. Uh this is the same family of GPUs as V100, I think, but I don't have much information about this GPU. They train for 30 days, uh which actually is not bad. That's uh That's not very long, but back then this was very long, I think I think. Their utilization they mention is 0.33, and the total flops is 0.6 petaflop. So, almost one petaflop. And for a reference, I think the TinyGrad machine is trying to give you one one petaflop or I could be wrong. Uh Yeah, I could be wrong about this. So, anyway, the the compute they use is almost one petaflop. Uh they use And this is like the the way they calculated this. And this is actually how the model looks like if you will try to load in the Transformers framework. So, you have like token embedding, position embedding, and then some dropouts, and then you you have your actual Transformer blocks. And this is like the the the the architecture of the language model. So, So, there is no head in here. So, the the block is just attention, and then layer norm, and then MLP, and then layer layer norm. Uh so, there are no know. If there are no questions, we can move on to the second step. Uh I think there was a question from Sean There was a one question from Sean, which is are perplexity numbers comparable across different models? And I think we we discussed just now that the model itself has a perplexity of 18.4. And so in this case, uh would it be okay to compare Like is it Is it a Is it a a number that's a metric that's invariant across different models in this sense? Yeah, so that's a good question. I think perplexity is just a metric. Like you're going to measure perplexity on a certain data set. Like let's say you have this data set that is 1 trillion tokens or 1 billion tokens and you measure perplexity of GPT-1 on this data set and you can measure the perplexity of Llama 2 on this data set. And I think it's you can compare this uh this metric. It's like saying uh Llama 2 has uh human eval of 70 and GPT-4 has a human eval of maybe 90. It's not totally fair, but we can do it, I guess. We can get away with doing it. Yeah, but it might not be 100% fair. Uh yeah, so that's uh that's a good point. Okay, cool. I think that's probably the only question. So I think we can proceed on to supervised fine-tuning. Yeah. Sure. So the second step is supervised fine-tuning and they use these data sets for this task. So for sentence similarity, they use these data sets and for classification, they they used Cola, which I think is popular now as well. Uh I'm not going to go into details about this because uh I don't have much information about these. And the architecture is just used the same backbone as pre-training and you add your head mostly classifier head with a dropout of 0.1. They train for three epochs with a batch size of 32. This is also still uh standard nowadays as well. Uh people usually train to fine-tuning for three epochs with a batch size of maybe 32 or 64. So, yeah, that's that's uh common nowadays as well. The learning rate is 6.25 e to the 5, and also people use very similar learning learning rates nowadays as well. So, this is very familiar. This architecture for fine-tuning is very familiar even today. And they also use uh learning rate decay with a warm-up. And when they use the auxiliary language uh the auxiliary objective, they used a lambda of 0.5. So, the weight of the auxiliary language modeling objective was 0.5. But this is not popular nowadays. Most people don't do this. Uh, any question about this? I think you should Before we go to the benchmarks. Yeah, the chat. There's no questions in the chat, so I think we can just move on to the benchmarks. Sure. So, the benchmark is like they do a lot of sort of performance on many tasks. Like you can see here, almost every single task except this one. And there are ones where they have significant improvement like this uh in QNLI. They have like 6% absolute improvement, and I think let's go back a bit. QNLI. Uh, where is it? In here. Uh I think the QNLI is is more of a uh natural language understanding where it requires reasoning, and this is where they make the most improvement, I think. But overall, like they are they are like doing a very good performance on many tasks. And they are comparing to LSTMs and other models. So, good news. GPT-1 tends to be a good model. Uh yeah, this is for natural language inference. For question answering and common sense, they also make very big gains, as you can see here. So, 76 compared to 86. Uh they compare also to multiple models, and one of them is actually an ensemble of nine models as the denoted here by 9x. And they also Also, the same happens for semantic similarity and classification, although the I would say the the boost in in performance is not as as big as the previous ones. Uh some they are good on some uh some metrics and bad on other metrics. Uh so, this is my favorite part of the paper. They did after the benchmarks, they now have a good model. So, they they try they are trying to understand why their model is good and why their GPT-1 is is suddenly so good. So, they do analysis and try to understand why this is happening. So, the first step is right to they trying to analyze the impact of the transfer learning, the number of layers transferred from the pre-trained model to the fine-tuned model. So, they just take the pre-trained transformer, and they take all the layers and do fine-tuning, and then they take 11 layers, and then they do fine-tuning, and then they take 10 layers and do fine-tuning, and so on. And they compare the performance of these models. And as you can see, the more layers you add, the more performance you get. And if you do zero layers transferred, you're not taking any of the pre-trained weights, you're just starting from scratch. Uh you get this performance, but if you just add one layer, you get a very significant boost. Uh the solid lines here lines here are the dev data sets. The dashed line is the training data set. So, you can see that a very big improvement in performance just by adding one layer. And this is almost like a continuous trend. So, we they observe the obvious fact that transferring more layers improves performance and just only transferring the embedding layers uh, also improves performance significantly. And they mentioned that each layer adds a boost of 9% for for for each layer you add on this task, multi-NLI. And this this analysis actually indicates that each layer in the pre-trained model actually has a purpose and it's it learns something that's useful and it it's very helpful in in the fine-tuned model. So, different layers learn different things. They learn different scales and they are all helpful. Uh, this is a very good finding of this work. So, each layer in the pre-trained model contains useful information on functionality for solving for target tasks like classification. Uh, the second piece of analysis they did is zero-shot evaluation. And this was uh, I would say radical back then like they are trying to evaluate how good the pre-trained model is on on on these tasks without even fine-tuning. Uh, and they want to answer the questions like why is language model pre-training effective? Why does pre-training a language model on Why does pre-training a transformer on language modeling help us when we are doing classification? Uh, they have a hypothesis is that the underlying generative model learns actually multiple tasks in in the pre-training. So, it's not just learning language modeling. Because if if you're if you think about it, if you're learning how to model language properly, you you you you're more likely to have a like a deeper understanding of the the language and just learn natural language. Like if you're for example, if you if you speak only English and French, uh and you you cannot speak let's say Spanish, you won't be able to uh do classification in Spanish. But if you speak English and you speak French, you probably have the understanding to do these tasks. So, this is uh something cool and something to think about a bit. Another thing is like attention is very helpful. Like transformers uh show very big improvement compared to LSTMs. Or that's what they are hypothesizing about and what they are trying to test. So, to test these two hypotheses, they designed a series of heuristics. Uh they basically try to evaluate the pre-trained model on these tasks before doing the supervised fine-tuning. And they have different modifications for each uh data set and each task. For example, for linguistic acceptability for the cola, they they used the examples and they take the average token log probability for each for each token and use this as a score and they have a threshold. And they just determine if the model if the model got this right or wrong. Uh something that's that's very cool is how they handle sentiment analysis. So, you get your let's say Amazon review and you append a token very to the review and restrict the language to generate only two tokens. One of two tokens, positive or negative. And you you see which token has higher score or higher probability and this is the prediction of the model on this uh sentiment or this paragraph. So, just restrict the language model prediction to these two these two tokens and see which one is higher. Uh this is actually pretty cool. It's kind of like uh similar constrained grammar and constrained output. Another example is is question answering. So, for each answer, as you said, concatenate the document, question, and answer in the input. And you average the token log probability of just the answer. And use this as the score for this answer, and do this for multiple answers, and just pick the one with the highest score. Uh And for for Winograd schemas, uh for for this task, you have for let's say you have a sentence that says "The city councilman refused the demonstrators a permit because they advocated violence." So, the goal here is to find this word "they". What does it refer to? Does it refer to the demonstrators or the city councilman? Uh this is what they mean by Winograd schema, and I think it's a very popular task. So, the way they do this is they take this word "they" and replace it with the two possible answers. In this case, councilman and demonstrators, see how uh example A, you use the word councilman, and example B, you use the word demonstrators, and just uh do just average the token probability of of what comes after this word. And then you just pick the one with the higher probability score. Uh so, this is like the after you want to do evaluate the model. Let's see actually the evaluation results. And surprise, surprise, uh the more the longer you train the model and the more data you give the model, the more it it tends to perform the the more it tends to perform on this task. So, the better is your slot performance it has, basically. Uh this is a very cool graph. So, we have different tasks here. And the solid line is the GPT transformer, and you can see with more steps, more training steps, the better performance you have. And they also did a very cool analysis where they trained an LSTM, which is the dashed line, and you can see that for almost every single task that Transformer is better than the LSTM. So, they observe the obvious that more training gives better performance. More More pre-training gives better performance even in zero-shot settings. Uh and this suggests that generative pre-training actually in generative pre-training the model learns a lot of tasks, not just language modeling. And they mentioned something here. They said they observed that the LSTM exhibits higher variance in its zero-shot performance, suggesting that the inclusive bits of the Transformer assists in transfer. But I'm not sure what they mean by high variance in this graph, actually. Uh so, if anyone has any clue about this, please share it with us. Yeah, it's it's better listen to you on them. GPUs are all we need. Exactly. So, the final uh like the final section in the paper is the ablation studies they did. Uh they have three ablation studies. They want to first see the effect of the auxiliary language modeling objective during fine-tuning. Uh so, in this case, we have the the the normal Transformer training with auxiliary language modeling, which is row one, and we have the one without auxiliary language modeling, which is row three. Uh they say that the trend suggests that uh larger data sets benefit from auxiliary training compared to smaller data sets. And this is kind of like actually why probably why people stop doing auxiliary language modeling. It doesn't seem to be quite uh important. Probably. The second ablation study is the of the transformer. So, they use they just train the usual transformer and they compare it with an LSTM. So, row one and row four. Uh and you can see in the the transformer has generally better performance than the LSTM on most tasks. So, they mentioned that they observe a 5.5 5.6 average drop when they use the LSTM. And it only out performs the transformer in on one data set, which is MRCP MRPC. Uh but I'm not sure what this is. The second application study and the I would say the most important one is the effect of pre-training. So, they compare the transformer that has been trained in what the the framework they proposed, the two-step framework. And they they also compare it to a transformer that was directly trained on the supervised task. That is without any pre-training. So, we have row one and row two. You can see that this is where the huge difference shows up. Like 74 to 59. 50 45 compared to 18. 88 compared to 71. So, this is like a very big difference. Uh Yeah, and we observe that no pre-training actually hurts the performance quite a lot on almost all tasks. Uh and this results in 15% decrease. And this is actually the worst performing model out of all these models. Uh yeah. So, I would say that the conclusion from this application study is that pre-training is very very important. I think this is the final section before the future studies do you want to discuss? Any questions so far? I think chat seems good. Do you want to You have one more slide I think you said you future studies. Yeah. Do you want to just finish it up then we can open up the questions if people are Sure. Sure. So, the section I I don't think it was in the paper. I think it was only on the blog post. Uh they discuss what future work could be. And the first approaches surprise, surprise, just a scale up. So, they mentioned that they you know to some improvement in language modeling and they're correlates with downstream tasks. And they're only using very limited hardware, only eight GPUs on a machine on training on a very small data set. So, maybe there is the room for improvement if you scale the model, the training, and the data set. And I think we know the answer to this question. Uh or the answer to this hypothesis. Uh the second approach is uh the second futuristic approach they they want to try is like uh try to see if there is like you can tweak the fine-tuning approach instead of just doing vanilla fine-tuning. Maybe you can use adaptation or any of the cool one of the other fancy ways of doing fine-tuning. And I don't think this this I don't think this is as important as they might have thought about this. Cuz people now right now are doing just simple fine-tuning and it works. So, yeah, not quite as as promising as the first approach, which is just scaling up. And the third one is understanding of why generative pretraining helps. Uh and they've done some ablations and analysis about this. They want to do even more further targeted and research to understand this. So, basically observability and explainability in machine learning. Uh and one very good question they ask is like how much does longer context help like compared to just improved world knowledge when you're doing pretraining? So, for example, this GPT model is able to process a a context because it's a transformer. So, is this the thing that makes it SOTA performance or is it the fact that they train on a bigger data set with a longer training time and it obtains world knowledge. And I think both are both are important, I would say. But that's a very good question. Uh so, yeah. That was That was it in the GPT. They introduce a framework for like achieving SOTA performance by doing two-step approach, doing pre-training and then fine-tuning. Uh Pre-training The goal of pre-training is to It's like to obtain a very good world knowledge and a very good starting point. And you do the pre-training on a diverse corpus with long stretches of text. So, the model acquires actually world knowledge. And then you actually do transfer learning by fine-tuning on like tasks like question answering and so on. And the results that they improve the state-of-the-art performance on nine data sets out of 12. And they used the They successfully have utilized the unsupervised approach to do a transfer learning to discriminative tasks. So, now we have a clear way of how to do uh unsupervised training and semi-supervised training. And the work also highlights that transformers is very is a very good architecture and larger data sets are good. And this is a very important push in the direction of of scaling up and pre-training. And this is actually what people are going to do for the next uh 6 years, from 2018 to 2024. Yeah, so very good paper, very good work. And with that we've come to the end of this paper. So, if you have any questions, I think we can take questions now.