Implementing GPT Model from Scratch: Transformer Blocks & Text Generation | Chapter 4

A GPT model is built from a few core parts, and getting those parts right matters because they determine how text is represented, processed, and generated. The focus here is the full model pipeline. The architecture itself, the training stability tricks that make deep networks behave, and the size calculations that tell us how large these models really are. The goal is to assemble a language model that can learn patterns in text and produce coherent continuations. That requires the same transformer style machinery used in modern GPT systems, but built piece by piece from scratch. Layer normalization keeps activations in a healthier range, so gradients do not become too erratic as signals move through the network. That stability becomes especially important once the model gets deep. Shortcut connections let information bypass a layer and travel more directly through the network. They help optimization by making it easier for deep models to preserve useful signals instead of losing them as the stack grows. >> Once the building blocks are in place, they can be repeated and scaled into larger transformer stacks. That is how one design turns into multiple GPT variants with different depths and capacities. A model's parameter count gives a concrete measure of complexity, and its storage footprint tells us what it costs to keep and run. Those numbers are useful before training starts because they shape compute and memory planning. Multi-head attention gave us the main mechanism for mixing context, but a working language model needs more than that. The remaining components fill in architecture around attention so the whole system can be trained as a GPT-like model and then used for text generation. The figure organizes the work into three stages, and the part to focus on now is the third step in stage one, building the architecture itself. That is the bridge from individual mechanisms to a complete model. >> The architecture stage comes before training and evaluation because the model has to exist in code before it can learn from data. Keeping that order clear makes the rest of the chapter much easier to follow. A top-down view shows how the pieces fit together before any individual layer is examined. That overview gives the structure we need and then each component can be unpacked one by one. An LLM is built from a small set of repeating parts and the architecture becomes manageable once those parts are identified. GPT is a good example because the same core block is stacked many times to produce text one token at a time. Generative pre-trained transformers predict the next token from the tokens already seen. The model may be large, but its structure is not as complicated as it first appears because many components are reused throughout the network. >> Focus on the main path from embeddings into the repeated transformer blocks and then into the output layer. That top-level flow is the backbone of the whole model. So, the details make more sense once that path is clear. The key pieces are the embedding layers, the stack of transformer blocks, and a masked attention inside each block. Those are the parts that do most of the work, and the rest is there to move data cleanly through the model. Those earlier pieces give us the inputs and the attention mechanism. So, the remaining job is to assemble them into the full GPT core. The transformer blocks are the central structure, and training them is what makes the model useful for text generation. The jump here is mostly about size, not about changing the architecture itself. Parameter count means trainable weights, and 124 million gives a realistic small GPT-2 configuration that is still workable on modest hardware. Parameters are the numbers the optimizer adjust during training. They store the model's learned behavior. So, when a loss goes down, it means those weights are being tuned to match the data better. A matrix layer contributes one parameter per entry. So, counting parameters is just counting all the values in the weight tensor. A square layer like this grows fast. Multiplying the two dimensions gives the total number of trainable values in that layer. The comparison matters because the two models share the same basic design, but the scale is very different. That makes GPT-2 the practical target for understanding the architecture in GPT-3, the large-scale version of the same idea. GPT-3 keeps the same model shape, but the parameter count and data size are far larger, which changes the computational cost dramatically. GPT-2 remains the better learning target because it is small enough to run and inspect without a cluster. A configuration dictionary gives a compact way to define the model before any code is written around it. Each setting will control a structural choice. So, the values here define the size and behavior of the GPT instance. Each entry fixes one structural choice in the model. Vocabulary size controls the output space. Context length sets how many tokens can be processed together. Embedding size sets the vector width, and the remaining values define the attention and block stack. These names are deliberately compact, so the code stays easy to scan. Once the dictionary is passed into the model, each key becomes a direct handle on a major architectural choice. That number matches the tokenizer vocabulary. Every output position ultimately chooses among those 50,257 token options. So, the embedding and output layers need to agree on that size. This is the longest token sequence the model can handle at once. Positional embeddings depend on it because the model needs a separate position for each step in the context window. Each token gets mapped into a 768-dimensional space, which is the working width used throughout the network. That size is what makes the hidden representations rich enough for attention and transformation layers to operate on. Multiple heads let the model look at different relationships in parallel. Each head focuses on a different learned subspace, so the attention mechanism can capture several patterns at once. This controls depth. More blocks mean the same basic transformer unit is repeated more times, which gives the model more stages for refining representations. >> Dropout randomly removes a fraction of hidden units during training, which helps keep the model from fitting the training set too tightly. A rate of 0.1 means 10% of those activations are dropped each time dropout is applied. The query, key, and value projections can include bias terms, but modern GPT-style models often leave them out. That keeps the attention layers simpler, and this choice will matter again when pre-trained GPT-2 weights are loaded later. A placeholder model is useful because it shows the whole structure before the missing pieces are filled in. That way, the architecture is visible as a complete pipeline even though some internal modules are still temporary stand-ins. The figure orders the work from the outer scaffold down to the internal building blocks. Start by tracking the backbone because that makes it easier to place the transformer blocks and the normalization layer when they appear later. >> The coding sequence follows the architecture from the outside in. That approach keeps the model organized because each smaller piece lands in a slot that already has a clear purpose. The first step is a shell that wires everything together before the real internals are built. Once that shell exists, the remaining components can be developed and dropped in a place one by one. This class is the scaffold for the model. It contains the same overall flow as GPT, but the heavy machinery inside the block and normalization pieces is still simplified for now. The model starts with token and positional embeddings, adds them together, applies dropout, then sends the result through a stack of transformer blocks. After that, a final normalization and a linear head map hidden states to vocabulary-sized logits, while the dummy block and dummy normalization simply preserve the interface for now. Those numbered notes mark the pieces that will later be replaced by real implementations. For the moment, the transformer block and normalization layer act as pass-through objects, which keeps the class runnable while preserving the final architecture. This class combines the major GPT components in one PyTorch module: embeddings, repeated blocks, normalization, and the output projection. The configuration dictionary supplies the sizes, so the same code can describe a small model or a larger one just by changing the settings. The forward path is the data flow through the model. Input indices become token and position vectors. Those vectors pass through the block stack, and the final linear layer turns the hidden states into vocabulary scores. The code is already executable because the missing pieces still have valid class shells. That makes it a useful intermediate stage. The full structure is there, even though the real transformer block and layer normalization are still to come. Before the model can produce any scores, the text has to be turned into token IDs and then into vectors. The next example shows the full path from raw text to model output at a high level. Track the direction of the data. Text becomes tokens, tokens become embeddings, and embeddings enter the GPT model. The important point is that the embedded token dimension lines up with the model's internal working size. The input embedding step now happens inside the GPT model rather than outside it. The output side uses the same dimensional width because each token position ends up represented by context vector in that shared space. A batch makes it easy to see that the model handles multiple sequences at once. Each text is encoded separately, and then the token ID tensors are stacked into one batch dimension for processing. The tokenizer converts each short sentence into token IDs, and stacking them produces a two-row tensor for the batch. That batch shape is what the model expects, one row per sample with token positions laid out across the sequence. The printed IDs let us verify that each sentence has become a sequence of integers. That matters because the embedding layer operates on token indices, not on raw words. Both rows have four token IDs, one row per text sample. The shared first token shows that the two sentences begin with the same tokenization pattern, while the later positions differ according to the words that follow. That row order is important because the batch dimension keeps the two samples separate while the model processes them in parallel. Each row maps back to one input sentence. Now, the configuration and the batch come together. Initializing the model with the GPT settings creates the full tensor shape expected for the forward pass, and feeding the batch through it produces vocabulary scores for every token position. Setting the seed makes the output reproducible. Then the batch is passed through the model to produce logits. The printed shape tells us exactly how many samples, token positions, and vocabulary scores are being produced. Logits are the raw scores before any softmax turns them into probabilities. They tell us how strongly the model favors each vocabulary token at each sequence position. The shape confirms the full output structure. Two sequences, four token positions each, and one score for every vocabulary token. That is exactly what the language model needs because each position can now be turned into a next token prediction. Each token position gets a 50,257-dimensional score vector, one value per vocabulary item. The batch dimension keeps the two examples separate, and the sequence dimension keeps the four token positions separate. That is why the last dimension matches the tokenizer vocabulary. Those scores can later be converted back into token IDs, and token IDs can be decoded into words or sub-word pieces. The architecture is clear now. We have the backbone, the data flow, and the output shape. The next step is to fill in the real layer normalization and transformer block. So, the placeholder model becomes a true GPT implementation. Layer normalization helps keep a network's internal value stable, which makes deep models easier to train and easier to tune. When gradients shrink too much or grow too large, weight updates become unreliable. That instability slows learning and can stop the model from finding good parameters. A little background helps, but the core idea can be followed without digging into the full calculus. The main goal is to understand what normalization changes in the activations. That rescaling keeps the values centered and on a consistent scale, which makes optimization smoother. In transformer blocks, this step is placed around attention and before the output layer, so the model stays numerically well-behaved. Focus on the two steps in the diagram. First, the activations are centered by subtracting the mean, then they're rescaled by the spread of the values. The result is a normalized vector for each input example. The six outputs are treated as one set, and that set is normalized independently for each example. That is the key difference from methods that mix statistics across a batch. A compact example makes the mechanics easier to see. We feed two input rows through a linear layer and then inspect the raw activations before normalization. The random seed makes the numbers repeatable, and the batch contains two examples with five features each. A linear projection expands each example to six values, and ReLU removes any negative responses before the outputs are printed. That shape matters because each row is treated separately. Layer normalization will later work across the six outputs within each row, not across the two rows. Each row now represents one transformed example. Before normalizing, it helps to inspect whether the values in a row are centered and how spread out they are. The zeros come from ReLU, which clips negative values to zero. That gives us a simple, positive-valued activation pattern to normalize next. A linear layer builds a weighted combination of the inputs, and ReLU adds the non-linearity by suppressing negatives. This is a common pattern, even though transformer models later use a different activation. Those two statistics tell us where the row is centered and how wide its values are. Layer normalization uses exactly that information to standardize each example. The last dimension is the set of features inside each row. So, the mean and variance are computed across those six values. Keeping the dimension makes the result line up cleanly with the original tensor shape. These values show the center and spread for each example separately. The two rows do not share statistics, which is exactly what layer normalization wants. Each input row has its own mean and variance. The numbers are not the same, and that is the point. Normalization adapts to the local distribution of each example. The tensor keeps the row structure, so each statistic is paired with its own example. That makes it easy to subtract the correct mean from the correct row. Without that option, the reduced result would lose a dimension and stop matching the original shape. Keeping the dimension avoids awkward reshaping when the statistic is used in later arithmetic. For 2D tensor, the last axis runs across the features in each row. That same rule also scales the transformer tensors, where the last dimension is the embedding size. Watch the direction of the reduction. One axis aggregates vertically across rows, the other horizontally across columns. For layer norm, the horizontal direction is the one that matters because each example is normalized across its features. The figure separates column-wise reduction from row-wise reduction. That distinction becomes important once the input is three-dimensional because the last axis still marks the feature vector we want to normalize. That formula shifts each row to be centered at zero and then scales it so the spread becomes one. It is the core transformation behind layer normalization. The subtraction centers each row and the square root of the variance gives the scaling factor. Recomputing the statistics after normalization checks whether the rows now sit near zero mean with unit spread. Now the values can be positive or negative because centering moved the distribution around zero. The scaling step then makes the row-to-row spread comparable. The means are essentially zero and the variances are exactly one up to numerical precision. Small residuals like 5.96 e 05 e 08 come from floating-point arithmetic, not from the method itself. Computers store real numbers approximately, so perfect zeros can become tiny residual values. When a residual is that small, it is just round-off error, not a problem with the algorithm. Turning off scientific notation makes the output easier to read when values are close to zero. That is useful for checking whether the normalization behaves the way we expect. This changes only the display format, not the numbers themselves. It is a practical trick for inspecting tensors when a default formatting hides small values. With scientific notation off, the zero means are easier to read directly. The variances still sit at one, so the normalization step has done its job. The display now makes the centered values obvious. That confirms the row-wise normalization in a clean, readable form. Writing the logic as a module makes it reusable inside a larger model. The next step is to package the mean, variance, scaling, and shifting into one class. The class turns the manual normalization steps into a layer that can be dropped into a network. Its job is still the same. Normalize each embedding vector and let the model learn the best scale and offset. The forward pass computes statistics across the last dimension, then normalizes with a small epsilon to avoid division by zero. After that, scale and shift let the model learn a better representation than pure standardization alone. The last dimension is the feature or embedding axis. So, each token or example is normalized independently along its own representation. The trainable scale and shift keep the layer flexible instead of forcing every output to have exactly the same final form. There is one implementation detail worth separating from the main idea. The variance formula can divide by n or by n minus one. For this layer, the code uses the simpler n-based form because that matches the behavior needed for GPT-2 compatibility. Setting unbiased equals false skips Bessel's correction and uses the larger divisor directly. For wide embedding vectors, the difference is tiny, but matching the original implementation matters when pretrained weights must load correctly later. Now the custom layer replaces the manual calculations. The same normalization is happening, but it is wrapped in a reusable module that fits into the model pipeline. The module is created for six features, then applied to the earlier activations. Recomputing the statistics verifies that the learned version still normalizes each row properly. The results stay very close to the target values with tiny differences coming from epsilon and floating point arithmetic. That is exactly what we want from a stable normalization layer. Those values are close to the ideal targets, but not perfectly exact because the epsilon term slightly changes the denominator. The important point is that the outputs remain tightly centered and scaled. Those pieces form part of the base architecture. The remaining component to complete the feedforward path is the Jello activation, which replaces the simpler ReLU used earlier. Track the blocks that are already in place. The core transformer backbone and a normalization layer. The next missing piece is the activation and feedforward subnetwork that sits inside each transformer block. The diagram marks the progress so far and points toward the remaining components. After normalization, the architecture still needs its activation function and feed-forward network to become complete. The comparison matters because these two methods normalize along different axis. That difference changes how they behave when batch sizes are small, variable, or unavailable. Batch normalization depends on statistics gathered from multiple examples at once, while layer normalization works on each example by itself. That makes layer norm much easier to use when batch size is limited or when the model runs one example at a time. This part builds the feed-forward sub-module used inside a transformer block and shows why GELU is the activation that fits it well. A transformer block includes a small feed-forward network after the attention step, and GELU is the non-linearity that gives that module its shape. The activation matters because it controls how the model mixes and transforms features between the linear layers. >> PyTorch basics are covered in the appendix. So, the focus here stays on the activation and the module structure rather than the library mechanics. ReLU is common because it is simple, but large language models often use smoother choices such as GELU and Swish-GELU. Those alternatives give the network a richer way to handle negative and positive values during training. Smoother activations change gradually instead of cutting off sharply at zero, which often makes optimization easier. GELU and Swish-GELU keep more information flowing through the network than a hard threshold does. The exact GELU multiplies the input by the Gaussian cumulative distribution. So, it acts like a soft probability shape gate. In practice, a cheaper approximation is usually used because it is easier to compute and closely matches the exact curve. Track the exact definition on one side and the approximation on the other. The important idea is that the approximation follows the same smooth shape closely enough for training. A PyTorch module packages the activation into a reusable component, so it can drop straight into the larger network later. The forward method carries the whole formula, while the class wrapper makes it fit the neural network API. This listing turns the approximation into code, and the forward pass is the part that matters most. The class inherits from nn.Module, so it behaves like any other layer in PyTorch. The forward method implements the common tan base approximation. The cubic term and square root constant shape the curve, and the final multiplication by X keeps the output close to a soft gate rather than a hard cut-off. A plot makes the difference between the two activations much easier to read. One curve is smooth, and the other changes direction abruptly at zero. >> The sample points come from torch.out_linspace, then each activation is evaluated on the same input range, so the curves can be compared fairly. The loop draws two subplots, one for GELU and one for ReLU with matching axes so the shapes line up clearly. Those 100 points give a smooth enough curve to see the shape of each activation without cluttering the graph. ReLU passes positive values through unchanged and clamps negative values to zero, so its graph has a hard kink at the origin. GELU stays smooth and still keeps a small response for many negative inputs, which means gradients can continue to flow more gently. Focus on the left curve being smooth and the right curve breaking at zero. That contrast explains why GELU often behaves better during optimization in deep models. The axes matter here. The horizontal axis is the input and the vertical axis is the activation output. Reading the graph that way makes the different shapes easy to compare. That smoother shape helps during training because parameter updates can adjust more gradually. Since GELU does not shut off all negative values, neurons with negative input can still contribute a little to learning instead of becoming completely silent. The activation now becomes part of a larger feedforward block, where it sits between two linear layers and give the module its non-linear behavior. This listing combines the layers into the compact module that transformer blocks reuse repeatedly. The first linear layer expands the embedding size by a factor of four. GELU adds the non-linearity, and the second linear layer compresses the representation back to the original size. Wrapping those steps in nn.Sequential keeps the forward pass short because the data simply flows through the stack in order. >> The input and output sizes stay the same, but the network briefly works in a larger hidden space. That hidden expansion gives the model room to combine features before projecting them back down. Follow the width change through the block. The representation gets wider after the first linear layer passes through GELU and then narrows again at the end. The point is not to change the external shape, but to let the model compute a richer internal transformation. The diagram marks the key constraint. Batch size and token count can vary, but the embedding size is fixed when the weights are initialized. That is why the module can process different input batches while still expecting the same vector width per token. A GPT 124M style model uses 768 numbers for each token embedding. So, the feed forward block is built around that width. The example input has two batches and three tokens per batch, which lets the shape behavior stay visible without changing the embedding size. >> The module is initialized from the GPT configuration. Then a tensor with shape 2 by 3 by 768 is passed through it. Because the feed forward network returns the same embedding with it received, the printed shape stays aligned with the input structure. That first dimension is the batch size. So the network processes two samples at once while keeping the token and embedding dimensions intact. The module changes the internal representation, but not the outer tensor shape. That makes it easy to place inside a transformer block because later layers can expect the same token embedding size. The shape matches the input exactly, which confirms that the feed forward block preserves the batch size, token count, and embedding width at its boundaries. That expand activate contract pattern is the heart of the feed forward The temporary increase in width gives the model more room to mix features before it compresses the result back into the token embedding space. Trace the arrow from the 768-dimensional input into the wider hidden layer, then back down again. The diagram is showing a temporary detour into a larger feature space, not a permanent change in representation size. The numbers on the figure make the pattern concrete. The first layer multiplies the width by four, and the second layer brings it back to 768. That expansion is where the module gains expressive power. Keeping the outer dimensions unchanged makes repeated transformer layers much easier to stack. Each block can pass its output straight into the next one without extra reshaping or adapter layers. With the embedding layer, attention, and feed-forward block in place, the main pieces of the model are ready. The next ingredient is the shortcut connection, which helps very deep networks train more reliably. Use the check marks to separate the pieces already covered from the remaining ones. The architecture is nearly complete, and the last major concept to add is the connection that lets signals skip across layers. The check marks mark the components already in hand, and the empty spots point to what still needs to be added. That sets up the move into shortcut connections and their role in training deep models. Shortcut connections give deep networks a direct path for information and gradients. The idea becomes important when ordinary back propagation starts to weaken across many layers. A shortcut connection lets the input of a layer skip ahead and combine with a later output. That extra path helps the signal travel through deep models, which reduces the chance that early layers stop learning effectively. >> Focus on the difference between the two paths through the network. On the right, the direct connection gives the gradient a shorter route. So, the early layers receive a stronger training signal. The comparison highlights a simple but important change. Instead of forcing the gradient through every layer, part of it can move around them. That makes optimization much easier in very deep models. During backpropagation, the gradient follows the same structure as the forward graph. When a layer output is added back into a later layer, the backward signal can pass through that addition without fading as quickly. The implementation keeps the network structure simple and makes the shortcut conditional. The key idea is to add the current input to the layer output only when their shapes match because the two tensors must line up to be combined. This model is built to isolate the effect of skip connections. It uses the same layer pattern throughout, so the difference between the two runs comes from the shortcut path itself. The layers are stored in a module list, so the network can step through them one by one. In the forward pass, each layer produces an output, and when the shortcut is enabled and the shapes agree, that output is added back to the incoming tensor instead of replacing it. Those three comments mark the whole mechanism. First, the model defines a chain of layers, then it computes one layer output at a time, and finally it decides whether the residual addition is valid for the current pair of tensors. Each block in the network is a linear transform followed by gelu, which gives the model non-linearity at every stage. The optional shortcut does not change the layer itself. It changes how the layer output is combined with the running activation. To see the effect clearly, the first run keeps the plain feed forward path only. The layer sizes are chosen so the first four layers preserve a three-dimensional activation, and the last layer reduces that to a single value. The input has three features, and a random seed fixes the initial weights, so the comparison is reproducible. With shortcut connections turned off, this model gives us the baseline gradient pattern of a deep network on its own. Fixing the seed matters because the initial weights affect the exact gradient values. That way, the two experiments differ only in whether the shortcut path is present. The next step is to measure how strongly each layer receives the training signal. A small helper function will run a forward pass, compute a loss, and then trigger back propagation. The forward pass produces a prediction. The loss compares that prediction with the target value zero, and then backward propagation fills in the gradients. Printing the mean absolute gradient for each weight matrix makes it easy to compare layers, even though each matrix contains many values. These three steps are the standard training loop at a small scale. First comes the prediction, then the error measurement, and finally the gradient computation that tells each weight how to change. Each weight matrix has its own gradient array, so the raw values are not easy to compare directly. Taking the mean absolute value gives one number per layer, which makes the gradient strength across depth much easier to read. That method saves you from writing the derivative rules by hand. For deep networks, automatic differentiation is what makes experiments like this practical. A deeper treatment of gradients helps if the training mechanics still feel abstract. The important point here is that the gradient is the learning signal flowing backward through the model. Now the baseline model is ready for measurement. The result will show whether a plain deep stack keeps the gradient strength stable from one layer to the next. Running this function sends the sample input through the network, computes the loss, and prints the gradient size for every weight layer. The pattern in the output tells us how the signal changes as it travels backward. >> The printed values are the layer-by-layer gradient strengths from the baseline network. The numbers will tell us whether the earliest layers are receiving a healthy signal or only a weak trace of it. The later layers have noticeably larger gradients, while the early layers are much smaller. That gap is the classic sign that the backward signal is fading as it moves through depth. A vanishing gradient means the first layers barely change during training because the error signal becomes tiny by the time it reaches them. When that happens, deep models struggle to learn useful early representations. The same test on a residual version will show whether the shortcut path keeps the backward signal from collapsing. The setup stays the same, so any change comes from the added bypass connection. The seed is reset, so the new model starts from the same initial weights as before. With use underscore shortcut set to true, the only structural difference is the extra addition across matching layers. These values come from the same gradient measurement, but now the model includes the shortcut path. The comparison with the earlier output is what matters. The gradients are now much larger in the early layers, and they stay in a healthier range across the network. The shortcut has made it easier for the backward signal to reach the beginning of the model. The last layer still has the strongest gradient, but the earlier layers no longer shrink to near nothing. That stability is exactly what deep networks need when they have to train many stack transformations. That preserved flow is why shortcut connections became a standard part of very deep architectures. They let large models train reliably even when the stack of layers is long. Those pieces now come together inside one transformer block. The shortcut connection will sit alongside normalization, activation, and a feed-forward path to form the basic unit of GPT-style models. A transformer block joins attention and feedforward layers into one repeatable unit. That combination is what lets GPT-style models build context while keeping the sequence shape unchanged. A transformer block bundles the pieces we have already built into a single layer that can be stacked many times. The important idea is that each block refines token representations without changing their dimensionality, which makes it easy to chain block after block in a deep language model. Focus on the input and output width. Each token enters as a 768 dimensional vector and leaves with the same size. The internal operations change the content of those vectors, but they keep the shape compatible with the rest of the network. The key point in this figure is the flow through the block. Token vectors enter, pass through attention and feedforward processing, then come out with the same width. That shape preservation is what allows the block to be used repeatedly in GPT. Self-attention compares tokens with one another. So, is the part that builds relationships across the sequence. The feed-forward network then works on each position independently, giving each token its own non-linear transformation after the context has been gathered. That division of labor matters. Attention moves information between positions, while the feed-forward layer increases the expressive power at each position. Together, they let the model both communicate across the sequence and refine each token's representation locally. The implementation follows the same structure as the diagram. Attention first, then a feed-forward network with normalization and residual connections around both parts. That arrangement is what makes the block stable to train and easy to stack. >> The class definition packages the block into a reusable pytorch module. Its structure will look familiar now because it mirrors the model design we just described on the board. The constructor builds two main sub layers, multi-head attention and feed forward processing, both sized to preserve the embedding width. In a forward pass, the input is saved as a shortcut, normalized, transformed, and then added back. The same pattern repeats around the feed forward layer. Those residual paths are the core of the block because they let the new information update the input instead of replacing it. Those four marks point to the residual connections. Each sub layer learns a correction and the original signal is added back so the model keeps the path for both information and gradients. The configuration dictionary supplies the embedding size, number of heads, context length, dropout rate, and whether the attention uses a bias. That makes the same block definition reusable across different GPT settings without changing the code itself. >> Layer normalization comes before attention and before the feed forward network, which is the pre-layer norm pattern. Compare with post-layer norm, this usually gives smoother optimization and more reliable gradient flow in deep stacks. The shortcut path makes the block easier to train because gradients can travel through the identity connection even when a learn transformation is difficult. That is one reason deep transformer models stay stable as you add many layers. A quick test with random input is enough to verify that the module is wired correctly. If the block preserves shape, the output should keep the same batch size, sequence length, and embedding dimension as the input. The random tensor has two sequences, four tokens each, and 768 features per token. Passing it through the block checks the whole pipeline end-to-end, and the print statements confirm whether the dimensions survive the attention, normalization, dropout, and feed forward steps unchanged. >> That shape matches the standard transformer layout. Batch size first, then token positions, then embedding features. Using this format keeps the block compatible with the rest of the model stack. The printout will verify the shape rule directly. Matching input and output dimensions is exactly what we want from transformer block. The two shapes are identical. So, the block changes the representation without changing the tensor size. That is the signature of a well-formed transformer block. Same sequence structure, richer token vectors. The important distinction is between size and meaning. The tensor keeps the same dimensions, but each output vector now carries information gathered from attention and refined by the feed-forward network. That shape preservation is essential for sequence models because every layer can hand its output to the next without any reshaping. Even though the dimensions stay fixed, the representation becomes more contextual at every step. With this block implemented, the remaining GPT architecture becomes a matter of stacking the pieces we already have. The block holds normalization, feed forward processing, GELU, and residual paths together as the main repeating unit. Focus on the check components in the diagram. Those are the pieces already completed. The remaining structure is just the larger GPT model assembled from these parts. The diagram closes the loop by showing how the completed modules fit into the full architecture. The transformer block now sits at the center of that design, ready to be repeated through the model. The focus here is turning the GPT architecture from a diagram into working code, and then checking that its shapes, parameter count, and memory use all match the model design. >> That earlier model gave us the right interface, but its internals were still fake. The point of those placeholders was to postpone the implementation details until the full structure of a network was clear. Now the placeholders get swapped out for the real components we already built. So the model becomes a complete GPT-2 style network that also sets up the two later steps, training our own version and loading pre-trained OpenAI weights. A GPT model is mostly repetition of the same core block. The 124 million parameter GPT-2 uses that block 12 times, while the largest GPT-2 repeats it 48 times, and NNs for layers setting controls that depth directly. Track the flow from tokenized text to embeddings, then add position information, then send the result through the stack transformer blocks. The final normalization and output layer come after that repeated core computation. >> The structure is layered in a very specific order. First, the model turns token IDs into vectors. Then it injects position, and only after that does it apply the repeated attention and MLP blocks that do the main sequence processing. After the last block, normalization makes the activations well-behaved before the linear head maps them into vocabulary space. That large final dimension is what gives one score for every possible next token. The next step is straightforward. Translate the diagram into a PyTorch module and keep the data flow in the same order as the picture. The implementation stays compact because the repeated transformer logic already lives inside its own class. That lets the top-level model focus on embeddings, stacking, normalization, and the output head. The constructor builds the two embedding tables, then creates a stack of transformer blocks whose length comes from the configuration. In the forward pass, token IDs are turned into embeddings, position vectors are added, dropout is applied, and the tensor flows through the blocks, the final norm, and the output head to produce logits for each token position. That device argument keeps the position indices on the same device as the input batch. Without that, a CPU tensor and a GPU tensor could end up mixed and a forward pass would fail. Most of the complexity is hidden inside the transformer block class, so the outer model stays easy to read. The top-level module mostly assembles pieces rather than re-implementing the attention and feed-forward details. The embedding layers do two different jobs. One converts discrete token indices into dense vectors, and the other tells the model where each token sits in the sequence, so order is preserved. The block stack does the repeated sequence modeling. The layer norm stabilizes the activations, and the linear head turns hidden states into the over the full vocabulary. That final projection is what makes next token prediction possible. The forward method is just the inference path written out in code. It takes a batch of token IDs, builds position-aware representations, sends them through the transformer stack, and returns unnormalized scores for each possible next token. At this point, the model can be instantiated from the configuration dictionary and tested on a real batch. That gives a quick check that the architecture matches the expected GPT-2 small setup before counting parameters. The seed makes the test run reproducible. Then the model is built from the configuration and applied to the batch. Printing both the input and the output confirms the shape transformation all the way through the network. The printed batch shows the token IDs that went in, and the tensor that comes out should have one score vector for every token position in every sequence. The two rows correspond to two text examples, and each row has four token IDs. The output shape 2 4 50,257 means the model produced a vocabulary size score vector at every position for both sequences. Those comments identify which row belongs to which text sample. The important part is that the model treats each sequence separately, but still produces the same vocabulary dimension at every position. That shape matches the setup exactly. Two inputs for positions each, and one score for every token in the tokenizer vocabulary. Those scores are still raw logits, so they need a later step before they become actual text. Now the question shifts from shape to size. Counting elements in every parameter tensor tells us how large the model is in a way that is independent of the batch input. This line walks through every parameter tensor in the model, counts its elements, and adds them up. The print statement turns that total into a readable number with commas. That count gives the raw parameter total before any weight sharing adjustment. The next step is to compare it with the published GPT-2 size and explain why the numbers differ. This is the full count when the output head is treated as its own trainable layer. It is larger than the published GPT-2 small figure because the original architecture ties some of those weights together. The mismatch comes from weight tying. The model we coded keeps the token embedding matrix and the output matrix separate. So, it counts both sets of parameters while the original GPT-2 shares them. With weight tying, the output layer does not need its own full copy of the vocabulary matrix because the embedding table and the prediction head have the same shape. One matrix can serve both roles. Printing the two shapes makes the symmetry obvious. Both layers use the same vocabulary by embedding size layout, which is exactly why tying the weights is possible. The matching shapes are the key fact. Once the dimensions line up, the output head can reuse the embedding weights instead of learning a separate matrix. The two tensors are identical in size, one row per vocabulary item, and one column per embedding dimension. That is the structure needed for weight tying. Once the shared output matrix is removed from the count, the parameter total drops to the published GPT-2 small size. The large vocabulary is what makes that matrix such a big part of the model. This computation subtracts the output head parameters from the full total. In other words, it simulates the parameter savings that weight tying gives in the original GPT-2 design. That adjusted number is the one to compare with GPT-2 small. It reflects the shared embedding and output matrix rather than two separate copies. The adjusted count lands at about 124 million, which matches the original GPT 2 small scale. That confirms the architecture is consistent with the published model once tied weights are accounted for. That agreement is the check we wanted. The code implements the right architecture and the parameter count now lines up with the reference model size. Using separate layers costs more memory, but it could be a reasonable design choice during implementation and experimentation. The tied version matters again when pre-trained GPT 2 weights are loaded later because that is how the original model was built. The useful comparison now is between the two main parts inside each transformer block, the feed-forward network and the attention module. Their parameter counts help explain where most of the model capacity lies. Both modules contribute a lot, but not in the same way. Working out their parameter totals makes it clear which part scales with width and which part scales with the attention projections. Parameter count is only part of the story. Memory matters, too. If every parameter is stored as a 32-bit float, the model size in bytes becomes easy to estimate. The first line converts parameters to bytes by assuming 4 bytes per float 32 value. The second line turns bytes into megabytes. So, the final printout gives a practical storage estimate. Those comments make the assumption explicit. If the precision changed, the memory estimate would change, too. So, the byte size depends directly on the numeric format. The calculation gives a concrete sense of scale. Even a model with a few hundred million parameters already occupies hundreds of megabytes in memory. That size estimate is the storage cost of the parameter tensors alone under float 32 precision. Training needs even more memory once gradients and optimizer states are included. This estimate shows why model size matters so much in practice. A seemingly modest language model already needs a substantial amount of memory before any training overhead is added. At this point the architecture is working end-to-end and the output has the exact shape needed for token prediction. The remaining task is to turn those logits into actual text tokens. The same model class can scale to much larger versions just by changing the configuration. That makes the configuration file the main control knob for depth, width, and attention heads. The exercise is to reuse the same architecture for much larger GPT-2 variants by adjusting embedding size, number of blocks, and number of heads. Counting the parameters for each one shows how quickly model size grows as those settings increase. >> Generating text means turning model outputs back into readable language, one token at a time. The main job here is to connect logits, token IDs, and decoded text so the generation loop makes sense. A language model does not produce finished sentences in one step. It produces scores over the vocabulary at each position. The key idea is to choose the next token from those scores, appended to the context, and repeat. Track the growing context across the iterations. The input starts with a short prompt, then each pass adds one more token. The important relationship is that the new token becomes part of the next input, so the sentence can keep extending. A generative model builds language step-by-step, not by predicting the whole sentence at once. Each round uses the updated prompt, which is why the output can stay consistent with what came before it. >> The model gives a score vector for every token position in a batch. To generate the next word, we only need the final position because that is the model's prediction after the full current context. That sequence turns raw tensor output into text. First, the scores become probabilities, then one token ID is selected, then that ID is fed back into the model for the next round. Focus on the flow from text to token IDs through the GPT model and back to text. The model never handles human words directly. It works with numbers the whole time. A single generation step has a clear pipeline. Encode the prompt, run the model, choose the next token, and decode it again. That one step is the unit repeated over and over in text generation. The model first produces logits, which are just unnormalized scores. Softmax turns them into a probability distribution, and argmax picks the most likely token, which is then decoded and appended to the prompt. >> That loop keeps running until we have generated as many tokens as we asked for. In code, the whole job is just a controlled repetition of the same next token step. The function below packages the whole generation loop into one reusable routine. The important parts of the context crop, the model call, and the token selection step. The loop runs once for each new token, and each pass starts by trimming the context to the model's maximum window. The last time step is kept, softmax converts scores to probabilities, argmax chooses the most likely token, and concatenation appends it to the running sequence. Each comment marks one stage of the same pipeline. The shapes matter because they show how a full sequence is reduced to one predicted token, then expanded again when that token is appended. This is the simplest form of generation, always choose the highest probability token. It is easy to implement and easy to reason about, which makes it a good starting point before introducing sampling. Softmax changes the scale into probabilities, but it does not change which entry is largest. That means argmax could be applied directly to the logits, and the result would be the same for greedy decoding. A model that always picks the top token can sound repetitive. Sampling changes the behavior by letting less likely tokens appear sometimes, which gives the text more variety. The sequence grows token by token, and each new prediction depends on the full prompt built so far. That is why the model can gradually form a sentence instead of guessing the whole response at once. Follow the chain from the initial prompt to the final sentence, and compare the token IDs at each step. The repeated append operation is what turns a short input into a longer output. Six passes through the loop are enough to turn the starting context into the full sentence. The figure makes it clear that generation is just repeated prediction with a growing input sequence. The first step is to convert the starting words into token IDs because the model only accepts numbers. That encoded prompt becomes the initial context for generation. The tokenizer turns the prompt into a list of token IDs and unsqueeze adds the batch dimension the model expects. That extra dimension is why the shape becomes one sequence inside a batch. The model expects input in batch form even when there is only one prompt. Adding that dimension keeps the tensor compatible with the GPT forward pass. Those numbers confirm that the prompt has been converted into tokens correctly and the tensor shape shows one batch with four tokens inside it. The IDs represent the four tokens in the prompt and the shape shows a single batch containing those four positions. That is exactly the format the generation function needs. Evaluation mode turns off training only randomness, such as dropout. Generation should be stable, so the model runs with those stochastic layers disabled. The model is switched into evaluation mode. Then the generation loop runs for six new tokens using the full context window. The output tensor includes both the original prompt and the newly generated token IDs, so the length increases by six. Dropout would randomly remove activations, which makes sense during training, but not during generation. Turning it off keeps the output deterministic for the same model weights and prompt. The printed tensor shows the original four IDs followed by six newly generated IDs. That confirms the loop appended one token at a time until the requested length was reached. The tensor now has 10 token IDs, which matches four prompt tokens plus six generated ones. That length is a direct check that the generation loop ran the expected number of times. Once the model has produced token IDs, the tokenizer can map them back into human-readable text. Decoding is the final step that turns a numeric sequence into a sentence. Squeeze removes the batch dimension. To list converts the tensor in a plain Python integers, and decode turns those integers into text. This is the bridge from model output back to language. The printed string is the model's generated text in readable form. At this point, the numeric token sequence has become an actual sentence fragment. The output is grammatically broken because the model has random initial weights and no training yet. A GPT architecture alone can run the generation loop, but it needs training before the text becomes coherent. That strange output is expected at this stage because nothing is taught the model which token sequences belong together. Training will give the logits meaningful structure and then generation starts to produce sensible text. This exercise asks for a cleaner way to manage dropout across the model. The goal is to separate the dropout settings so each part of the architecture can be controlled on its own. Three different places use dropout in the GPT model and each one could be given its own value instead of sharing one global setting. That change makes the architecture easier to tune and understand. These ideas fit together into the basic anatomy of a GPT model. Normalization, skip connections, transformer blocks, model scale, and text generation. The point is to see how the pieces support stable training and coherent output. Layer normalization makes the activations more predictable by keeping their mean and variance in a steady range. That steadier signal helps training move smoothly instead of bouncing around as values grow or shrink too much. Shortcut connections give a network a direct path around some layers. So, information and gradients do not have to pass through every transformation. In deep models, that makes optimization easier and reduces the vanishing gradient problem. A transformer block pairs masked multi-head attention with a feed-forward network. And gelu adds the non-linearity inside that second part. Masking keeps generation causal, so the model only uses earlier tokens when predicting the next one. GPT models are built by repeating that block many times, and the parameter count can get very large. More repeated structure gives the model more capacity to learn patterns in language, but it also makes stable training even more important. Different GPT sizes change the number of parameters, but the overall class structure can stay the same. That is why one GPT model Python class can describe several model scales just by changing the configuration. Text generation works by decoding model outputs into tokens, then choosing the next token from the current context again and again. Each step depends on what has already been produced, which is why the sequence can grow into readable text when the model is trained well. A model with a right architecture still needs training before its predictions become useful. Without learned weights, the token choices are essentially random, so the output loses coherence very quickly.