LLM Quantization Explained

When we use modern-day LLMs like Chatypt and Claude, an important consideration to ask is where do these models actually live? Behind chat interfaces, the model is actually a giant file filled with numbers called parameters and weights. At a high level, you can think of these numbers as knobs and dials that the model learned during training. And after the training is all completed, these knobs are tuned just right so that we can get the right output when we ask Chachi to write a poem about us. But there is a slight problem because all of these numbers need to be stored somewhere and they require space. Let's look at the llama 3.170 billion parameter model for example, which is one of the most popular open weights model that are out there. Meta Lama 3.170B model has roughly around 70 billion parameters for the model. If each parameters or each number takes up two bytes, that means the model would take up around 70 billion* 2 which adds up to about 140 GB in RAM. So if you actually wanted to run this model normally, you would need enough GPU memory to fit around 140 GB of weight, which is way more than what a normal consumer GPUs has. And that is what quantization tries to solve. Quantization is trying to make a giant model smaller, cheaper, and sometimes faster by storing their numbers that we talked about with fewer bytes. For LMS, it started to become popular around 2022 to 2023. And in 2022, research showed that you could take these massive transformer models and store them in a much lower 8bit, 4-bit, or even threebit precision without completely destroying the model. But the moment it really broke into the mainstream developer world was around 2023 when Meta released Llama and tools like Llama.cpp made it possible to run quantized LLMs locally on normal consumer hardware. So how exactly do we shrink the models using quantization? There are three ideas that people tend to mix together. Number format will include terms that you might have heard of like FP16, BF16, FP8, IN8, and IN4. These answer the questions of how many bits are we using to store the numbers. Same number of parameters but each number is stored with less precision each time. Second, there is quantization method. This includes methods like GPTQ or AWQ and these answer the question of how do we convert the model into a smaller format without destroying the output because you can't always just blindly round that number down and hope that the model still works. For example, AWQ tries to identify which weights actually matter and preserve the more important ones carefully. And third, there is the file format. And there are file formats like GGUF. And that's where it comes in. So just like the videos that we might have, they're stored in file formats like MP4. A quantized LM might be saved as GGUF file formats. And that GGUF file format contains the models weights plus extra information so that tools like Llama.cppama CBP, Olama and LM Studio can actually run them and load the model locally. So the question then comes down to who should actually quantize the model. It should really be the model provider, the inference team or whoever is actually deploying these models because quantization is not just making the model smaller. You need to know what hardware the model is actually running on, what runtime is being used, and most importantly, whether the model still performs well on the task that you care about after being quantized. And this brings us to the trade-off in quantization. Quantization makes the model smaller, cheaper, and sometimes faster. But the lower the precision you go, the more risk you run in actually hurting the quality of the model's output. the model might get worse at reasoning, coding, math, long context tasks, or following specific instructions. And speed is not guaranteed either. So a 4-bit model is smaller, but it is only faster if the hardware and the software that actually run are optimized for that file format. So the real question is not just can we shrink the model using quantization, but the real question is can we shrink the model while keeping it useful.