Parallel Processing: Why GPUs Dominate AI

At the lowest level, every chip on Earth does the same thing. It adds and multiplies numbers, and that's all there is to it. Every chip wins or loses on two questions. How many of those multiplications can you do at once, and how fast can you change your plan when the work changes shape? Those two questions alone split the entire chip universe. Start with the CPU, your central processing unit. A modern CPU has somewhere between eight and 24 cores, and each one of them is huge, complicated, and extremely smart. Inside a single core lives a branch predictor that guesses where your code will go next, an out-of-order engine that rearranges your instructions to run them faster than you wrote them, and three layers of cache that keep your data nanoseconds away from the math units. All that machinery exists because real code is unpredictable. So, the CPU compensates constantly. When you write an if-else, it guesses which way you'll go. When you write a loop, it figures out how to speed it up. And when you hit a memory address, it has the data ready before you even ask. The result is that a CPU can finish a single task in about 10 nanoseconds, which is the fastest single-task response in all of computing. This is the chip that runs your operating system, your browser, the database query you just hit, and the Python interpreter you opened this morning. It handles anything that has to make a decision. The trade-off is that each core is expensive in silicon area, so you simply cannot fit thousands of them on one chip. If your job is to do the same thing a million times in a row, the CPU will get there eventually, but slowly. Now, the GPU, or graphics processing unit. It's doing the same job at the bottom, multiplying numbers, but it has a completely different shape. An Nvidia H100 packs 16,896 cores, and a consumer RTX 4090 has 16,384. Each one of those cores is tiny and dumb, with no branch predictor, no out-of-order engine, and almost no cache. All it has is an arithmetic unit and a place to park a few numbers. A GPU is built for one specific situation. The same dumb operation done 16,000 times in parallel on 16,000 different pieces of data. This all started with graphics because every pixel on your 4K screen needs the exact same lighting math. With 8 million pixels all running one calculation, that's a perfect job for a chip with 16,000 identical workers. Then someone noticed that training a neural network is mostly multiplying giant matrices, which is also the same dumb operation done millions of times. And just like that, the graphics chip became the AI chip without changing a single transistor. A GPU also drinks data 30 times faster than a CPU thanks to stacked memory called HBM that pushes 3 terabytes per second straight into those 16,000 cores. It's a beast and it has to be constantly fed, but there's a trade-off. The GPU is terrible at branching code. Give it 16,000 different things to do and it stalls. It only wins when you have one job to do and a lot of it. Now, the TPU or Tensor Processing Unit. Google built it for one specific reason. Even GPUs were still too general and most of what an AI model does is one specific operation, matrix multiplication. So Google asked, what if we built a chip that does nothing else? Inside a TPU lives a thing called a systolic array. Picture a grid of multiplier accumulator cells where numbers stream in from the top and the left, flow through the grid like water through pipes, and the results pour out the bottom. There are no instructions to decode and no control logic to manage, just numbers moving through silicon. A single TPU V4 chip performs 65,536 multiply adds in one clock cycle, while a high-end GPU manages maybe 4,000. That makes the TPU roughly 16 times denser at the one thing it actually does. The trade-off here is extreme. Try to run a Python script on a TPU and nothing happens. Try to render Cyberpunk on it and again, nothing happens. All it does is neural network training and serving, and that is the entire menu. And it doesn't have to be a TPU. Apple has the neural engine, AWS has Trainium, and Tesla has Dojo. They all share one idea. Kill flexibility, win at one job. So, which chip wins where? If the job is loading a webpage, running a database, or executing the code you wrote yesterday, the answer is the CPU, every single time. Latency matters more than parallelism. If the job is training a neural network with custom layers, running a physics simulation, or re-encoding a 4K video, you want the GPU. You need massive parallelism, but you also need to occasionally change what those parallel cores are doing. And if the job is serving billions of inference requests on a fixed model architecture at the lowest possible cost per query, you want a TPU, or whichever specialized accelerator your cloud sells you. You've committed to one shape of math, and now you just want the cheapest way to do that math forever. In the real world, your laptop has a CPU and a GPU. Your phone has a CPU, a GPU, and a small NPU, which is basically a phone-sized TPU. Google's data centers run all three at once. ChatGPT lives on H100 GPUs, while Gemini lives on TPUs. Different teams making different bets on the same problem underneath. Here is the entire video in three sentences. The CPU is fast at deciding what to do next. The GPU is fast at doing the same thing many times. And the TPU is fast at one thing and one thing only. General, parallel, specialized. Pick the wrong chip and your hardware bill explodes. But pick the right one and your workload runs at a tenth of the cost of everyone else's. If you actually want to use these chips, whether it's training a model, fine-tuning an LLM, or building inference at scale, the link below has hands-on labs for all of it.