What exactly is a diffusion language model?

Key Takeaways

What exactly is a diffusion language model? Let's try to understand in a simple way. A normal LLM writes one word at a time, strictly left to right. Watch the arrow crawl across the sentence. Each token waits for the one before. A diffusion language model starts somewhere odd. The whole sentence is blank. A full row of gray masked tokens, every slot hidden. The idea comes from image diffusion. Instead of cleaning up noisy pixels, here we clean up masked words. We denoise the entire row together. So, the model looks at all the blanks and guesses everyone in parallel. See the gray boxes light up with words, the whole sentence in one pass. But, early guesses are rough. So, we keep only the tokens the model is most confident about. Those lock in, glowing. The rest go back to gray. Then, we run it again. The locked words stay, the blanks get fresh guesses, and a few more snap into place. The row keeps filling in. And it doesn't take many rounds. After just a handful of steps, every slot is solid, a clean sentence denoised from blanks. Why do this? Because the words don't wait in line. The model fills many positions in one shot, so it can be fast. And these are real. Models like LLAMA and Mercury work this way. Diffusion, the trick behind AI images, writing text. So, quick recap. Start with a fully masked sentence. Guess every word in parallel. Keep the confident ones, blank the rest, and repeat. If diffusion for text just clicked, hit subscribe. More short explainers are coming.