Multi-Head Latent Attention (MLA) - Explained

The KV cache, not the matrix math, is what fills your memory when a language model generates long text. For every token, attention stores a key and a value for every head, and that store grows without bound. The classic fixes, grouped-query and multi-query attention, shrink the cache by making heads share keys and values, but the moment heads read the same key they all look at each token in the same way, so you buy a smaller cache by erasing what made the heads different. Multi-Head Latent Attention (MLA), introduced with DeepSeek-V2, refuses that trade. Instead of sharing heads, it compresses what it stores: a down-projection squeezes the hidden state into one small latent vector, and that single vector is the only thing cached. Per-head up-projections rebuild full, distinct keys and values on demand, so the cache is head-agnostic while the heads stay diverse. Then comes the free lunch, absorption: the up-projections fold straight into the query and output matrices, so attention runs directly in latent space with no reconstruction at all. The result looks like multi-query attention but keeps full multi-head diversity. We finish with the RoPE wrinkle, where a position-dependent rotation breaks absorption, and the clean fix of splitting the key into an absorbable content lane plus a tiny shared positional lane. *Related Videos* ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ Self-Attention - Explained: https://youtu.be/chSVRI_tQXk Attention Mechanism Variations: https://youtu.be/LremmIyKr0w Sliding Window Attention: https://youtu.be/_8Vcui3DSbQ Softmax function - Explained: https://youtu.be/oJU6-qW6xZU Word2Vec — How Words Became Vectors: https://youtu.be/9Rd6sh-PmlY Mixture of Experts (MoE) - Explained: https://youtu.be/X-gfpWG6XDU Recurrent Neural Networks (RNNs) - Explained: https://youtu.be/8G1fImBCMcQ An LSTM is a Conveyor Belt of Memory: https://youtu.be/Lx0NdM5RBXU *Contents* ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 00:00 - The KV cache memory wall 00:44 - MHA, GQA, MQA: cache vs diversity 01:31