What it's for
- Long context: the KV cache from chapter 2 grows with every token, and plain attention keeps one copy per head — sharing is what makes hundred-thousand-token conversations servable.
- Capacity: a smaller cache per token means longer histories, or more conversations, on the same GPU — it multiplies the wins from continuous batching.
Good to know
GQA shares whole keys and values inside each head group: eight query heads keep 4, 2 or 1 KV copies instead of 8 — keeping all 8 is plain multi-head attention, keeping 1 is multi-query, the two endpoints. In the reported results for the models tested, training with sharing holds quality essentially even with private copies. MLA goes further: each token's keys and values are compressed into one small shared latent, and every head re-expands its own view on the fly — the cache stores the latent, not the eight expansions. The re-expand is real compute the model is trained for, not a free trick, and the positional part of the attention state is kept and accounted separately. Both are training-time choices; this page shows the serving-side memory they buy.
How it works, step by step
Why the steps go in this order
- Attention needs the past: every head reads all previous keys and values, so they sit in memory — that is the cache from chapter 2.
- Plain multi-head attention keeps a separate copy per head, so the cache per token scales with the head count.
- Heads in a group end up needing much the same past, so one shared copy per group buys most of the memory back — GQA; one group total is MQA.
- MLA compresses instead of sharing: one small latent per token, re-expanded per head, shrinks the cache further while keeping each head's own view.
Before you rely on it
Eight heads and the per-token copy counts are a toy scale — real models use dozens of heads, and the exact MLA ratio depends on the model. The meter counts the KV cache only: keys and values per token, per layer, with head size and storage type held fixed — a model's query weights are not the cache. The direction and the mechanism are exactly as shown: sharing or compressing the cache multiplies how much past a GPU can hold. GQA and MLA are trained in, not serving-time switches — a live model cannot hop between them without trained or adapted weights.