What it's for
- Every chat with an LLM: after the first token, the KV cache is the difference between reading the past once and re-reading it for every word.
- Paged attention (vLLM's idea) is why serving many long conversations fits in GPU memory at all.
Good to know
The cache trades memory for speed: bytes per token ≈ 2 × layers × KV heads × head size × 2 bytes. An 8B-scale model is roughly 128 KB per token, so a 128k-token conversation is about 16 GB of cache.
How it works, step by step
Why each step follows
- Attention needs every past token's K and V, and those never change once written — so re-computing them per token pays a quadratic bill for zero new information.
- Keeping them turns the per-token cost into one small append and a read in place; the cost moves from traffic to capacity.
- Capacity then dominates, so reservation waste matters; fixed-size pages allocated on demand remove the empty space, exactly like OS paging.
- Pages with stable addresses can be mapped by many sequences, which is why shared prefixes, forks and beam search read one copy.
Before you rely on it
Six slabs stand in for thousands of tokens, and the byte numbers come from an illustrative 8B-scale decoder, not a measured GPU. The quadratic re-read, the per-token append, the reservation waste and the page sharing are exactly as shown.