What it's for
- The whole book, not a page of it: codebases, legal archives, a year of support tickets. Long context is what lets the model read everything you have instead of a summary of it.
- Two walls stand in the way, and this chapter opens both doors: attention needs to know where tokens are (positions run out past the training length — RoPE's door), and the KV cache from the earlier chapter grows with every token (memory runs out — the ring's door).
Good to know
Two different mechanisms share this chapter, and they are not the same idea wearing two hats. The first is positional encoding. RoPE — rotary position embedding — doesn't add a position to a token; it rotates pairs of components of each token's query and key vectors by position-dependent angles, at several frequencies (the scene shows one component pair in a 2D plane, not the whole embedding; the value vectors are untouched). Under the standard assumptions the rotated query-key dot product depends on the relative rotation between two positions — but the tokens' content still decides what actually gets attended; distance alone determines nothing. The honest caveat: rotate far enough and you're in angles the model never saw in training, and extrapolation past the training length is not automatically reliable. Quality degrades rather than cliffs, and position interpolation — linearly rescaling the angles so unseen ones land in trained territory — is the named fix (that's the third ending). The second mechanism is distributed computation. Ring attention splits the sequence into blocks across devices and circulates the key/value blocks around the ring, while each device accumulates exact attention with an online softmax — no averaging of independently normalized block outputs, and masks and global token positions kept consistent. After one lap, every token has attended to everything, and no device ever held the whole cache. The ring is an algorithm schedule, not hardware cabling: it cuts per-device storage, not the quadratic attention arithmetic, and the network between devices becomes the bottleneck the load balancer chapter would recognize. Training and prefill parallelize this way; autoregressive decoding is a different regime. And this is neither sliding-window nor sparse attention — every token still sees every token. It is the system-design part's own move, sharding, wearing a math costume.
How it works, step by step
Why the steps go in this order
- The growing prompt comes first: two walls are about to appear, and both need the same motivation.
- The position wall before the memory wall: RoPE is the smaller, prettier idea, and it fails alone — the cache still doesn't fit.
- The rotation is shown before it's named: tokens visibly turning beats the formula.
- The ring comes after the cache problem is stated plainly: four devices, one lap, everything attended.
- Extrapolation comes last: it's the honest limit of door number one, and the reason "a million tokens" is an engineering number, not a math one.
Before you rely on it
This is a toy: dozens of token cubes stand in for a million tokens, four devices for a datacenter, and the "quality" of extrapolated positions is a hand-set fade, not a measurement. Real long-context systems report their own context lengths and degradation curves — the cited papers carry the measured behavior. Blockwise attention can also differ numerically from dense attention under finite precision, and real speedups depend on block size, topology, communication overlap and hardware. And long context doesn't make attention cheap: a million-token prompt is a bill, not a feature flag.