What it's for
- Running a big model on one GPU: quantization is why an 8B model leaves room for a long chat, and why a 70B model can run on a laptop at all.
- Every local-LLM download you have seen — the 4-bit files behind Ollama, llama.cpp and friends — is this trade, packaged.
Good to know
Weight memory = parameters × bytes per weight. Real formats add a few percent for scales and zero-points, and the quality cost is measured as a small rise in perplexity — the exact number depends on the model, the method and the task.
How it works, step by step
Why the steps go in this order
- Weights are storage, not live computation: rounding a stored number to a nearby level changes it by less than half a percent of the range, and the network tolerates that.
- Halving the bytes halves the memory — the mechanism is the rounding, the saving is arithmetic.
- Weights are the bulk of the bill, so shrinking them is what frees room — and the freed room is exactly what the KV cache from the previous chapter needs.
- Coarser levels buy more room at a slightly larger error, which is why INT8 is the safe default and INT4 is the aggressive one.
Before you rely on it
Forty-eight blocks stand in for eight billion weights, and the quality percentages are illustrative — real perplexity deltas depend on the model, the method (GPTQ, AWQ, GGUF) and the task. The snap-to-the-nearest-level mechanism and the memory arithmetic are exactly as shown.