What it's for
- Every token a model writes starts as scores over its whole vocabulary. Decoding is the rule that turns those scores into one pick — greedy, sampled, filtered.
- The same model can sound careful or wild without any retraining: temperature and filters reshape the shares before the pick, and each sampled roll can land differently.
Good to know
The bars here come from six declared, fixed toy logits — the complete toy candidate set, not a production vocabulary of tens of thousands. The order of operations is fixed and shown: temperature divides the logits first, softmax makes shares, an optional top-p cut keeps the smallest set of candidates covering 90% (including the token that crosses the threshold), the survivors are rescaled to sum to 1, and only then is u drawn. Greedy bypasses all of it: it is argmax, the cold limit as T shrinks — never a roll. Tied top scores would break toward the lower id; this toy never needs the rule. Temperature reshapes the decoder's distribution for fixed logits; it does not retrain the model, though the token it picks becomes context and changes the next step's scores.
How it works, step by step
Why the steps go in this order
- The model's output is one score per candidate — raw logits, unnormalized. Softmax turns them into shares of one whole so they can drive a pick.
- Greedy is the simplest rule: take the max. It is deterministic — same scores, same word, forever.
- Temperature divides the logits before softmax: cold (T below 1) sharpens the gaps toward greedy, hot flattens them toward uniform over the candidates still in play.
- Sampling draws one uniform number u from [0, 1) and takes the candidate whose cumulative interval contains it — bigger share, bigger interval, likelier pick. The die on screen only cues the draw; the strip and the printed u are the mechanism.
- Top-p trims the tail before rolling, so near-zero candidates stop drawing an occasional surprise.
Before you rely on it
A cut candidate is not impossible under the original distribution — the filter removes it and rescales the rest, and different filters or a different order change the result. Higher temperature is not universally more creative or more accurate; it approaches uniform over the unmasked candidates, no more. Real services' shares differ with the model version, settings and context, so the same prompt does not promise the same distribution. The six candidates, logits, shares and draws on this page are a declared toy; the mechanism is exactly as shown: stable softmax over z/T, one uniform draw, cumulative intervals that match the printed numbers.