What it's for
- Every chapter so far started at "tokens": the tokenizer is where text becomes them — the model's alphabet, fixed before training begins.
- Tokens are the unit of everything downstream: context windows, prices, speed and even strange spelling behavior are all counted in these tiles.
Good to know
This is a character-based, BPE-style toy — faithful to the merge idea, not to any production tokenizer. Real systems like byte-level BPE start from bytes, learn a ranked merge list from gigabytes of text, then freeze it and ship it with the model. The replay here applies that kind of frozen list in its fixed rank order, leftmost occurrence first — it never re-learns from the sentence on the tray. A token's id is its row number in the vocabulary: not a meaning, not an embedding, not a character code; the numbers carry no distances between words.
How it works, step by step
Why the steps go in this order
- A neural network multiplies numbers, so text must become numbers before anything else can happen.
- One number per letter wastes the budget on spelling; one per word needs an unbounded vocabulary. Learned merges sit in between.
- Frequent pairs earn their own tile, so common words compress to one token while rare words stay spelled out in pieces.
- Because the merge list is frozen, every model splits text its own way — and every bill, limit and window counts its tokens.
Before you rely on it
The vocabulary, ids and token counts on this page are a toy of a few dozen entries — toy counts, not any provider's billing counts. Real vocabularies hold tens of thousands of tokens and split text differently per model, and only the preset words in the sandbox are supported input. The mechanism is exactly as shown: learned, frozen merges applied in rank order; one id per tile; tokens, not letters, are what get counted.