Byte-Pair Encoding: Building It From Scratch

The last lesson ended on a question: is there a way to get word-level tokenization's short sequences on common text, without word-level's total failure on anything unfamiliar? There is, and the specific algorithm virtually every modern LLM uses for it is Byte-Pair Encoding (BPE).

BPE is genuinely simple:

1. Start with every word split into individual characters — the character-level tokenizer from last lesson.
2. Find the pair of adjacent symbols that occurs most often across the whole training corpus.
3. Merge that pair into a single new symbol.
4. Repeat, building progressively larger chunks out of whatever appears most frequently.

No linguist decides where the boundaries go — the data does, purely by frequency. A word that appears constantly in training (the) quickly merges into one token; a rare or novel word stays broken into smaller, guaranteed-to-exist pieces — exactly the middle ground the previous lesson was missing.

That tiny corpus is the classic example from the original BPE paper. Step through the merges and watch common patterns emerge — e+s merges early because "newest" and "widest" both contain it, then es+t merges into est for the same reason. Once you've learned a few merges, type any text below and see it tokenized using exactly those rules — including words the tokenizer never saw during training, which still get represented by falling back to smaller pieces, never an [UNK].

That grid showed merges as a flat list. But "merge" is a genuinely hierarchical idea — each new symbol is built out of two earlier ones, which might themselves have been built out of two earlier ones. Here's the exact same real merges, for a single word, drawn as the literal tree they're building:

Every leaf at the bottom is a raw character. Every node above it is one real merge event — click through and watch the tree grow upward, each new node literally built from gluing its two children together. This is the same computation as the flat merge list above; drawing it as a tree just makes the hierarchy visible — why a symbol like est isn't a mysterious atomic chunk, it's e+s merged, then that result merged with t.

Notice how common whole words stay together once enough merges have been learned, while rare or unfamiliar text splits into smaller, more common pieces. This tokenized sequence — a list of numbers, one per learned symbol — is what actually gets fed into the model. But raw BPE isn't quite what production tokenizers ship with; the next lesson covers the real variants (WordPiece, SentencePiece) and tokenizes text with an actual model's real vocabulary.

The whole algorithm — training the merges, then applying them — in plain Python: