Playground

Tokenizer Playground

Compare real tokenizers side by side on your own text.

Compare real tokenizers on your own text

Character-level is trivial. BPE trains real merges live on whatever you type below. WordPiece runs the real greedy-longest-match algorithm against BERT's actual ~30,500-token vocabulary. No scheme here is a simplification.

Type or paste any text

BPE merges learned13 / 13

Real merges, trained live on the text above, the same algorithm the Model Builder playground uses to tokenize your own training text.

Inspect one word's real merge tree

Loading BERT's real vocabulary (once per visit)…

SentencePiece isn't shown here, this site has never reimplemented it in JavaScript, the Model Builder's Python export calls the real sentencepiece library instead of faking it in the browser, and this playground holds that same line.

Character-level

vocab ~96 · 53 tokens

unhappiness·transformers·antidisestablishmentarianism

BPE (live-trained)

vocab 20 · 33 tokens

unhappiness·transformers·antidisestablishmentarianism·

WordPiece (real BERT vocab)

vocab … · 0 tokens

"unhappiness", as a real merge tree

Every leaf is a single character. Each gray line is one real merge, learned from the text itself, moving the slider replays them in the exact order they were actually learned.