Character-Level & Word-Level Tokenization
A model can't read letters — it only understands numbers. So the very first step in any LLM pipeline is tokenization: splitting text into chunks called tokens, then mapping each chunk to a number.
Before getting to what real models actually do, it's worth seeing the two most obvious strategies fail — because the way they fail is exactly what motivates everything else in this course's tokenization arc.
Character-level tokenization is the simplest possible approach: every single character — letters, spaces, punctuation — is its own token. The vocabulary is tiny (every printable character fits in well under 100 slots), and it can represent any string, including words it has never seen, typos, made-up words, anything. There's no such thing as an unknown character.
Word-level tokenization goes the other direction: split on whitespace, and give every distinct word its own token. This makes sequences much shorter — a sentence is a handful of word-tokens instead of dozens of character-tokens — which matters a lot, since every token costs compute.
But it has a real, structural problem: the vocabulary has to be decided in advance, and English alone has hundreds of thousands of word forms (run, running, runner, reruns, every name, every typo). Any word outside that fixed vocabulary has nowhere to go.
Predict before you look
The word-level tokenizer below only knows about 70 common words. Before you look: what do you think happens to a word it's never seen — does it get split into pieces, dropped, or something else?
That's the core tradeoff, concretely: character-level tokenization never fails, but sequences balloon in length. Word-level tokenization keeps sequences short, but any word outside its fixed vocabulary collapses into a single, indistinguishable [UNK] token — the model loses all information about what that word actually was. A model trained this way would see "cryptocurrency" and "antidisestablishmentarianism" as the exact same token, no matter how different their meanings are.
Real tokenizers split the difference: keep common whole words as single tokens for efficiency, but fall back to smaller, guaranteed-to-exist pieces for anything rare — getting word-level's short sequences on common text and character-level's total coverage on everything else. The next lesson builds the algorithm that does exactly that.
Both tokenizers above, in plain Python — matching the exact same numbers:
A word-level tokenizer encounters a word that isn't in its fixed vocabulary. What happens?