Lesson 6 of 15

WordPiece, SentencePiece & Choosing a Real Tokenizer

The BPE you just built picks merges by raw frequency: whichever adjacent pair shows up most often, merge it. That's simple and effective, but it's not exactly what most production tokenizers do. Two real variants are worth knowing by name, because you'll see them constantly: WordPiece and SentencePiece.

WordPiece (used by BERT and its relatives) is BPE's close cousin with one key difference: instead of merging whichever pair is most frequent, it merges whichever pair most increases the likelihood of the training data — roughly, "does merging this pair make the corpus more predictable?" rather than "does this pair just show up a lot?" In practice the results look similar to BPE, but the selection criterion is genuinely different math.

The other visible difference is bookkeeping: WordPiece marks any piece that continues a word (rather than starting one) with a ## prefix — so unhappiness becomes un + ##ha + ##pp + ##iness in this model's real vocabulary (try it in the widget below). That prefix is how the tokenizer (and anything reading its output) knows where word boundaries actually are.

Predict before you look

Type a long, unusual word below. Before you look: do you expect it to fall back to a single [UNK], the way the word-level tokenizer did two lessons ago, or something else?

This isn't a simulation — it's the real WordPiece algorithm running against the real vocabulary of an actual trained model (the same one powering the attention and embedding widgets elsewhere in this course). Notice it essentially never falls back to [UNK]: even a word like antidisestablishmentarianism decomposes into real, meaningful sub-pieces. That's the entire point of subword tokenization — the fixed-vocabulary failure mode from two lessons ago is gone, because the vocabulary always has some valid decomposition down to individual characters if it needs to.

SentencePiece (used by T5, Llama, and many others) solves a different problem: everything so far assumed you can split on whitespace before tokenizing — but that assumption breaks for languages like Japanese or Chinese that don't use spaces between words at all. SentencePiece treats the input as a raw stream of characters (including spaces as ordinary characters, often marked with a ▁ symbol) and learns its subword vocabulary directly from that stream — no language-specific pre-splitting required. It also supports a unigram language-model algorithm as an alternative to BPE-style merging, which starts from a large vocabulary and prunes it down rather than starting small and merging up.

How the same real WordPiece split from the widget above would be marked by each scheme (the marking conventions are real; only the WordPiece row is actually computed from a live model — SentencePiece isn't wired up on this page):

WordPiece (real, from the widget above): un ##ha ##pp ##iness — ## marks "continues the previous piece"
BPE (raw, illustrative): un ha pp iness — no boundary marking at all
SentencePiece (illustrative): ▁un ha pp iness — ▁ marks "this piece starts a new word"

All three are doing fundamentally the same subword-decomposition job — the differences are in how pieces get chosen (frequency vs. likelihood vs. unigram pruning) and how boundaries get marked (## prefix vs. ▁ prefix vs. nothing).

The widget above fetches the real vocabulary and runs the real WordPiece algorithm live, right here, in Python:

Python

from pyodide.http import pyfetch

manifest_resp = await pyfetch("/data/bert-tiny-manifest.json")
data = await manifest_resp.json()
vocab = set(data["wordpieceVocab"])

def tokenize_wordpiece(text, vocab, max_word_len=100):
    words = text.lower().split()
    tokens = []
    for word in words:
        if len(word) > max_word_len:
            tokens.append("[UNK]")
            continue
        sub_tokens = []
        start = 0
        failed = False
        while start < len(word):
            end = len(word)
            match = None
            while start < end:
                piece = ("##" if start > 0 else "") + word[start:end]
                if piece in vocab:
                    match = piece
                    break
                end -= 1
            if match is None:
                failed = True
                break
            sub_tokens.append(match)
            start = end
        tokens.extend(["[UNK]"] if failed else sub_tokens)
    return tokens

print(f"real vocabulary size: {len(vocab)} tokens")
print()
print(tokenize_wordpiece("unhappiness transformers", vocab))

What does a "##" prefix mean in WordPiece tokenization?

← Byte-Pair Encoding: Building It From Scratch Embeddings: Turning Tokens Into Meaning →