Skip to content

WordPiece: BERT's Tokenizer

Overview

WordPiece builds a subword vocabulary by greedily merging pairs that most improve the likelihood of the training data (a language-model score), and marks word continuations with ##. It was created for Google's voice search (Schuster & Nakajima, 2012) and became famous as BERT's tokenizer β€” also used by DistilBERT, ELECTRA, and many multilingual models.

  • Paper: "Japanese and Korean Voice Search" (Schuster & Nakajima, 2012); popularized by BERT (Devlin et al., 2018)
  • Adoption: BERT, DistilBERT, ELECTRA, ALBERT, many multilingual BERTs
  • Trademark: ## continuation prefix β€” e.g. "tokenization" β†’ ["token", "##ization"]
  • vs BPE: same bottom-up merging idea, different merge criterion (likelihood gain vs raw frequency)
  • Prerequisites: 00 Tokenization Fundamentals, 01 Bpe (Byte Pair Encoding)

How WordPiece Is Trained

The merge criterion: likelihood gain, not frequency

BPE picks the pair with the HIGHEST FREQUENCY:
  pair score = count(pair)

WordPiece picks the pair that most INCREASES data likelihood:
  pair score = freq(xy) / (freq(x) Β· freq(y))

Intuition:
  - freq(xy): how often x and y actually appear together
  - freq(x)Β·freq(y): how often they would appear together BY CHANCE
  - ratio > 1 β†’ xy is a real collocation worth merging
  - ratio ≀ 1 β†’ merging adds nothing (x,y co-occur by luck)

Example:
  pair "th" (freq 10M) vs pair "qk" (freq 10)
  BPE:            "th" merges first (10M >> 10)
  WordPiece:      compares against chance:
                    freq(th)/(freq(t)Β·freq(h))  β†’ moderate gain
                    freq(qk)/(freq(q)Β·freq(k))  β†’ HUGE gain (rare together
                                                    beyond chance β€” a real unit)

β†’ WordPiece prioritizes *linguistically meaningful* units,
  not merely frequent ones. This is its main quality advantage.

The training loop

1. Start: vocabulary = all single characters (+ special tokens)
2. Repeat until target vocab size:
   a. compute pair scores: gain(xy) = freq(xy) / (freq(x)Β·freq(y))
   b. merge the highest-gain pair into a new token
3. Done: vocab contains characters + merged units

Note: the exact gain formula varies slightly between papers
(Wu et al. 2016 use freq(xy)/(freq(x)Β·freq(y)); BERT's original
implementation used a likelihood-based approximation). The idea
is the same: merge the pair that buys the most likelihood.

How WordPiece Encodes Text

Greedy longest-match (left to right)

Algorithm:
  word = "tokenization"

  1. Find the LONGEST prefix of the word that is in the vocab
     β†’ "token" is in vocab (longer "tokeniz" is not) β†’ take "token"
  2. Mark the rest as a continuation, repeat on the remainder
     β†’ remainder "ization"
  3. Find the longest prefix of the remainder in the vocab
     β†’ "##ization" is in vocab β†’ take it
  4. Output: ["token", "##ization"]

Key rules:
  - the FIRST piece has no prefix; continuations are prefixed with ##
  - greedy: always take the longest vocab match available
  - if a character isn't in the vocab at all β†’ [UNK]

Worked examples (BERT uncased vocab)

"playing"          β†’ ["playing"]            (whole word in vocab)
"tokenization"     β†’ ["token", "##ization"] (## = continuation)
"unhappiness"      β†’ ["un", "##happi", "##ness"]
"ChatGPT"          β†’ ["chat", "##gp", "##t"]  (cased: "Chat", "##GP", "##T")
"unaffable"        β†’ ["un", "##aff", "##able"]

Notice the pattern:
  - common roots (un-, -ness, -able) are kept whole
  - the first piece is the word "head", the rest carry ##

Why ## instead of Δ /▁ (vs BPE/SentencePiece)

BPE/SentencePiece:   space is a PREFIX  β†’ "Δ world", "▁world"
WordPiece:           continuation is a SUFFIX β†’ "##world"

Both solve the same problem β€” keeping word boundaries β€” from
opposite sides. WordPiece's design makes the FIRST token of a word
look like a normal token and pushes the boundary signal to the end
(a design choice that suits BERT's full-sentence training).

WordPiece vs BPE vs SentencePiece (Comparison)

Aspect            BPE (GPT)          WordPiece (BERT)      SentencePiece (LLaMA)
────────────────────────────────────────────────────────────────────────────
Merge criterion   pair frequency     likelihood gain       BPE or Unigram mode
Word boundary     Δ  / ▁ prefix       ## suffix             ▁ prefix
Pre-tokenization  required           required              none (raw text)
Encoding          greedy merges      greedy longest-match  Viterbi (unigram)
Languages         English-centric    English-centric       language-agnostic
Special tokens    <|endoftext|>      [CLS][SEP][MASK]     <s> </s> <unk>
Typical vocab     50K                30K                   32K–128K
Models            GPT-2/3/4          BERT, ELECTRA         LLaMA, Mistral

Rule of thumb:
  - English-only understanding tasks (BERT-style): WordPiece is proven
  - Generation / multilingual: SentencePiece or byte-level BPE

BERT's Special Token Setup

BERT reserves the first IDs of its vocab for structural tokens:

BERT vocab (30,522 tokens) starts with:
  0: [PAD]      1: [UNK]     2: [CLS]     3: [SEP]     4: [MASK]

Usage:
  [CLS]   prepended to every input β†’ its output = sentence summary
          (used for classification)
  [SEP]   separates sentences/segments:
          "[CLS] my dog is cute [SEP] he likes playing [SEP]"
  [PAD]   pads batches to equal length (masked in attention)
  [MASK]  the token BERT predicts during masked-language-model training
  [UNK]   unknown characters (rare, thanks to 30K vocab + coverage)

BERT also keeps "unused" placeholder tokens:
  [unused1] ... [unused99]  β†’ reserved for future fine-tuning use
  (you can repurpose them for custom special tokens WITHOUT
   resizing the embedding layer!)

πŸ”— More on special tokens: Special Tokens.

WordPiece in Practice (Hugging Face)

from transformers import AutoTokenizer

# BERT's tokenizer IS WordPiece
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

# Tokenize a sentence
text = "tokenization is playing with unhappiness"
tokens = tokenizer.tokenize(text)
print(tokens)
# ['token', '##ization', 'is', 'playing', 'with', 'un', '##happi', '##ness']

# Full encode β†’ IDs (adds [CLS] and [SEP])
ids = tokenizer.encode(text)
print(ids)
# [101, 19204, 3721, 2003, 2692, 2007, 2439, 28352, 3813, 102]
#   ↑[CLS]                                              ↑[SEP]

# Decode back β€” identical text (tokenization is lossless)
print(tokenizer.decode(ids))
# "[CLS] tokenization is playing with unhappiness [SEP]"

Training a custom WordPiece tokenizer

from tokenizers import Tokenizer, models, pre_tokenizers, trainers

tokenizer = Tokenizer(models.WordPiece(unk_token="[UNK]"))
tokenizer.pre_tokenizer = pre_tokenizers.Whitespace()

trainer = trainers.WordPieceTrainer(
    vocab_size=30000,
    special_tokens=["[PAD]", "[UNK]", "[CLS]", "[SEP]", "[MASK]"],
)

tokenizer.train(files=["corpus.txt"], trainer=trainer)
encoded = tokenizer.encode("tokenization")
print(encoded.tokens)  # ['token', '##ization']

Limitations & Gotchas

❌ Requires pre-tokenization (space-splitting)
   β†’ weak for Chinese/Japanese/Thai (no spaces)
   β†’ multilingual WordPiece models pad the vocab to 100K+ and
     still lose efficiency β€” see [05 Multilingual Tokenization](/02-llm-modeling/00-fundamentals/00-tokenization/05-multilingual-tokenization/)

❌ Greedy longest-match encoding is fast but NOT optimal:
   - a rare word may tokenize badly if a long-but-rare prefix
     is in the vocab ("un" + "##affable" when "##unaffable" exists)
   - Viterbi/Unigram decoding (SentencePiece) can do better

❌ Fixed special-token design is BERT-specific:
   - [CLS]/[SEP]/[MASK] don't transfer to generation models
   - modern LLMs (LLaMA, GPT) use different token designs

βœ… Still a great choice for:
   - classification/NLU tasks (BERT heritage, huge ecosystem)
   - English + European languages
   - tasks needing [CLS]-style sentence pooling

Key Takeaways

πŸ“ Merge by likelihood gain, not raw frequency β€” the WordPiece signature

Continuation marker: the first piece is the head, the rest carry ##

🧠 Greedy longest-match encoding: fast, good enough in practice
πŸ† BERT legacy: 30K vocab + [CLS]/[SEP]/[MASK] still powers NLU stacks
🌍 Weak for no-space languages: pre-tokenization assumption limits it
πŸ”€ Pick by task: BERT-style NLU β†’ WordPiece; generation/multilingual β†’ SentencePiece or byte-level BPE