Skip to content

WordPiece

Overview

WordPiece builds a subword vocabulary by greedily merging pairs that most improve the likelihood of the training data (a language-model score), and marks word continuations with ##. It was created for Google's voice search (Schuster & Nakajima, 2012) and became famous as BERT's tokenizer— also used by DistilBERT, ELECTRA, and many multilingual models.

  • Paper: "Japanese and Korean Voice Search" (Schuster & Nakajima, 2012); popularized by BERT (Devlin et al., 2018)
  • Adoption: BERT, DistilBERT, ELECTRA, ALBERT, many multilingual BERTs
  • Trademark: ## continuation prefix— e.g. "tokenization" → ["token", "##ization"]
  • vs BPE: same bottom-up merging idea, different merge criterion (likelihood gain vs raw frequency)
  • Prerequisites: 00 Tokenization Fundamentals, 01 Bpe (Byte Pair Encoding)

-

How WordPiece Is Trained

The merge criterion: likelihood gain, not frequency

BPE picks the pair with the HIGHEST FREQUENCY:
 pair score = count(pair)

WordPiece picks the pair that most INCREASES data likelihood:
 pair score = freq(xy) / (freq(x) · freq(y))

Intuition:
 - freq(xy): how often x and y actually appear together
 - freq(x)·freq(y): how often they would appear together BY CHANCE
 - ratio > 1 → xy is a real collocation worth merging
 - ratio ≤ 1 → merging adds nothing (x,y co-occur by luck)

Example:
 pair "th" (freq 10M) vs pair "qk" (freq 10)
 BPE: "th" merges first (10M >> 10)
 WordPiece: compares against chance:
 freq(th)/(freq(t)·freq(h)) → moderate gain
 freq(qk)/(freq(q)·freq(k)) → HUGE gain (rare together
 beyond chance — a real unit)

→ WordPiece prioritizes *linguistically meaningful* units,
 not merely frequent ones. This is its main quality advantage.

The training loop

1. Start: vocabulary = all single characters (+ special tokens)
2. Repeat until target vocab size:
 a. compute pair scores: gain(xy) = freq(xy) / (freq(x)·freq(y))
 b. merge the highest-gain pair into a new token
3. Done: vocab contains characters + merged units

Note: the exact gain formula varies slightly between papers
(Wu et al. 2016 use freq(xy)/(freq(x)·freq(y)); BERT's original
implementation used a likelihood-based approximation). The idea
is the same: merge the pair that buys the most likelihood.

-

How WordPiece Encodes Text

Greedy longest-match (left to right)

Algorithm:
 word = "tokenization"

 1. Find the LONGEST prefix of the word that is in the vocab
 → "token" is in vocab (longer "tokeniz" is not) → take "token"
 2. Mark the rest as a continuation, repeat on the remainder
 → remainder "ization"
 3. Find the longest prefix of the remainder in the vocab
 → "##ization" is in vocab → take it
 4. Output: ["token", "##ization"]

Key rules:
 - the FIRST piece has no prefix; continuations are prefixed with ##
 - greedy: always take the longest vocab match available
 - if a character isn't in the vocab at all → [UNK]

Worked examples (BERT uncased vocab)

"playing" → ["playing"] (whole word in vocab)
"tokenization" → ["token", "##ization"] (## = continuation)
"unhappiness" → ["un", "##happi", "##ness"]
"ChatGPT" → ["chat", "##gp", "##t"] (cased: "Chat", "##GP", "##T")
"unaffable" → ["un", "##aff", "##able"]

Notice the pattern:
 - common roots (un-, -ness, -able) are kept whole
 - the first piece is the word "head", the rest carry ##

Why ## instead of Ġ/▁ (vs BPE/SentencePiece)

BPE/SentencePiece: space is a PREFIX → "Ġworld", "▁world"
WordPiece: continuation is a SUFFIX → "##world"

Both solve the same problem — keeping word boundaries — from
opposite sides. WordPiece's design makes the FIRST token of a word
look like a normal token and pushes the boundary signal to the end
(a design choice that suits BERT's full-sentence training).

-

WordPiece vs BPE vs SentencePiece (Comparison)

Aspect BPE (GPT) WordPiece (BERT) SentencePiece (LLaMA)
────────────────────────────────────────────────────────────────────────────
Merge criterion pair frequency likelihood gain BPE or Unigram mode
Word boundary Ġ / ▁ prefix ## suffix ▁ prefix
Pre-tokenization required required none (raw text)
Encoding greedy merges greedy longest-match Viterbi (unigram)
Languages English-centric English-centric language-agnostic
Special tokens <|endoftext|> [CLS][SEP][MASK] <s> </s> <unk>
Typical vocab 50K 30K 32K–128K
Models GPT-2/3/4 BERT, ELECTRA LLaMA, Mistral

Rule of thumb:
 - English-only understanding tasks (BERT-style): WordPiece is proven
 - Generation / multilingual: SentencePiece or byte-level BPE

-

BERT's Special Token Setup

BERT reserves the first IDs of its vocab for structural tokens:

BERT vocab (30,522 tokens) starts with:
 0: [PAD] 1: [UNK] 2: [CLS] 3: [SEP] 4: [MASK]

Usage:
 [CLS] prepended to every input → its output = sentence summary
 (used for classification)
 [SEP] separates sentences/segments:
 "[CLS] my dog is cute [SEP] he likes playing [SEP]"
 [PAD] pads batches to equal length (masked in attention)
 [MASK] the token BERT predicts during masked-language-model training
 [UNK] unknown characters (rare, thanks to 30K vocab + coverage)

BERT also keeps "unused" placeholder tokens:
 [unused1]... [unused99] → reserved for future fine-tuning use
 (you can repurpose them for custom special tokens WITHOUT
 resizing the embedding layer!)

More on special tokens: Special Tokens.

WordPiece in Practice (Hugging Face)

from transformers import AutoTokenizer

# BERT's tokenizer IS WordPiece
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

# Tokenize a sentence
text = "tokenization is playing with unhappiness"
tokens = tokenizer.tokenize(text)
print(tokens)
# ['token', '##ization', 'is', 'playing', 'with', 'un', '##happi', '##ness']

# Full encode → IDs (adds [CLS] and [SEP])
ids = tokenizer.encode(text)
print(ids)
# [101, 19204, 3721, 2003, 2692, 2007, 2439, 28352, 3813, 102]
# ↑[CLS] ↑[SEP]

# Decode back — identical text (tokenization is lossless)
print(tokenizer.decode(ids))
# "[CLS] tokenization is playing with unhappiness [SEP]"

Training a custom WordPiece tokenizer

from tokenizers import Tokenizer, models, pre_tokenizers, trainers

tokenizer = Tokenizer(models.WordPiece(unk_token="[UNK]"))
tokenizer.pre_tokenizer = pre_tokenizers.Whitespace()

trainer = trainers.WordPieceTrainer(
 vocab_size=30000,
 special_tokens=["[PAD]", "[UNK]", "[CLS]", "[SEP]", "[MASK]"],
)

tokenizer.train(files=["corpus.txt"], trainer=trainer)
encoded = tokenizer.encode("tokenization")
print(encoded.tokens) # ['token', '##ization']

-

Limitations & Gotchas

Requires pre-tokenization (space-splitting)
 → weak for Chinese/Japanese/Thai (no spaces)
 → multilingual WordPiece models pad the vocab to 100K+ and
 still lose efficiency — see [05 Multilingual Tokenization](/01-modeling/00-fundamentals/00-tokenization/05-multilingual-tokenization/)

Greedy longest-match encoding is fast but NOT optimal:
 - a rare word may tokenize badly if a long-but-rare prefix
 is in the vocab ("un" + "##affable" when "##unaffable" exists)
 - Viterbi/Unigram decoding (SentencePiece) can do better

Fixed special-token design is BERT-specific:
 - [CLS]/[SEP]/[MASK] don't transfer to generation models
 - modern LLMs (LLaMA, GPT) use different token designs

Still a great choice for:
 - classification/NLU tasks (BERT heritage, huge ecosystem)
 - English + European languages
 - tasks needing [CLS]-style sentence pooling

-

Key Takeaways

Merge by likelihood gain, not raw frequency— the WordPiece signature

Continuation marker: the first piece is the head, the rest carry ##

Greedy longest-match encoding: fast, good enough in practice BERT legacy: 30K vocab + [CLS]/[SEP]/[MASK] still powers NLU stacks Weak for no-space languages: pre-tokenization assumption limits it Pick by task: BERT-style NLU → WordPiece; generation/multilingual → SentencePiece or byte-level BPE

-