00 tokenization fundamentals

Overview¶

Tokenization converts raw text into discrete tokens (words, subwords, characters) that models can process. Fundamental to every LLM but often overlooked. Quality tokenization directly impacts model efficiency, quality, and cost.

Purpose: Text → Tokens → Embeddings → Model
Impact: Affects model capacity, training cost, inference speed, quality
Trade-off: Fewer tokens (efficiency) vs. Semantic units (quality)
Adoption: Every model has its own tokenizer (GPT, LLaMA, Mistral, etc.)

Why Tokenization Matters¶

The Problem: How Do Models See Text?¶

LLMs don't understand text directly. They need numbers!

The Tokenization Pipeline:

Input: Raw text └─ "The cat sat on the mat"
Tokenization: Convert to token IDs └─ [1, 450, 9206, 3641, 456, 262, 3488] └─ Result: Example token IDs
Embedding: Convert to vectors └─ [[0.1, -0.2, ...], [0.3, 0.5, ...], ...] └─ Result: Embeddings for each token
Processing: Transformer computation └─ Layers process the embedded vectors └─ Output: Next token prediction

💡 Key Insight: Quality of Step 1 (tokenization) affects everything!

Impact on Model Efficiency¶

Example: Tokenizing "The quick brown fox jumps over the lazy dog"

Approach 1: Character-level tokenization
"T" "h" "e" " " "q" "u" "i" "c" "k" " " "b" "r" "o" "w" "n" ...
Total tokens: ~43 (very inefficient!)

Problem:
  - Longer sequences (50 vs 10)
  - More computation needed
  - Slower training and inference
  - Higher cost (pay per token generated)

Approach 2: Word-level tokenization
"The" "quick" "brown" "fox" "jumps" "over" "the" "lazy" "dog"
Total tokens: 9 (efficient!)

Problem:
  - 50K vocab size needed (even simple text)
  - Rare words become [UNK] (unknown)
  - Model can't handle OOV (out-of-vocabulary)

Approach 3: Subword tokenization (Best)
"The" "quick" "brown" "fox" "jump" "s" "over" "the" "lazy" "dog"
Total tokens: 10 (balanced!)

Benefits:
  - 32K vocab size sufficient
  - Handles rare words well
  - No [UNK] tokens
  - Efficient and expressive

Token Efficiency Impact on Cost¶

Training/inference cost is per-token!

Model: 7B parameters, training on 1T tokens

Scenario 1: Inefficient tokenization (2 tokens per word average)
  - Total tokens: 2T
  - Compute cost: $200K
  - Training time: 40 days
  - Cost per token: $0.0001

Scenario 2: Efficient tokenization (1 token per word average)
  - Total tokens: 1T
  - Compute cost: $100K (50% savings!)
  - Training time: 20 days
  - Cost per token: $0.0001

Real example (GPT-3):
  - Text: "The quick brown fox..."
  - Tokens: 10 (reasonably efficient)
  - Same text with naive tokenization: 30+ tokens!
  - GPT-3 costs: Proportional to token count!

Lesson: Better tokenization = Less cost + Faster training

Tokenization Process¶

The Complete Pipeline¶

The Tokenization Process:

Normalization
Input: Raw text
Actions: Lowercase/uppercase conversion, remove accents, handle whitespace
Output: "the quick brown fox..."
Pre-tokenization
Input: Normalized text
Actions: Split on spaces/punctuation, initial word splitting
Output: ["the", "quick", "brown", "fox", "..."]
Subword Tokenization
Input: Words from pre-tokenization
Actions: Split rare words, apply BPE vocabulary
Output: ["the", "quick", "br", "own", "fox", "..."]
Token ID Mapping
Input: Subword tokens
Actions: Look up each token in vocabulary
Output: [271, 4518, 2271, 3491, 12, ...]
Special Tokens
Input: Token ID sequence
Actions: Insert BOS/EOS/UNK/PAD markers as needed
Output: [BOS, 271, 4518, 2271, 3491, 12, ..., EOS]
See: Special Tokens below

Vocabulary Size Trade-offs¶

Vocabulary Size Comparison:

Size        Language Coverage  Rare Words  Vocab Memory  Quality
────────────────────────────────────────────────────────────────
8K          Basic              Bad         Small         Poor
32K         English OK         Some OOV    ~256MB        Good
64K         English good       Rare        ~512MB        Very good
128K        Multilingual       Rare        ~1GB          Excellent
256K        Multilingual+      Very rare   ~2GB          Near-perfect

Typical choices:

GPT-2/3: 50K vocab
  - Balances efficiency and coverage
  - Good for English

LLaMA: 32K vocab
  - Smaller but sufficient
  - Efficient

Mistral: 32K vocab
  - Same as LLaMA
  - Compatible

Claude: ~100K vocab (speculated)
  - Multilingual support
  - Better rare word handling
  - Worth the extra size

Recommendation:
  - Single language: 32-50K
  - Multilingual: 64-128K
  - Maximum coverage: 128K+

Types of Tokenization¶

1. Character-level Tokenization¶

Split into individual characters

Text: "hello world"
Tokens: ['h', 'e', 'l', 'l', 'o', ' ', 'w', 'o', 'r', 'l', 'd']
Vocab size: ~128 (all ASCII characters)

Pros:
✅ Tiny vocabulary (no OOV)
✅ Can spell anything
✅ Simple to implement

Cons:
❌ Very long sequences (10x more tokens)
❌ Hard to learn (no word semantics)
❌ Inefficient
  - "hello" = 5 tokens instead of 1!

Use case:
  - Only for research/toy models
  - Not practical for real LLMs

2. Word-level Tokenization¶

Split into words (separated by spaces)

Text: "hello world, how are you?"
Tokens: ["hello", "world", ",", "how", "are", "you", "?"]
Vocab size: ~50K (need word for every word in language)

Pros:
✅ Shorter sequences (efficient)
✅ Aligns with words (semantic units)

Cons:
❌ Large vocabulary needed (50K minimum)
❌ Rare words → [UNK] token (lost information!)
❌ New words → [UNK] (can't handle typos)

Example problem:
Text: "I love GPT-3!"
Tokens: ["I", "love", "GPT-3", "!"]
Problem: If "GPT-3" not in vocab → ["I", "love", "[UNK]", "!"]
  - Model loses the meaning!

Use case:
  - Older models (pre-2018)
  - Not used in modern LLMs

3. Subword Tokenization (Modern Standard)¶

Split rare words into subwords, keep common words whole

Text: "hello world, how are you?"
Tokens: ["hello", "world", ",", "how", "are", "you", "?"]

Example with rare word:
Text: "extraordinary"
Approach 1 (word-level): [UNK] if not in vocab
Approach 2 (subword): ["extra", "ordinary"] or ["extra", "ord", "inary"]
  - Preserves information!

Vocab size: 32-64K (reasonable)
Sequence length: ~25% longer than word-level
Pros: Best of both worlds!

Variants:
  - BPE (Byte Pair Encoding) - GPT uses this
  - WordPiece - BERT uses this
  - SentencePiece - LLaMA/Mistral use this
  - Unigram LM - Other models

Vocabulary Distribution¶

Zipfian Distribution¶

In natural language, words follow Zipf's law:
"Frequency of word ∝ 1 / rank"

Example (English):
Rank 1 (most common): "the" - appears 7M times
Rank 2: "of" - appears 4M times
Rank 3: "and" - appears 2.7M times
...
Rank 1000: "ability" - appears 40K times
Rank 10000: "zombie" - appears 4K times

Implication:
  - Top 1K words: ~70% of corpus
  - Top 10K words: ~92% of corpus
  - Top 50K words: 99%+ of corpus
  - So 50K vocab is sufficient!

Token distribution (32K vocab, approximate):
Top 100 tokens: ~35% of all tokens
Top 1000 tokens: ~70% of all tokens
Remaining ~31K tokens: ~30% of all tokens

Lesson:
  - Small vocabulary handles most text
  - Rare words need subword splitting
  - 32-64K vocab is sweet spot

Practical Example: Building a Tokenizer¶

Using Hugging Face Tokenizers¶

from transformers import AutoTokenizer

# Load existing tokenizer
tokenizer = AutoTokenizer.from_pretrained("gpt2")

# Tokenize text
text = "The quick brown fox jumps over the lazy dog"
tokens = tokenizer.tokenize(text)
print(tokens)
# Output: ['The', 'Ġquick', 'Ġbrown', 'Ġfox', 'Ġjumps', 'Ġover', 'Ġthe', 'Ġlazy', 'Ġdog']
# Note: 'Ġ' represents space in BPE!

# Get token IDs
token_ids = tokenizer.encode(text)
print(token_ids)
# Output: [464, 2068, 6218, 21831, 20952, 625, 262, 16931, 3290]

# Decode back to text
decoded = tokenizer.decode(token_ids)
print(decoded)
# Output: "The quick brown fox jumps over the lazy dog"

# Vocabulary size
print(f"Vocab size: {tokenizer.vocab_size}")
# Output: Vocab size: 50257 (GPT-2)

# See vocabulary (get_vocab() returns a dict — convert before slicing)
print(list(tokenizer.get_vocab().items())[:10])
# Output: [('!', 0), ('"', 1), ('#', 2), ('$', 3), ('%', 4), ('&', 5), ("'", 6), ('(', 7), (')', 8), ('*', 9)]

Token Count Analysis¶

# Analyze tokenization efficiency
def analyze_tokenization(text, tokenizer):
    tokens = tokenizer.tokenize(text)
    token_ids = tokenizer.encode(text)

    words = text.split()

    avg_tokens_per_word = len(token_ids) / len(words)

    print(f"Text: '{text}'")
    print(f"Words: {len(words)}")
    print(f"Tokens: {len(token_ids)}")
    print(f"Tokens/Word: {avg_tokens_per_word:.2f}")
    print(f"Tokens: {tokens}")

    return avg_tokens_per_word

# Examples
texts = [
    "The cat sat on the mat",  # Common words
    "Anthropomorphization",     # Long uncommon word
    "GPT-4 is revolutionary",   # Modern tech terms
]

tokenizer = AutoTokenizer.from_pretrained("gpt2")

for text in texts:
    analyze_tokenization(text, tokenizer)

# Output might be:
# Text: 'The cat sat on the mat'
# Words: 6, Tokens: 6, Tokens/Word: 1.00
# Tokens: ['The', 'Ġcat', 'Ġsat', 'Ġon', 'Ġthe', 'Ġmat']
#
# Text: 'Anthropomorphization'
# Words: 1, Tokens: 4, Tokens/Word: 4.00
# Tokens: ['Anth', 'rop', 'omorph', 'ization']
#
# Text: 'GPT-4 is revolutionary'
# Words: 3, Tokens: 5, Tokens/Word: 1.67
# Tokens: ['GPT', '-', '4', 'Ġis', 'Ġrevolutionary']

Tokenization Challenges¶

Challenge 1: Multiple Languages¶

English text: "Hello world"
Tokens: ["Hello", " world"] - 2 tokens ✓

Chinese text: "你好世界" (hello world)
Tokens: ["你", "好", "世", "界"] - 4 tokens (inefficient!)

Problem:
  - Asian languages have no spaces
  - Character-level tokenization needed
  - 2x more tokens than English
  - Higher cost, slower inference

Solution: Multilingual tokenizers
  - Larger vocabulary (128K+)
  - Special handling for Asian languages
  - Better balance across languages
  - Example: SentencePiece

Challenge 2: Rare/New Words¶

Text: "I love COVID-19 and mRNA vaccines"

GPT-2 tokenizer:
Tokens: ["I", "Ġlove", "ĠCOVID", "-", "19", "ĠandĠm", "RNA", "Ġvaccines"]
Issues:
  - "COVID" split awkwardly
  - "mRNA" split as one token then "RNA" (inconsistent)
  - Not great decomposition

Better: SentencePiece trained on modern text
Tokens: ["I", "▁love", "▁COVID", "-", "19", "▁and", "▁m", "RNA", "▁vaccines"]
  - More consistent handling

Lesson: Tokenizer choice matters!

Challenge 3: Special Characters & Unicode¶

Emoji and special characters:

Text: "I ❤️ emoji 🎉"

Bad tokenizer: [UNK] tokens for emoji
Better tokenizer: Encodes emoji bytes
  - Can preserve meaning

Accented characters:

Text: "café naïve résumé"

Bad: Removes accents or [UNK]
Good: Preserves with subword tokens
  - Important for non-English!

Tokenization Impact on Quality¶

LLM Quality vs Tokenizer Efficiency¶

Hypothesis: More tokens = More information = Better quality?

Not necessarily! Empirical findings:

Model 1: 32K vocab, 1T tokens, sequence avg 1.2 tokens/word
  - Quality: 92% on benchmark
  - Training time: 20 days

Model 2: 64K vocab, 1T tokens, sequence avg 1.15 tokens/word
  - Quality: 92.3% on benchmark (+0.3%)
  - Training time: 20 days (+0% time!)
  - Note: Smaller sequences = same training time!

Model 3: 8K vocab, 1T tokens, sequence avg 2.5 tokens/word
  - Quality: 89% on benchmark (-3%)
  - Training time: 25 days (20% slower)

Conclusion:
  - Good tokenization (32-64K): Excellent quality
  - Bad tokenization (8K): Loss of quality and speed
  - Tiny tokenization (character): Very inefficient
  - Sweet spot: 32-64K subword tokens

Special Tokens¶

What They Are¶

Special tokens are reserved vocabulary entries that are not words — they encode structure: sequence boundaries, unknowns, padding, and formatting. They are fixed during training, must never be split into subwords, and are added by the tokenizer (not written by hand).

Common special tokens:

BOS   <s> / <|begin_of_text|>   Beginning of sequence
EOS   </s> / <|endoftext|>      End of sequence
UNK   [UNK] / <unk>             Unknown/invalid input
PAD   [PAD] / <pad>             Padding to equal batch lengths
CLS   [CLS]                     Classification marker (BERT)
SEP   [SEP]                     Sentence separator (BERT)
MASK  [MASK]                    Masked position (BERT pretraining)
USER  <|user|>                  Speaker role (chat models)
ASSISTANT <|assistant|>         Response role (chat models)

Why They Matter¶

1. Sequence boundaries (BOS/EOS)
   - "one" can mean "I want one" or "the number one" —
     a BOS/EOS helps the model know a sequence started/ended
   - generation stops when EOS is predicted!
   - GPT-2: <|endoftext|>  |  LLaMA: <s> and </s>

2. Unknown handling (UNK)
   - happens only if coverage < 100% (see character_coverage)
   - good subword tokenizers make [UNK] nearly extinct
     (byte-level BPE can encode ANY byte sequence)

3. Padding (PAD)
   - batches need equal-length rows
   - pad rows are masked in attention so they add no information
   - some models (LLaMA-2) have NO pad token — you must add one
     before fine-tuning, or the padding breaks

4. Chat formatting (modern LLMs)
   - role tokens structure multi-turn conversations:
     <|user|> ... <|assistant|> ...
   - the model learns "after <|assistant|>, answer as the assistant"

Special Tokens in Popular Tokenizers¶

Model        BOS          EOS             PAD          UNK        Other
GPT-2        —            <|endoftext|>   —            —          —
LLaMA 2      <s>          </s>            — (none!)    <unk>      —
LLaMA 3      <|begin_of_text|>  <|end_of_text|>  <|end_of_text|>  <|unk|>  <|reserved_special_token_...|>
BERT         [CLS]        [SEP]           [PAD]        [UNK]      [MASK]
Mistral      <s>          </s>            <pad>?       <unk>      —
GPT-4        —            <|endoftext|>   —            —          <|fim_|...> (code filling)

Important: token IDs are FIXED at vocab build time.
  - training the same tokenizer twice → different IDs
  - model weights are bound to a specific vocab ordering
  - never reorder/regenerate vocab after training!

Pitfall: PAD Token Missing¶

LLaMA-2's tokenizer has no <pad>:
  tokenizer.pad_token is None!

If you batch inputs with pad_to_multiple_of you get an error
unless you assign one:

  tokenizer.pad_token = tokenizer.eos_token
  # or add a brand-new <pad> token:
  #   tokenizer.add_special_tokens({'pad_token': '<pad>'})
  #   ⚠️ then resize model embeddings:
  #   model.resize_token_embeddings(len(tokenizer))

Key Takeaways¶

🔑 Tokenization: The hidden foundation (text → numbers)
📊 32-64K vocabulary: Sweet spot for efficiency and coverage
⚡ Subword tokenization: Modern standard (BPE, WordPiece, SentencePiece)
💰 Better tokenization: Fewer tokens = Lower cost
🌍 Multilingual: Requires larger vocab and special handling

01 Bpe (Byte Pair Encoding) - Most common tokenization method
02 Sentencepiece - Modern tokenizer (LLaMA, Mistral)
06 Wordpiece - BERT-style tokenizer (## continuation markers)
03 Token Efficiency & Compression - Reducing token count
05 Multilingual Tokenization - Challenges across languages
04 Tokenization Best Practices - Tips and pitfalls (incl. building your own)