00 tokenization fundamentals

Overview

Tokenization converts raw text into discrete tokens (words, subwords, characters) that models can process. Fundamental to every LLM but often overlooked. Quality tokenization directly impacts model efficiency, quality, and cost.

• Purpose: Text → Tokens → Embeddings → Model
• Impact: Affects model capacity, training cost, inference speed, quality
• Trade-off: Fewer tokens (efficiency) vs. Semantic units (quality)
• Adoption: Every model has its own tokenizer (GPT, LLaMA, Mistral, etc.)

-

Why Tokenization Matters

The Problem: How Do Models See Text?

LLMs don't understand text directly. They need numbers!

The Tokenization Pipeline:

1. Input: Raw text └─ "The cat sat on the mat"

2. Tokenization: Convert to token IDs └─ [1, 450, 9206, 3641, 456, 262, 3488] └─ Result: Example token IDs

3. Embedding: Convert to vectors └─ [[0.1, -0.2,...], [0.3, 0.5,...],...] └─ Result: Embeddings for each token

4. Processing: Transformer computation └─ Layers process the embedded vectors └─ Output: Next token prediction

Key Insight: Quality of Step 1 (tokenization) affects everything!

Impact on Model Efficiency

Example: Tokenizing "The quick brown fox jumps over the lazy dog"

Approach 1: Character-level tokenization
"T" "h" "e" " " "q" "u" "i" "c" "k" " " "b" "r" "o" "w" "n"...
Total tokens: ~43 (very inefficient!)

Problem:
 - Longer sequences (50 vs 10)
 - More computation needed
 - Slower training and inference
 - Higher cost (pay per token generated)

Approach 2: Word-level tokenization
"The" "quick" "brown" "fox" "jumps" "over" "the" "lazy" "dog"
Total tokens: 9 (efficient!)

Problem:
 - 50K vocab size needed (even simple text)
 - Rare words become [UNK] (unknown)
 - Model can't handle OOV (out-of-vocabulary)

Approach 3: Subword tokenization (Best)
"The" "quick" "brown" "fox" "jump" "s" "over" "the" "lazy" "dog"
Total tokens: 10 (balanced!)

Benefits:
 - 32K vocab size sufficient
 - Handles rare words well
 - No [UNK] tokens
 - Efficient and expressive

Token Efficiency Impact on Cost

Training/inference cost is per-token!

Model: 7B parameters, training on 1T tokens

Scenario 1: Inefficient tokenization (2 tokens per word average)
 - Total tokens: 2T
 - Compute cost: $200K
 - Training time: 40 days
 - Cost per token: $0.0001

Scenario 2: Efficient tokenization (1 token per word average)
 - Total tokens: 1T
 - Compute cost: $100K (50% savings!)
 - Training time: 20 days
 - Cost per token: $0.0001

Real example (GPT-3):
 - Text: "The quick brown fox..."
 - Tokens: 10 (reasonably efficient)
 - Same text with naive tokenization: 30+ tokens!
 - GPT-3 costs: Proportional to token count!

Lesson: Better tokenization = Less cost + Faster training

-

Tokenization Process

The Complete Pipeline

The Tokenization Process:

1. Normalization

2. Input: Raw text

3. Actions: Lowercase/uppercase conversion, remove accents, handle whitespace

4. Output: "the quick brown fox..."

5. Pre-tokenization

6. Input: Normalized text

7. Actions: Split on spaces/punctuation, initial word splitting

8. Output: ["the", "quick", "brown", "fox", "..."]

9. Subword Tokenization

10. Input: Words from pre-tokenization

11. Actions: Split rare words, apply BPE vocabulary

12. Output: ["the", "quick", "br", "own", "fox", "..."]

13. Token ID Mapping

14. Input: Subword tokens

15. Actions: Look up each token in vocabulary

16. Output: [271, 4518, 2271, 3491, 12,...]

17. Special Tokens

18. Input: Token ID sequence

19. Actions: Insert BOS/EOS/UNK/PAD markers as needed
20. Output: [BOS, 271, 4518, 2271, 3491, 12,..., EOS]
21. See: Special Tokens below

Vocabulary Size Trade-offs

Vocabulary Size Comparison:

Size Language Coverage Rare Words Vocab Memory Quality
────────────────────────────────────────────────────────────────
8K Basic Bad Small Poor
32K English OK Some OOV ~256MB Good
64K English good Rare ~512MB Very good
128K Multilingual Rare ~1GB Excellent
256K Multilingual+ Very rare ~2GB Near-perfect

Typical choices:

GPT-2/3: 50K vocab
 - Balances efficiency and coverage
 - Good for English

LLaMA: 32K vocab
 - Smaller but sufficient
 - Efficient

Mistral: 32K vocab
 - Same as LLaMA
 - Compatible

Claude: ~100K vocab (speculated)
 - Multilingual support
 - Better rare word handling
 - Worth the extra size

Recommendation:
 - Single language: 32-50K
 - Multilingual: 64-128K
 - Maximum coverage: 128K+

-

Types of Tokenization

1. Character-level Tokenization

Split into individual characters

Text: "hello world"
Tokens: ['h', 'e', 'l', 'l', 'o', ' ', 'w', 'o', 'r', 'l', 'd']
Vocab size: ~128 (all ASCII characters)

Pros:
Tiny vocabulary (no OOV)
Can spell anything
Simple to implement

Cons:
Very long sequences (10x more tokens)
Hard to learn (no word semantics)
Inefficient
 - "hello" = 5 tokens instead of 1!

Use case:
 - Only for research/toy models
 - Not practical for real LLMs

2. Word-level Tokenization

Split into words (separated by spaces)

Text: "hello world, how are you?"
Tokens: ["hello", "world", ",", "how", "are", "you", "?"]
Vocab size: ~50K (need word for every word in language)

Pros:
Shorter sequences (efficient)
Aligns with words (semantic units)

Cons:
Large vocabulary needed (50K minimum)
Rare words → [UNK] token (lost information!)
New words → [UNK] (can't handle typos)

Example problem:
Text: "I love GPT-3!"
Tokens: ["I", "love", "GPT-3", "!"]
Problem: If "GPT-3" not in vocab → ["I", "love", "[UNK]", "!"]
 - Model loses the meaning!

Use case:
 - Older models (pre-2018)
 - Not used in modern LLMs

3. Subword Tokenization (Modern Standard)

Split rare words into subwords, keep common words whole

Text: "hello world, how are you?"
Tokens: ["hello", "world", ",", "how", "are", "you", "?"]

Example with rare word:
Text: "extraordinary"
Approach 1 (word-level): [UNK] if not in vocab
Approach 2 (subword): ["extra", "ordinary"] or ["extra", "ord", "inary"]
 - Preserves information!

Vocab size: 32-64K (reasonable)
Sequence length: ~25% longer than word-level
Pros: Best of both worlds!

Variants:
 - BPE (Byte Pair Encoding) - GPT uses this
 - WordPiece - BERT uses this
 - SentencePiece - LLaMA/Mistral use this
 - Unigram LM - Other models

-

Vocabulary Distribution

Zipfian Distribution

In natural language, words follow Zipf's law:
"Frequency of word ∝ 1 / rank"

Example (English):
Rank 1 (most common): "the" - appears 7M times
Rank 2: "of" - appears 4M times
Rank 3: "and" - appears 2.7M times
...
Rank 1000: "ability" - appears 40K times
Rank 10000: "zombie" - appears 4K times

Implication:
 - Top 1K words: ~70% of corpus
 - Top 10K words: ~92% of corpus
 - Top 50K words: 99%+ of corpus
 - So 50K vocab is sufficient!

Token distribution (32K vocab, approximate):
Top 100 tokens: ~35% of all tokens
Top 1000 tokens: ~70% of all tokens
Remaining ~31K tokens: ~30% of all tokens

Lesson:
 - Small vocabulary handles most text
 - Rare words need subword splitting
 - 32-64K vocab is sweet spot

-

Practical Example: Building a Tokenizer

Using Hugging Face Tokenizers

from transformers import AutoTokenizer

# Load existing tokenizer
tokenizer = AutoTokenizer.from_pretrained("gpt2")

# Tokenize text
text = "The quick brown fox jumps over the lazy dog"
tokens = tokenizer.tokenize(text)
print(tokens)
# Output
# Note

# Get token IDs
token_ids = tokenizer.encode(text)
print(token_ids)
# Output

# Decode back to text
decoded = tokenizer.decode(token_ids)
print(decoded)
# Output

# Vocabulary size
print(f"Vocab size: {tokenizer.vocab_size}")
# Output

# See vocabulary (get_vocab() returns a dict — convert before slicing)
print(list(tokenizer.get_vocab().items())[:10])
# Output

Token Count Analysis

# Analyze tokenization efficiency
def analyze_tokenization(text, tokenizer):
 tokens = tokenizer.tokenize(text)
 token_ids = tokenizer.encode(text)

 words = text.split()

 avg_tokens_per_word = len(token_ids) / len(words)

 print(f"Text: '{text}'")
 print(f"Words: {len(words)}")
 print(f"Tokens: {len(token_ids)}")
 print(f"Tokens/Word: {avg_tokens_per_word:.2f}")
 print(f"Tokens: {tokens}")

 return avg_tokens_per_word

# Examples
texts = [
 "The cat sat on the mat", # Common words
 "Anthropomorphization", # Long uncommon word
 "GPT-4 is revolutionary", # Modern tech terms
]

tokenizer = AutoTokenizer.from_pretrained("gpt2")

for text in texts:
 analyze_tokenization(text, tokenizer)

# Output might be:
# Text
# Words
# Tokens
#
# Text
# Words
# Tokens
#
# Text
# Words
# Tokens

-

Tokenization Challenges

Challenge 1: Multiple Languages

English text: "Hello world"
Tokens: ["Hello", " world"] - 2 tokens 

Chinese text: "你好世界" (hello world)
Tokens: ["你", "好", "世", "界"] - 4 tokens (inefficient!)

Problem:
 - Asian languages have no spaces
 - Character-level tokenization needed
 - 2x more tokens than English
 - Higher cost, slower inference

Solution: Multilingual tokenizers
 - Larger vocabulary (128K+)
 - Special handling for Asian languages
 - Better balance across languages
 - Example: SentencePiece

Challenge 2: Rare/New Words

Text: "I love COVID-19 and mRNA vaccines"

GPT-2 tokenizer:
Tokens: ["I", "Ġlove", "ĠCOVID", "-", "19", "ĠandĠm", "RNA", "Ġvaccines"]
Issues:
 - "COVID" split awkwardly
 - "mRNA" split as one token then "RNA" (inconsistent)
 - Not great decomposition

Better: SentencePiece trained on modern text
Tokens: ["I", "▁love", "▁COVID", "-", "19", "▁and", "▁m", "RNA", "▁vaccines"]
 - More consistent handling

Lesson: Tokenizer choice matters!

Challenge 3: Special Characters & Unicode

Emoji and special characters:

Text: "I ❤ emoji 🎉"

Bad tokenizer: [UNK] tokens for emoji
Better tokenizer: Encodes emoji bytes
 - Can preserve meaning

Accented characters:

Text: "café naïve résumé"

Bad: Removes accents or [UNK]
Good: Preserves with subword tokens
 - Important for non-English!

-

Tokenization Impact on Quality

LLM Quality vs Tokenizer Efficiency

Hypothesis: More tokens = More information = Better quality?

Not necessarily! Empirical findings:

Model 1: 32K vocab, 1T tokens, sequence avg 1.2 tokens/word
 - Quality: 92% on benchmark
 - Training time: 20 days

Model 2: 64K vocab, 1T tokens, sequence avg 1.15 tokens/word
 - Quality: 92.3% on benchmark (+0.3%)
 - Training time: 20 days (+0% time!)
 - Note: Smaller sequences = same training time!

Model 3: 8K vocab, 1T tokens, sequence avg 2.5 tokens/word
 - Quality: 89% on benchmark (-3%)
 - Training time: 25 days (20% slower)

Conclusion:
 - Good tokenization (32-64K): Excellent quality
 - Bad tokenization (8K): Loss of quality and speed
 - Tiny tokenization (character): Very inefficient
 - Sweet spot: 32-64K subword tokens

-

Special Tokens

What They Are

Special tokens are reserved vocabulary entries that are not words— they encode structure: sequence boundaries, unknowns, padding, and formatting. They are fixed during training, must never be split into subwords, and are added by the tokenizer (not written by hand).

Common special tokens:

BOS <s> / <|begin_of_text|> Beginning of sequence
EOS </s> / <|endoftext|> End of sequence
UNK [UNK] / <unk> Unknown/invalid input
PAD [PAD] / <pad> Padding to equal batch lengths
CLS [CLS] Classification marker (BERT)
SEP [SEP] Sentence separator (BERT)
MASK [MASK] Masked position (BERT pretraining)
USER <|user|> Speaker role (chat models)
ASSISTANT <|assistant|> Response role (chat models)

Why They Matter

1. Sequence boundaries (BOS/EOS)
 - "one" can mean "I want one" or "the number one" —
 a BOS/EOS helps the model know a sequence started/ended
 - generation stops when EOS is predicted!
 - GPT-2: <|endoftext|>| LLaMA: <s> and </s>

2. Unknown handling (UNK)
 - happens only if coverage < 100% (see character_coverage)
 - good subword tokenizers make [UNK] nearly extinct
 (byte-level BPE can encode ANY byte sequence)

3. Padding (PAD)
 - batches need equal-length rows
 - pad rows are masked in attention so they add no information
 - some models (LLaMA-2) have NO pad token — you must add one
 before fine-tuning, or the padding breaks

4. Chat formatting (modern LLMs)
 - role tokens structure multi-turn conversations:
 <|user|>... <|assistant|>...
 - the model learns "after <|assistant|>, answer as the assistant"

Special Tokens in Popular Tokenizers

Model BOS EOS PAD UNK Other
GPT-2 — <|endoftext|> — — —
LLaMA 2 <s> </s> — (none!) <unk> —
LLaMA 3 <|begin_of_text|> <|end_of_text|> <|end_of_text|> <|unk|> <|reserved_special_token_...|>
BERT [CLS] [SEP] [PAD] [UNK] [MASK]
Mistral <s> </s> <pad>? <unk> —
GPT-4 — <|endoftext|> — — <|fim_|...> (code filling)

Important: token IDs are FIXED at vocab build time.
 - training the same tokenizer twice → different IDs
 - model weights are bound to a specific vocab ordering
 - never reorder/regenerate vocab after training!

Pitfall: PAD Token Missing

LLaMA-2's tokenizer has no <pad>:
 tokenizer.pad_token is None!

If you batch inputs with pad_to_multiple_of you get an error
unless you assign one:

 tokenizer.pad_token = tokenizer.eos_token
 # or add a brand-new <pad> token:
 # tokenizer.add_special_tokens({'pad_token': '<pad>'})
 # then resize model embeddings:
 # model.resize_token_embeddings(len(tokenizer))

---

Key Takeaways

Tokenization: The hidden foundation (text → numbers) 32-64K vocabulary: Sweet spot for efficiency and coverage Subword tokenization: Modern standard (BPE, WordPiece, SentencePiece) Better tokenization: Fewer tokens = Lower cost Multilingual: Requires larger vocab and special handling

-

Related Notes in Tokenization Subdirectory

• 01 Bpe (Byte Pair Encoding) - Most common tokenization method
• 02 Sentencepiece - Modern tokenizer (LLaMA, Mistral)
• 06 Wordpiece - BERT-style tokenizer (## continuation markers)
• [03 Token Efficiency & Compression](/01-modeling/00-fundamentals/00-tokenization/(03-token-efficiency-compression/) - Reducing token count
• 05 Multilingual Tokenization - Challenges across languages
• 04 Tokenization Best Practices - Tips and pitfalls (incl. building your own)