Tokenization Best Practices: Tips, Tricks & Common Pitfalls

Overview

Best Practices for tokenization: practical tips to optimize tokenization for your use case and avoid common pitfalls.

• Measure: Baseline your efficiency
• Iterate: Test different tokenizers
• Optimize: Domain-specific when worthwhile
• Monitor: Track token count in production

---

Before You Tokenize: Questions to Ask

1. What's Your Use Case?

Question: What are you doing with the model?

If: Training
  - Token efficiency impacts training cost and time
  - Small improvement = significant savings
  - Consider custom tokenizer training

If: Inference (API/interactive)
  - Token count affects:
  - User cost (charged per token)
  - Latency (more tokens = longer)
  - Throughput (fewer tokens = more queries per GPU)
  - Efficiency matters a lot!

If: Deployment (internal)
  - Cost per token less critical
  - Latency more important
  - Standard tokenizer usually fine

If: Research
  - Compatibility > efficiency
  - Use standard tokenizer (GPT, BERT, etc.)
  - Easier to reproduce

2. Do You Have Domain-Specific Data?

Questions to answer:

Q: Is 80%+ of my text from specialized domain?
  - Yes: Consider domain-specific tokenizer
    - Medical, legal, code, finance
    - 10-15% token savings possible
    - Worth the effort
│
  - No: Standard tokenizer fine
  - Not enough domain-specific benefit

Q: How much data do I have for training?
  - >1M documents: Train custom
  - 100K-1M: Maybe train custom
  - <100K: Use pre-trained

Q: What's the cost of a 5% token increase?
  - <$100/day: Not worth optimizing
  - $100-1000/day: Consider optimization
  - >$1000/day: Definitely optimize!

---

Choosing a Tokenizer

Decision Tree

Start: Which tokenizer should I use?
  │
  - Using existing model?
    - Yes → Use model's tokenizer
    - (GPT-3 uses BPE, LLaMA uses SentencePiece, etc.)
  │  │
    - No → Continue below
  │
  - Multilingual support needed?
    - Yes → SentencePiece
    - (language-independent)
  │  │
    - No → Continue below
  │
  - English only?
    - Yes, generic → BPE (GPT-style)
    - Fast, standard
  │  │
    - Yes, custom domain → Train custom SentencePiece
  - 10-15% savings possible
  │
  - Other language
  - Asian languages → SentencePiece
  - (handles no-space languages)
     │
  - European → BPE or SentencePiece
                   (both work well)

Specific Recommendations

Scenario 1: Fine-tuning existing model
  - Use: Model's tokenizer
  - Why: Compatible
  - Cost: $0
  - Effort: Minimal

Scenario 2: New model for English
  - Use: Standard BPE (50K vocab)
  - Why: Simple, tested, efficient
  - Cost: $100-500
  - Effort: Low

Scenario 3: New model multilingual
  - Use: SentencePiece (64K vocab)
  - Why: Language-agnostic
  - Cost: $200-500
  - Effort: Low

Scenario 4: Specialized domain (medical, legal, code)
  - Use: Domain-specific SentencePiece
  - Why: 10-15% token savings
  - Cost: $1000-5000 (including data prep)
  - Effort: Medium

Scenario 5: Maximum efficiency needed
  - Use: Custom SentencePiece + compression
  - Why: 30-50% total token savings
  - Cost: $5000+
  - Effort: High

---

Training Your Own Tokenizer

Checklist

Step 1: Prepare training data
  - ✓ Collect representative text (1M+ documents recommended)
  - ✓ Clean: Remove corrupted, encoded incorrectly
  - ✓ Sample: Use subset for initial training
  - ✓ Size: Start with 1GB, scale up as needed
  - ✓ Format: Plain text file (one sentence per line)

Step 2: Choose tokenizer type
  - ✓ Decide: SentencePiece or BPE?
  - ✓ For multilingual or unknown: SentencePiece
  - ✓ For English only: Either (SentencePiece simpler)
  - ✓ Config: BPE or Unigram model type?

Step 3: Set vocabulary size
  - ✓ Single language: 32-50K
  - ✓ Multilingual: 64-128K
  - ✓ Try multiple: 32K, 50K, 64K
  - ✓ Measure efficiency: Compare token counts

Step 4: Train multiple variations
  - ✓ Different vocab sizes
  - ✓ Different data subsets
  - ✓ Different model types (BPE vs Unigram)
  - ✓ Test on validation set

Step 5: Evaluate
  - ✓ Measure: Tokens per word
  - ✓ Test: Encoding/decoding correctness
  - ✓ Analyze: Coverage of vocabulary
  - ✓ Compare: Efficiency vs baseline

Step 6: Choose best
  - ✓ Select: Highest efficiency
  - ✓ Validate: Check edge cases
  - ✓ Document: Save configuration
  - ✓ Deploy: Use in production

Example timeline:
Day 1: Data prep + initial training (2-4 tokenizers)
Day 2: Evaluation and analysis
Day 3: Final training and validation
  - Total: 3 days for custom tokenizer

Implementation

import sentencepiece as spm

# Step 1: Prepare data
with open('training_data.txt', 'w') as f:
    # Write all texts, one per line
    f.write("Text 1\n")
    f.write("Text 2\n")
    # ... millions of texts

# Step 2: Train multiple tokenizers
for vocab_size in [32000, 50000, 64000]:
    for model_type in ['bpe', 'unigram']:
        print(f"Training {model_type} with vocab {vocab_size}...")

        spm.SentencePieceTrainer.train(
            input='training_data.txt',
            model_prefix=f'tokenizer_{model_type}_{vocab_size}',
            vocab_size=vocab_size,
            model_type=model_type,
            character_coverage=0.9999,
        )

# Step 3: Evaluate
test_texts = [
    "Common text",
    "Rare word: pneumonoultramicroscopicsilicovolcanoconiosis",
    "Mixed English 和 Chinese",
    "Code: def function(): pass",
]

for model_file in ['tokenizer_bpe_32000.model', 'tokenizer_bpe_50000.model', ...]:
    sp = spm.SentencePieceProcessor(model_file=model_file)

    total_tokens = 0
    total_words = 0

    for text in test_texts:
        tokens = sp.encode(text)
        words = text.split()

        total_tokens += len(tokens)
        total_words += len(words)

    efficiency = total_tokens / total_words
    print(f"{model_file}: {efficiency:.2f} tokens/word")

# Step 4: Choose best
print("Best tokenizer: tokenizer_bpe_50000 (1.15 tokens/word)")

# Step 5: Save for deployment
best_tokenizer = spm.SentencePieceProcessor(model_file='tokenizer_bpe_50000.model')
print(best_tokenizer.model_file)  # path of the model — use this file in production

---

Common Pitfalls & Solutions

Pitfall 1: Incompatible Tokenizer

Problem: Fine-tuning model with different tokenizer

Example:
  - Pre-trained: GPT-3 with 50K BPE vocabulary
  - Fine-tune: With custom 32K SentencePiece
  - Problem: Token IDs don't match!
  - Result: Training fails or produces gibberish

Solution:
  - Keep same tokenizer as base model
  - Only change vocab if retraining from scratch

Pitfall 2: Poor Character Coverage

Problem: Rare characters become [UNK] token

Example: - Tokenizer trained on English - Process Chinese text: "你好" - Output: "[UNK] [UNK]" (lost information!)

Solution: - Train on diverse data (all languages you need) - Set character_coverage=0.9999 - Verify coverage on test set

Implementation:

# Verify coverage
sp = spm.SentencePieceProcessor(model_file='tokenizer.model')

test_text = "你好世界"  # Chinese
tokens = sp.encode_as_pieces(test_text)

if any('[UNK]' in t for t in tokens):
    print("ERROR: Untrained characters detected!")
    print(f"Text: {test_text}")
    print(f"Tokens: {tokens}")
else:
    print("OK: All characters handled")

Pitfall 3: Overfitting to Training Data

Problem: Tokenizer overfits to training distribution

Example: - Tokenizer trained on Wikipedia - Deploy on social media text - Efficiency drops: More tokens than expected - Reason: Social media has different vocabulary

Solution: - Train on representative data - If multi-domain: Mix domains in training - Test on validation set from different source

Validation strategy:

# Train on domain A
spm.SentencePieceTrainer.train(
    input='domain_a_data.txt',
    model_prefix='domain_a',
    vocab_size=32000,
)

# Test on domains A, B, C
test_files = ['test_a.txt', 'test_b.txt', 'test_c.txt']
for domain, test_file in zip('ABC', test_files):
    efficiency = evaluate_efficiency(test_file)
    print(f"Efficiency on {domain}: {efficiency:.2f}")

# Ideally: Similar efficiency across domains

Pitfall 4: Vocabulary Size Mismatch

Problem: Vocab size too small or too large

Too small (8K): - Many words split (inefficient) - Tokens/word: 1.5-2.0 (bad!) - But: Smaller embedding layer

Too large (256K): - Few words split (efficient) - Tokens/word: 1.05 (good!) - But: Massive embedding layer (wasted memory)

Solution: Test multiple sizes - 32K: Safe middle ground - 50K: Good for English - 64K: Good for multilingual - Measure efficiency, don't guess!

Cost analysis:

# Compare vocab sizes vs model memory

vocab_sizes = [32000, 50000, 64000]
hidden_dim = 4096

for vocab_size in vocab_sizes:
    embedding_memory = vocab_size * hidden_dim * 4 / 1e9  # GB (float32)
    print(f"Vocab {vocab_size}: {embedding_memory:.2f}GB embedding")

# Trade-off: Larger vocab = larger embedding
# But: Fewer tokens = faster inference
# Break-even usually around 50K for most uses

---

Monitoring in Production

Metrics to Track

Metric 1: Average tokens per query
  - Track: Daily average
  - Alert: If increases >5% (possible data shift)
  - Action: Investigate, possibly retrain

Metric 2: Unknown tokens [UNK] rate
  - Track: Percentage of tokens that are [UNK]
  - Target: <0.01% (rare)
  - Alert: If increases (possible data shift)

Metric 3: Tokenization latency
  - Track: Average encoding time per query
  - Target: <10ms
  - Alert: If increases (performance issue)

Metric 4: Vocabulary coverage
  - Track: Quarterly check
  - Verify: New domains handled well
  - Action: Retrain if needed

Implementation:

import time
from collections import defaultdict

class TokenizationMonitor:
    def __init__(self, tokenizer):
        self.tokenizer = tokenizer
        self.stats = defaultdict(list)
        self.unk_count = 0  # plain integer counter (not a list!)

    def encode(self, text):
        start_time = time.time()
        tokens = self.tokenizer.encode(text)
        latency = time.time() - start_time

        # Track metrics
        self.stats['tokens'].append(len(tokens))
        self.stats['latency'].append(latency)

        # Check for [UNK] — the UNK ID differs per tokenizer!
        unk_id = getattr(self.tokenizer, 'unk_token_id', None)
        if unk_id is not None and unk_id in tokens:
            self.unk_count += 1

        return tokens

    def get_stats(self):
        n = len(self.stats['tokens'])
        avg_tokens = sum(self.stats['tokens']) / n
        avg_latency = sum(self.stats['latency']) / n
        unk_rate = self.unk_count / n

        return {
            'avg_tokens': avg_tokens,
            'avg_latency_ms': avg_latency * 1000,
            'unk_rate': unk_rate,
        }

# Use in production
monitor = TokenizationMonitor(tokenizer)
tokens = monitor.encode(user_query)
stats = monitor.get_stats()
log_metrics(stats)  # Send to monitoring system

---

Key Takeaways

📏 Measure first: Baseline efficiency before optimizing
🎯 Choose tokenizer: Match use case (standard vs domain-specific)
🔄 Test multiple: Don't rely on single tokenizer
📊 Monitor production: Track tokens/query over time
💡 Domain tokenizer: 10-15% savings for specialized domains

---

Related Notes in Tokenization Subdirectory

• 00 Tokenization Fundamentals - Overview (incl. special tokens)
• 01 Bpe (Byte Pair Encoding) - Standard approach
• 02 Sentencepiece - Modern alternative
• 06 Wordpiece - BERT-style variant
• 03 Token Efficiency & Compression - Optimization techniques
• 05 Multilingual Tokenization - Multilingual challenges