Skip to content

Tokenization Best Practices

Overview

Best Practices for tokenization: practical tips to optimize tokenization for your use case and avoid common pitfalls.

  • Measure: Baseline your efficiency
  • Iterate: Test different tokenizers
  • Optimize: Domain-specific when worthwhile
  • Monitor: Track token count in production

-

Before You Tokenize: Questions to Ask

1. What's Your Use Case?

Question: What are you doing with the model?

If: Training
 - Token efficiency impacts training cost and time
 - Small improvement = significant savings
 - Consider custom tokenizer training

If: Inference (API/interactive)
 - Token count affects:
 - User cost (charged per token)
 - Latency (more tokens = longer)
 - Throughput (fewer tokens = more queries per GPU)
 - Efficiency matters a lot!

If: Deployment (internal)
 - Cost per token less critical
 - Latency more important
 - Standard tokenizer usually fine

If: Research
 - Compatibility > efficiency
 - Use standard tokenizer (GPT, BERT, etc.)
 - Easier to reproduce

2. Do You Have Domain-Specific Data?

Questions to answer:

Q: Is 80%+ of my text from specialized domain?
 - Yes: Consider domain-specific tokenizer
 - Medical, legal, code, finance
 - 10-15% token savings possible
 - Worth the effort
│
 - No: Standard tokenizer fine
 - Not enough domain-specific benefit

Q: How much data do I have for training?
 - >1M documents: Train custom
 - 100K-1M: Maybe train custom
 - <100K: Use pre-trained

Q: What's the cost of a 5% token increase?
 - <$100/day: Not worth optimizing
 - $100-1000/day: Consider optimization
 - >$1000/day: Definitely optimize!

Choosing a Tokenizer

Decision Tree

Start: Which tokenizer should I use?
 │
 - Using existing model?
 - Yes → Use model's tokenizer
 - (GPT-3 uses BPE, LLaMA uses SentencePiece, etc.)
 │ │
 - No → Continue below
 │
 - Multilingual support needed?
 - Yes → SentencePiece
 - (language-independent)
 │ │
 - No → Continue below
 │
 - English only?
 - Yes, generic → BPE (GPT-style)
 - Fast, standard
 │ │
 - Yes, custom domain → Train custom SentencePiece
 - 10-15% savings possible
 │
 - Other language
 - Asian languages → SentencePiece
 - (handles no-space languages)
 │
 - European → BPE or SentencePiece
 (both work well)

Specific Recommendations

Scenario 1: Fine-tuning existing model
 - Use: Model's tokenizer
 - Why: Compatible
 - Cost: $0
 - Effort: Minimal

Scenario 2: New model for English
 - Use: Standard BPE (50K vocab)
 - Why: Simple, tested, efficient
 - Cost: $100-500
 - Effort: Low

Scenario 3: New model multilingual
 - Use: SentencePiece (64K vocab)
 - Why: Language-agnostic
 - Cost: $200-500
 - Effort: Low

Scenario 4: Specialized domain (medical, legal, code)
 - Use: Domain-specific SentencePiece
 - Why: 10-15% token savings
 - Cost: $1000-5000 (including data prep)
 - Effort: Medium

Scenario 5: Maximum efficiency needed
 - Use: Custom SentencePiece + compression
 - Why: 30-50% total token savings
 - Cost: $5000+
 - Effort: High

-

Training Your Own Tokenizer

Checklist

Step 1: Prepare training data
 - Collect representative text (1M+ documents recommended)
 - Clean: Remove corrupted, encoded incorrectly
 - Sample: Use subset for initial training
 - Size: Start with 1GB, scale up as needed
 - Format: Plain text file (one sentence per line)

Step 2: Choose tokenizer type
 - Decide: SentencePiece or BPE?
 - For multilingual or unknown: SentencePiece
 - For English only: Either (SentencePiece simpler)
 - Config: BPE or Unigram model type?

Step 3: Set vocabulary size
 - Single language: 32-50K
 - Multilingual: 64-128K
 - Try multiple: 32K, 50K, 64K
 - Measure efficiency: Compare token counts

Step 4: Train multiple variations
 - Different vocab sizes
 - Different data subsets
 - Different model types (BPE vs Unigram)
 - Test on validation set

Step 5: Evaluate
 - Measure: Tokens per word
 - Test: Encoding/decoding correctness
 - Analyze: Coverage of vocabulary
 - Compare: Efficiency vs baseline

Step 6: Choose best
 - Select: Highest efficiency
 - Validate: Check edge cases
 - Document: Save configuration
 - Deploy: Use in production

Example timeline:
Day 1: Data prep + initial training (2-4 tokenizers)
Day 2: Evaluation and analysis
Day 3: Final training and validation
 - Total: 3 days for custom tokenizer

Implementation

import sentencepiece as spm

# Step 1
with open('training_data.txt', 'w') as f:
 # Write all texts, one per line
 f.write("Text 1\n")
 f.write("Text 2\n")
 #... millions of texts

# Step 2
for vocab_size in [32000, 50000, 64000]:
 for model_type in ['bpe', 'unigram']:
 print(f"Training {model_type} with vocab {vocab_size}...")

 spm.SentencePieceTrainer.train(
 input='training_data.txt',
 model_prefix=f'tokenizer_{model_type}_{vocab_size}',
 vocab_size=vocab_size,
 model_type=model_type,
 character_coverage=0.9999,
)

# Step 3
test_texts = [
 "Common text",
 "Rare word: pneumonoultramicroscopicsilicovolcanoconiosis",
 "Mixed English 和 Chinese",
 "Code: def function(): pass",
]

for model_file in ['tokenizer_bpe_32000.model', 'tokenizer_bpe_50000.model',...]:
 sp = spm.SentencePieceProcessor(model_file=model_file)

 total_tokens = 0
 total_words = 0

 for text in test_texts:
 tokens = sp.encode(text)
 words = text.split()

 total_tokens += len(tokens)
 total_words += len(words)

 efficiency = total_tokens / total_words
 print(f"{model_file}: {efficiency:.2f} tokens/word")

# Step 4
print("Best tokenizer: tokenizer_bpe_50000 (1.15 tokens/word)")

# Step 5
best_tokenizer = spm.SentencePieceProcessor(model_file='tokenizer_bpe_50000.model')
print(best_tokenizer.model_file) # path of the model — use this file in production

-

Common Pitfalls & Solutions

Pitfall 1: Incompatible Tokenizer

Problem: Fine-tuning model with different tokenizer

Example:
 - Pre-trained: GPT-3 with 50K BPE vocabulary
 - Fine-tune: With custom 32K SentencePiece
 - Problem: Token IDs don't match!
 - Result: Training fails or produces gibberish

Solution:
 - Keep same tokenizer as base model
 - Only change vocab if retraining from scratch

Pitfall 2: Poor Character Coverage

Problem: Rare characters become [UNK] token

Example:

  • Tokenizer trained on English
  • Process Chinese text: "你好"
  • Output: "[UNK] [UNK]" (lost information!)

Solution:

  • Train on diverse data (all languages you need)
  • Set character_coverage=0.9999
  • Verify coverage on test set

Implementation:

# Verify coverage
sp = spm.SentencePieceProcessor(model_file='tokenizer.model')

test_text = "你好世界" # Chinese
tokens = sp.encode_as_pieces(test_text)

if any('[UNK]' in t for t in tokens):
 print("ERROR: Untrained characters detected!")
 print(f"Text: {test_text}")
 print(f"Tokens: {tokens}")
else:
 print("OK: All characters handled")

Pitfall 3: Overfitting to Training Data

Problem: Tokenizer overfits to training distribution

Example:

  • Tokenizer trained on Wikipedia
  • Deploy on social media text
  • Efficiency drops: More tokens than expected
  • Reason: Social media has different vocabulary

Solution:

  • Train on representative data
  • If multi-domain: Mix domains in training
  • Test on validation set from different source

Validation strategy:

# Train on domain A
spm.SentencePieceTrainer.train(
 input='domain_a_data.txt',
 model_prefix='domain_a',
 vocab_size=32000,
)

# Test on domains A, B, C
test_files = ['test_a.txt', 'test_b.txt', 'test_c.txt']
for domain, test_file in zip('ABC', test_files):
 efficiency = evaluate_efficiency(test_file)
 print(f"Efficiency on {domain}: {efficiency:.2f}")

# Ideally

Pitfall 4: Vocabulary Size Mismatch

Problem: Vocab size too small or too large

Too small (8K):

  • Many words split (inefficient)
  • Tokens/word: 1.5-2.0 (bad!)
  • But: Smaller embedding layer

Too large (256K):

  • Few words split (efficient)
  • Tokens/word: 1.05 (good!)
  • But: Massive embedding layer (wasted memory)

Solution: Test multiple sizes

  • 32K: Safe middle ground
  • 50K: Good for English
  • 64K: Good for multilingual
  • Measure efficiency, don't guess!

Cost analysis:

# Compare vocab sizes vs model memory

vocab_sizes = [32000, 50000, 64000]
hidden_dim = 4096

for vocab_size in vocab_sizes:
 embedding_memory = vocab_size * hidden_dim * 4 / 1e9 # GB (float32)
 print(f"Vocab {vocab_size}: {embedding_memory:.2f}GB embedding")

# Trade-off
# But
# Break-even usually around 50K for most uses

-

Monitoring in Production

Metrics to Track

Metric 1: Average tokens per query
 - Track: Daily average
 - Alert: If increases >5% (possible data shift)
 - Action: Investigate, possibly retrain

Metric 2: Unknown tokens [UNK] rate
 - Track: Percentage of tokens that are [UNK]
 - Target: <0.01% (rare)
 - Alert: If increases (possible data shift)

Metric 3: Tokenization latency
 - Track: Average encoding time per query
 - Target: <10ms
 - Alert: If increases (performance issue)

Metric 4: Vocabulary coverage
 - Track: Quarterly check
 - Verify: New domains handled well
 - Action: Retrain if needed

Implementation:

import time
from collections import defaultdict

class TokenizationMonitor:
 def __init__(self, tokenizer):
 self.tokenizer = tokenizer
 self.stats = defaultdict(list)
 self.unk_count = 0 # plain integer counter (not a list!)

 def encode(self, text):
 start_time = time.time()
 tokens = self.tokenizer.encode(text)
 latency = time.time() - start_time

 # Track metrics
 self.stats['tokens'].append(len(tokens))
 self.stats['latency'].append(latency)

 # Check for [UNK] — the UNK ID differs per tokenizer!
 unk_id = getattr(self.tokenizer, 'unk_token_id', None)
 if unk_id is not None and unk_id in tokens:
 self.unk_count += 1

 return tokens

 def get_stats(self):
 n = len(self.stats['tokens'])
 avg_tokens = sum(self.stats['tokens']) / n
 avg_latency = sum(self.stats['latency']) / n
 unk_rate = self.unk_count / n

 return {
 'avg_tokens': avg_tokens,
 'avg_latency_ms': avg_latency * 1000,
 'unk_rate': unk_rate,
 }

# Use in production
monitor = TokenizationMonitor(tokenizer)
tokens = monitor.encode(user_query)
stats = monitor.get_stats()
log_metrics(stats) # Send to monitoring system

-

Key Takeaways

Measure first: Baseline efficiency before optimizing Choose tokenizer: Match use case (standard vs domain-specific) Test multiple: Don't rely on single tokenizer Monitor production: Track tokens/query over time Domain tokenizer: 10-15% savings for specialized domains

-