Skip to content

SentencePiece: Language-Agnostic Tokenization

Overview

SentencePiece is a language-independent tokenization library that trains directly on raw text (no pre-tokenization). Used by LLaMA, Mistral, and many modern models. Simpler and more language-agnostic than BPE variants.

  • Paper: "SentencePiece: A simple and language independent subword tokenizer and detokenizer for NMT" (Kudo & Richardson, 2018)
  • Adoption: LLaMA, Mistral, Qwen, Falcon, many modern LLMs
  • Key Feature: Treats space as special token, works for any language
  • Advantage: Language-agnostic, no pre-tokenization needed

How SentencePiece Works

Core Idea: Space as Token

Traditional BPE:
  - Pre-tokenize: "hello world" → ["hello", "world"]
  - Then apply BPE on tokens
  - Problem: Pre-tokenization is language-specific!

SentencePiece approach:
  - Treat space as special character
  - "hello world" → "hello▁world"
  - (▁ represents space/underscore)
  - Apply BPE/Unigram on raw text
  - Result: Language-independent!

Example encoding:

Text: "hello world"
↓
Raw with space token: "hello▁world"
↓
Tokenization: ["hello", "▁world"]
↓
Token IDs: [1234, 5678]

Decoding:
Token IDs: [1234, 5678]
↓
Tokens: ["hello", "▁world"]
↓
Text: "hello world" (▁ → space automatically)

Training SentencePiece

import sentencepiece as spm

# Train SentencePiece model
spm.SentencePieceTrainer.train(
    input='text.txt',           # Raw text file
    model_prefix='m_user',      # Output model name
    vocab_size=32000,           # Vocabulary size
    model_type='bpe',           # Can also use 'unigram'
    character_coverage=0.9995,  # Coverage of characters
    pad_id=0,
    eos_id=2,
    unk_id=1,
    bos_id=3,
)

# Load and use
sp = spm.SentencePieceProcessor(model_file='m_user.model')

# Encode (text → token IDs)
text = "hello world"
encoded = sp.encode(text)  # Returns list of IDs
print(encoded)  # [1234, 5678]

# Encode as pieces (text → tokens)
pieces = sp.encode_as_pieces(text)
print(pieces)  # ['hello', '▁world']

# Decode (token IDs → text)
decoded = sp.decode(encoded)
print(decoded)  # "hello world"

# Vocabulary size
print(f"Vocab size: {sp.vocab_size()}")  # 32000

SentencePiece vs BPE

Comparison

Aspect              SentencePiece       BPE
──────────────────────────────────────────────────
Pre-tokenization    None                Required
Language support    Language-agnostic   English-optimized
Space handling      Built-in (▁)        Separate handling
Implementation      Simpler (no prereq)  More complex
Language coverage   All scripts          Mainly Latin
Efficiency          Good                 Good
Adoption            Modern LLMs          GPT models

Practical example:

Text: "你好世界" (Chinese: hello world)

BPE:
  - Pre-tokenization: ??? (no spaces!)
  - Problem: Not designed for this
  - Typically falls back to character-level

SentencePiece:
  - Raw text: "你好世界"
  - With space marker: "你好▁世界" (no spaces in Chinese)
  - Tokenize: ["你", "好", "▁世", "界"]
  - Works natively!

Mixed language:

Text: "Hello 世界"

BPE:
  - Pre-tokenize: "Hello" "世" "界" (awkward!)
  - Applies BPE separately
  - Suboptimal results

SentencePiece:
  - Raw: "Hello▁世界"
  - Tokenize: ["Hello", "▁世", "界"]
  - Seamless handling!

SentencePiece Model Types

BPE Mode

Algorithm: Same as standard BPE, but language-independent

Configuration:
```python
spm.SentencePieceTrainer.train(
    model_type='bpe',
    vocab_size=32000,
    ...
)

Characteristics: - Start with characters - Iteratively merge frequent pairs - Build vocabulary bottom-up - Greedy (similar to BPE)

Use case: - Most models (LLaMA uses BPE) 

### Unigram Mode
Algorithm: Probabilistic, start with large vocab and prune

Configuration: 

spm.SentencePieceTrainer.train(
    model_type='unigram',
    vocab_size=32000,
    ...
)

Characteristics: - Start with many tokens - Remove low-frequency tokens iteratively - Build vocabulary top-down - More computational cost than BPE

Advantages: - Better quality (theoretically) - More optimal - Adaptive to data

Disadvantages: - Slower training - More complex

Use case: - When quality is critical - Research (not production usually) 

---

## Practical SentencePiece Example

### Training on Custom Data

```python
# Train SentencePiece on domain data
import sentencepiece as spm

# Prepare data
with open('medical_texts.txt', 'w') as f:
    # Write all medical texts
    f.write("medical text 1...\n")
    f.write("medical text 2...\n")
    # ... more texts

# Train model
spm.SentencePieceTrainer.train(
    input='medical_texts.txt',
    model_prefix='medical_sp',
    vocab_size=32000,
    model_type='bpe',
    normalization_rule_name='identity',  # No normalization
    character_coverage=0.9999,           # Ensure coverage
    pad_id=0,
    eos_id=2,
    unk_id=1,
    bos_id=3,
)

# Load model
sp = spm.SentencePieceProcessor(model_file='medical_sp.model')

# Test tokenization
medical_texts = [
    "Pneumonoultramicroscopicsilicovolcanoconiosis is a lung disease",
    "COVID-19 vaccines use mRNA technology",
    "你好医生",  # Hello doctor in Chinese
]

for text in medical_texts:
    tokens = sp.encode_as_pieces(text)
    ids = sp.encode(text)
    print(f"Text: {text}")
    print(f"Tokens: {tokens}")
    print(f"Num tokens: {len(ids)}")
    print()

Analyzing Vocabulary

# Analyze SentencePiece vocabulary
sp = spm.SentencePieceProcessor(model_file='medical_sp.model')

# Get specific token
token_id = 100
piece = sp.id_to_piece(token_id)
print(f"Token {token_id}: {piece}")

# Get token ID
piece = "▁hello"
token_id = sp.piece_to_id(piece)
print(f"Piece '{piece}': ID {token_id}")

# Analyze token statistics
print(f"Vocabulary size: {sp.vocab_size()}")
print(f"Number of pieces: {sp.get_piece_size()}")

# Sample tokens
print("Sample tokens:")
for i in range(0, min(100, sp.vocab_size()), 10):
    piece = sp.id_to_piece(i)
    print(f"  {i}: {piece}")

Language Coverage

Multilingual Training

# Train on multiple languages
spm.SentencePieceTrainer.train(
    input='texts.txt',  # Mix of English, Chinese, Arabic, etc.
    model_prefix='multilingual_sp',
    vocab_size=64000,   # Larger vocab for multiple languages
    model_type='bpe',
    character_coverage=0.9999,  # Ensure all scripts covered
)

# Test multilingual support
sp = spm.SentencePieceProcessor(model_file='multilingual_sp.model')

texts = {
    'English': "Hello world",
    'Chinese': "你好世界",
    'Arabic': "مرحبا بالعالم",
    'Russian': "Привет мир",
    'Japanese': "こんにちは世界",
}

for lang, text in texts.items():
    tokens = sp.encode_as_pieces(text)
    print(f"{lang}: {text}")
    print(f"  Tokens ({len(tokens)}): {tokens}")
    print()

# Results:
# English: Hello world
#   Tokens (2): ['▁Hello', '▁world']
#
# Chinese: 你好世界
#   Tokens (4): ['你', '好', '世', '界']
#
# Arabic: مرحبا بالعالم
#   Tokens (4): ['مرحبا', '▁ب', 'الع', 'الم']
#
# Russian: Привет мир
#   Tokens (2): ['▁При', 'вет', '▁мир']
#
# Japanese: こんにちは世界
#   Tokens (4): ['こ', 'んに', 'ちは', '▁世界']

Character Coverage

Parameter: character_coverage

Effect on training:

coverage=0.99:
  - Trains on 99% of unique characters
  - Smaller vocab size (won't include rare scripts)
  - Unknown character → [UNK] token

coverage=0.9999:
  - Trains on 99.99% of unique characters
  - Larger vocab size (includes rare characters)
  - Unknown character: rare but possible
  - Better for multilingual

coverage=1.0 (not recommended):
  - Must include ALL unique characters
  - Can make vocab very large
  - May include noise from corrupted text
  - Not practical

Recommendation:
  - English only: 0.999
  - Multilingual: 0.9999
  - Mix with rare languages: 0.99999

Byte Fallback (The [UNK] Eliminator)

Option: byte_fallback=True  (used by LLaMA, Mistral, Gemma)

How it works:
  - if a character/piece is NOT in the vocabulary,
    encode it as raw UTF-8 BYTES instead of [UNK]
  - bytes are guaranteed in the vocab (256 of them), so
    ANY input can be represented — even emoji, typos, new scripts

LLaMA-2 config:
  spm.SentencePieceTrainer.train(
      model_type='bpe',
      byte_fallback=True,        # ← never emit [UNK] for unknown text
      character_coverage=0.9995, # rare chars fall back to bytes
  )

Result: [UNK] rate ≈ 0 on real data.
  - trade-off: unknown words become long byte sequences
    (e.g., a new emoji → several byte tokens)
  - but the information is preserved — no lossy [UNK]

Compare with [01 Bpe (Byte Pair Encoding)](/02-llm-modeling/00-fundamentals/00-tokenization/01-bpe-(byte-pair-encoding)/): GPT-2's byte-level BPE
does the same thing by starting from bytes from the very beginning.

Integration with Models

Using SentencePiece with Transformers

from transformers import AutoTokenizer, LlamaForCausalLM

# Load LLaMA (uses SentencePiece)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")
model = LlamaForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")

# Tokenize
text = "Hello world"
encoded = tokenizer.encode(text)
print(encoded)

# Generate with proper tokenization
prompt = "The quick brown fox"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_length=50)
decoded = tokenizer.decode(outputs[0])
print(decoded)

Custom Tokenizer with Transformers

from transformers import PreTrainedTokenizer
import sentencepiece as spm

class MyCustomTokenizer(PreTrainedTokenizer):
    def __init__(self, model_file, **kwargs):
        self.sp_model = spm.SentencePieceProcessor(model_file=model_file)
        super().__init__(**kwargs)

    def tokenize(self, text):
        return self.sp_model.encode_as_pieces(text)

    def convert_tokens_to_ids(self, tokens):
        return [self.sp_model.piece_to_id(t) for t in tokens]

    def convert_ids_to_tokens(self, ids):
        return [self.sp_model.id_to_piece(i) for i in ids]

    @property
    def vocab_size(self):
        return self.sp_model.vocab_size()

# Use custom tokenizer
tokenizer = MyCustomTokenizer(model_file='my_model.model')
tokens = tokenizer.encode("hello world")
print(tokens)

Key Takeaways

🌐 SentencePiece: Language-agnostic tokenization
▁ Space as token: Built-in handling eliminates pre-tokenization
📊 Two algorithms: BPE (fast) or Unigram (better quality)
🎯 Modern standard: LLaMA, Mistral use SentencePiece
🔄 No pre-tokenization: Works for any script/language