Skip to content

SentencePiece

Overview

SentencePiece is a language-independent tokenization library that trains directly on raw text (no pre-tokenization). Used by LLaMA, Mistral, and many modern models. Simpler and more language-agnostic than BPE variants.

  • Paper: "SentencePiece: A simple and language independent subword tokenizer and detokenizer for NMT" (Kudo & Richardson, 2018)
  • Adoption: LLaMA, Mistral, Qwen, Falcon, many modern LLMs
  • Key Feature: Treats space as special token, works for any language
  • Advantage: Language-agnostic, no pre-tokenization needed

-

How SentencePiece Works

Core Idea: Space as Token

Traditional BPE:
 - Pre-tokenize: "hello world" → ["hello", "world"]
 - Then apply BPE on tokens
 - Problem: Pre-tokenization is language-specific!

SentencePiece approach:
 - Treat space as special character
 - "hello world" → "hello▁world"
 - (▁ represents space/underscore)
 - Apply BPE/Unigram on raw text
 - Result: Language-independent!

Example encoding:

Text: "hello world"
↓
Raw with space token: "hello▁world"
↓
Tokenization: ["hello", "▁world"]
↓
Token IDs: [1234, 5678]

Decoding:
Token IDs: [1234, 5678]
↓
Tokens: ["hello", "▁world"]
↓
Text: "hello world" (▁ → space automatically)

Training SentencePiece

import sentencepiece as spm

# Train SentencePiece model
spm.SentencePieceTrainer.train(
 input='text.txt', # Raw text file
 model_prefix='m_user', # Output model name
 vocab_size=32000, # Vocabulary size
 model_type='bpe', # Can also use 'unigram'
 character_coverage=0.9995, # Coverage of characters
 pad_id=0,
 eos_id=2,
 unk_id=1,
 bos_id=3,
)

# Load and use
sp = spm.SentencePieceProcessor(model_file='m_user.model')

# Encode (text → token IDs)
text = "hello world"
encoded = sp.encode(text) # Returns list of IDs
print(encoded) # [1234, 5678]

# Encode as pieces (text → tokens)
pieces = sp.encode_as_pieces(text)
print(pieces) # ['hello', '▁world']

# Decode (token IDs → text)
decoded = sp.decode(encoded)
print(decoded) # "hello world"

# Vocabulary size
print(f"Vocab size: {sp.vocab_size()}") # 32000

SentencePiece vs BPE

Comparison

Aspect SentencePiece BPE
──────────────────────────────────────────────────
Pre-tokenization None Required
Language support Language-agnostic English-optimized
Space handling Built-in (▁) Separate handling
Implementation Simpler (no prereq) More complex
Language coverage All scripts Mainly Latin
Efficiency Good Good
Adoption Modern LLMs GPT models

Practical example:

Text: "你好世界" (Chinese: hello world)

BPE:
 - Pre-tokenization: ??? (no spaces!)
 - Problem: Not designed for this
 - Typically falls back to character-level

SentencePiece:
 - Raw text: "你好世界"
 - With space marker: "你好▁世界" (no spaces in Chinese)
 - Tokenize: ["你", "好", "▁世", "界"]
 - Works natively!

Mixed language:

Text: "Hello 世界"

BPE:
 - Pre-tokenize: "Hello" "世" "界" (awkward!)
 - Applies BPE separately
 - Suboptimal results

SentencePiece:
 - Raw: "Hello▁世界"
 - Tokenize: ["Hello", "▁世", "界"]
 - Seamless handling!

SentencePiece Model Types

BPE Mode

Algorithm: Same as standard BPE, but language-independent

Configuration:
```python
spm.SentencePieceTrainer.train(
 model_type='bpe',
 vocab_size=32000,
...
)

Characteristics:

  • Start with characters
  • Iteratively merge frequent pairs
  • Build vocabulary bottom-up
  • Greedy (similar to BPE)

Use case:

  • Most models (LLaMA uses BPE)
### Unigram Mode

Algorithm: Probabilistic, start with large vocab and prune

Configuration:

spm.SentencePieceTrainer.train(
 model_type='unigram',
 vocab_size=32000,
...
)

Characteristics:

  • Start with many tokens
  • Remove low-frequency tokens iteratively
  • Build vocabulary top-down
  • More computational cost than BPE

Advantages:

  • Better quality (theoretically)
  • More optimal
  • Adaptive to data

Disadvantages:

  • Slower training
  • More complex

Use case:

  • When quality is critical
  • Research (not production usually)
---

## Practical SentencePiece Example

### Training on Custom Data

```python
# Train SentencePiece on domain data
import sentencepiece as spm

# Prepare data
with open('medical_texts.txt', 'w') as f:
 # Write all medical texts
 f.write("medical text 1...\n")
 f.write("medical text 2...\n")
 #... more texts

# Train model
spm.SentencePieceTrainer.train(
 input='medical_texts.txt',
 model_prefix='medical_sp',
 vocab_size=32000,
 model_type='bpe',
 normalization_rule_name='identity', # No normalization
 character_coverage=0.9999, # Ensure coverage
 pad_id=0,
 eos_id=2,
 unk_id=1,
 bos_id=3,
)

# Load model
sp = spm.SentencePieceProcessor(model_file='medical_sp.model')

# Test tokenization
medical_texts = [
 "Pneumonoultramicroscopicsilicovolcanoconiosis is a lung disease",
 "COVID-19 vaccines use mRNA technology",
 "你好医生", # Hello doctor in Chinese
]

for text in medical_texts:
 tokens = sp.encode_as_pieces(text)
 ids = sp.encode(text)
 print(f"Text: {text}")
 print(f"Tokens: {tokens}")
 print(f"Num tokens: {len(ids)}")
 print()

Analyzing Vocabulary

# Analyze SentencePiece vocabulary
sp = spm.SentencePieceProcessor(model_file='medical_sp.model')

# Get specific token
token_id = 100
piece = sp.id_to_piece(token_id)
print(f"Token {token_id}: {piece}")

# Get token ID
piece = "▁hello"
token_id = sp.piece_to_id(piece)
print(f"Piece '{piece}': ID {token_id}")

# Analyze token statistics
print(f"Vocabulary size: {sp.vocab_size()}")
print(f"Number of pieces: {sp.get_piece_size()}")

# Sample tokens
print("Sample tokens:")
for i in range(0, min(100, sp.vocab_size()), 10):
 piece = sp.id_to_piece(i)
 print(f" {i}: {piece}")

-

Language Coverage

Multilingual Training

# Train on multiple languages
spm.SentencePieceTrainer.train(
 input='texts.txt', # Mix of English, Chinese, Arabic, etc.
 model_prefix='multilingual_sp',
 vocab_size=64000, # Larger vocab for multiple languages
 model_type='bpe',
 character_coverage=0.9999, # Ensure all scripts covered
)

# Test multilingual support
sp = spm.SentencePieceProcessor(model_file='multilingual_sp.model')

texts = {
 'English': "Hello world",
 'Chinese': "你好世界",
 'Arabic': "مرحبا بالعالم",
 'Russian': "Привет мир",
 'Japanese': "こんにちは世界",
}

for lang, text in texts.items():
 tokens = sp.encode_as_pieces(text)
 print(f"{lang}: {text}")
 print(f" Tokens ({len(tokens)}): {tokens}")
 print()

# Results:
# English
# Tokens (2)
#
# Chinese
# Tokens (4)
#
# Arabic
# Tokens (4)
#
# Russian
# Tokens (2)
#
# Japanese
# Tokens (4)

Character Coverage

Parameter: character_coverage

Effect on training:

coverage=0.99:
 - Trains on 99% of unique characters
 - Smaller vocab size (won't include rare scripts)
 - Unknown character → [UNK] token

coverage=0.9999:
 - Trains on 99.99% of unique characters
 - Larger vocab size (includes rare characters)
 - Unknown character: rare but possible
 - Better for multilingual

coverage=1.0 (not recommended):
 - Must include ALL unique characters
 - Can make vocab very large
 - May include noise from corrupted text
 - Not practical

Recommendation:
 - English only: 0.999
 - Multilingual: 0.9999
 - Mix with rare languages: 0.99999

Byte Fallback (The [UNK] Eliminator)

Option: byte_fallback=True (used by LLaMA, Mistral, Gemma)

How it works:
 - if a character/piece is NOT in the vocabulary,
 encode it as raw UTF-8 BYTES instead of [UNK]
 - bytes are guaranteed in the vocab (256 of them), so
 ANY input can be represented — even emoji, typos, new scripts

LLaMA-2 config:
 spm.SentencePieceTrainer.train(
 model_type='bpe',
 byte_fallback=True, # ← never emit [UNK] for unknown text
 character_coverage=0.9995, # rare chars fall back to bytes
)

Result: [UNK] rate ≈ 0 on real data.
 - trade-off: unknown words become long byte sequences
 (e.g., a new emoji → several byte tokens)
 - but the information is preserved — no lossy [UNK]

Compare with [01 Bpe (Byte Pair Encoding)](/01-modeling/00-fundamentals/00-tokenization/01-bpe-(byte-pair-encoding)/): GPT-2's byte-level BPE
does the same thing by starting from bytes from the very beginning.

-

Integration with Models

Using SentencePiece with Transformers

from transformers import AutoTokenizer, LlamaForCausalLM

# Load LLaMA (uses SentencePiece)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")
model = LlamaForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")

# Tokenize
text = "Hello world"
encoded = tokenizer.encode(text)
print(encoded)

# Generate with proper tokenization
prompt = "The quick brown fox"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_length=50)
decoded = tokenizer.decode(outputs[0])
print(decoded)

Custom Tokenizer with Transformers

from transformers import PreTrainedTokenizer
import sentencepiece as spm

class MyCustomTokenizer(PreTrainedTokenizer):
 def __init__(self, model_file, **kwargs):
 self.sp_model = spm.SentencePieceProcessor(model_file=model_file)
 super().__init__(**kwargs)

 def tokenize(self, text):
 return self.sp_model.encode_as_pieces(text)

 def convert_tokens_to_ids(self, tokens):
 return [self.sp_model.piece_to_id(t) for t in tokens]

 def convert_ids_to_tokens(self, ids):
 return [self.sp_model.id_to_piece(i) for i in ids]

 @property
 def vocab_size(self):
 return self.sp_model.vocab_size()

# Use custom tokenizer
tokenizer = MyCustomTokenizer(model_file='my_model.model')
tokens = tokenizer.encode("hello world")
print(tokens)

-

Key Takeaways

SentencePiece: Language-agnostic tokenization ▁ Space as token: Built-in handling eliminates pre-tokenization Two algorithms: BPE (fast) or Unigram (better quality) Modern standard: LLaMA, Mistral use SentencePiece No pre-tokenization: Works for any script/language

-