Skip to content

Multilingual Tokenization: Scripts, Spaces & Costs

Overview

Multilingual tokenization is where tokenizer design decisions become expensive. Languages without spaces (Chinese, Japanese, Thai), scripts with thousands of characters (CJK, Devanagari, Arabic), and unbalanced training data can inflate token counts 2–4× compared to English — directly multiplying API cost, context-window usage, and training compute. This note covers why it's hard and the toolbox modern tokenizers use to cope.

Why English-Centric Tokenizers Break

Assumption 1: Spaces Separate Words

BPE/WordPiece training:
  - pre-tokenize: split on spaces → "hello" "world"
  - learn merges WITHIN words

This works for:  English, Spanish, French, German, Russian...
Breaks for:      Chinese 你好世界, Japanese こんにちは, Thai สวัสดี

  "你好世界" (hello world):
    - no spaces → pre-tokenization returns the WHOLE string as one "word"
    - BPE then merges characters of the entire sentence together
    - or falls back to per-character tokens — 4 tokens for 4 chars

Assumption 2: A Small Alphabet Covers Everything

Latin alphabet:  ~26 letters + digits + punctuation ≈ 100–200 characters
CJK scripts:     thousands of common characters (GB2312: 6,763; JIS: 6,355)
Devanagari:      ~60 chars + thousands of conjuncts
Emoji:           〜2,000+ distinct pictographs (and growing)

character_coverage=0.9995 on a CJK corpus → the 0.05% tail is still
thousands of characters — each unseen one becomes [UNK] or a byte
sequence.

Assumption 3: One Token ≈ One Word

English:     "hello" ≈ 1 token   (word ~ 5 chars, tokenizer learns it)
Japanese:    "こんにちは" = 5 chars; a good tokenizer may pack common
             words, but many stay 1 token per 1–2 chars
Chinese:     ~1 token per character (characters ARE the morphemes)
             → 2–3 tokens per English word

The same meaning needs 2–4× more tokens in CJK.

The Token Inflation Problem (Concrete)

Same text, different languages, GPT-4-style tokenizer (approximate):

Text: "The quick brown fox jumps over the lazy dog"

English:    11 tokens  (1.00× baseline)
Spanish:    ~12 tokens (1.1×)
German:     ~14 tokens (1.3×)   (compounds: "Rechtsschutzversicherung")
Arabic:     ~15 tokens (1.4×)   (diacritics, joining forms)
Hindi:      ~18 tokens (1.6×)   (conjuncts, vowel signs)
Japanese:   ~24 tokens (2.2×)
Chinese:    ~28 tokens (2.5×)   (1 token per character)
Thai:       ~30 tokens (2.7×)   (no spaces, complex clusters)

Why this hurts (real numbers)

Context window: 8K tokens
  - English:  ~6,000 words fit
  - Chinese:  ~2,500 "words" fit (characters)  → 60% less context!

API cost: $1 per 1M tokens
  - English doc: 10K tokens  → $0.01
  - Chinese doc (same meaning): 25K tokens → $0.025  (2.5×!)

Training: 1T token corpus
  - English: 1T tokens ≈ 800B words
  - Chinese: 1T tokens ≈ 400B characters  → 2× less real content learned

💡 Key Insight: token inflation is not an efficiency nit — it directly shrinks context, doubles cost, and reduces the effective training data per language.

Why No-Space Languages Are Hard (Deep Dive)

Chinese / Japanese: no spaces

Chinese: 每个汉字基本就是一个语素 (every character ≈ a morpheme)
  - word boundaries require SEGMENTATION ("北京欢迎你" = 北京/欢迎/你)
  - good tokenizers learn common character bigrams as single tokens
  - e.g. a well-trained Chinese tokenizer:
      "北京" → 1 token   (not 北 + 京)

Japanese: こんにちは世界
  - mixed scripts: hiragana, katakana, kanji
  - tokenizers often use a pre-tokenizer that splits kanji runs
    from kana runs (SentencePiece has Japanese pre-tokenization built in)

Thai / Lao: no spaces AND complex clusters

Thai: สวัสดี (hello)
  - no spaces between words
  - consonants + vowel signs + tone marks combine into clusters
  - naive character tokenization breaks the clusters → worse quality

Practice: tokenizers either treat clusters carefully or use
byte-level fallback for rare combinations.

Indic scripts (Devanagari, Tamil, Bengali...)

Hindi: नमस्ते (hello)
  - characters combine: consonant + vowel sign (म + ् + ...)
  - "matras" (vowel signs) attach to consonants
  - normalization (NFKC) must not destroy the combining structure

The Toolbox: How Modern Tokenizers Cope

1. Bigger Vocabulary

English-only:        32–50K
Multilingual:        64–128K
Maximum coverage:    128–256K

Real models:
  mT5:          250K vocab (100+ languages)
  Qwen:         151K (Chinese + English + more)
  LLaMA-3:      128K (30+ languages)
  DeepSeek-V3:  128K (multilingual)
  GPT-4:        ~100K (speculated, multilingual)

2. character_coverage → 0.9999+

coverage=0.99    → ~1% of unique characters become [UNK] ❌
coverage=0.9999  → rare scripts included (larger vocab, no [UNK])
coverage=0.99999 → extremely rare characters (emoji, archaic)

Rule of thumb: multilingual ⇒ 0.9999+ (see [02 Sentencepiece](/02-llm-modeling/00-fundamentals/00-tokenization/02-sentencepiece/))

3. byte_fallback (the [UNK] eliminator)

byte_fallback=True (LLaMA, Mistral, Gemma):
  - unknown characters encoded as raw UTF-8 bytes
  - 256 byte tokens guaranteed in vocab → ANY input works
  - unknown Chinese char → 3 byte tokens instead of [UNK]
  - information preserved, at the cost of longer sequences

Byte-level BPE (GPT-2 style) achieves the same from the start:
  - starts from 256 bytes → every script is representable

4. Language-Balanced Sampling (training data)

Problem: English dominates the web → English dominates the vocab
         a Chinese word seen 1M times vs English word 1B times
         → Chinese gets starved of vocab slots

SentencePiece solution — input_sentence_size + unigram sampling
  (or exponential language sampling used by LLaMA/mT5):

  sample_count(lang) ∝ (corpus_share(lang))^α    with α ≈ 0.3

  α = 0.3 boosts low-resource languages:
    corpus share 0.1%  → effective ~4.4% of samples
  This gives every language a fair chance at vocab allocation.

5. Normalization (NFKC)

Normalization rule:
  - NFKC normalizes full-width/半角 to half-width, ligatures to
    base chars, compatibility forms together
  - "Ａ" (full-width A) → "A"  (both become ONE token)
  - "fi" ligature → "fi"
  - CJK: ensures visually-similar chars share tokens

SentencePiece: normalization_rule_name='nfkc' (default: 'nmt_nfkc')
⚠️  Don't apply NFKC to scripts where it destroys meaning
   (e.g., some Indic combining marks).

6. Per-Script Pre-tokenization

Tokenizer libraries split input by script before BPE:
  - Latin runs: "The quick"
  - CJK runs: "你好世界"  → segment into character/cJK-word pieces
  - Kana runs: "こんにちは"
  - Numbers:  "12345"

This prevents "the quick你好" from being merged as one token.
(SentencePiece ships a Japanese pre-tokenizer; Hugging Face
 tokenizers has 'Metaspace' and 'ByteLevel' pre-tokenizers.)

Measuring Multilingual Quality

1. Tokens per word / tokens per language:
     report efficiency PER LANGUAGE, not just overall
     (a 1.2 overall hides 3.0 for Chinese!)

2. [UNK] rate per language:
     target < 0.01%; check each script separately

3. Coverage of common words:
     do the top-1000 words of each language exist as SINGLE tokens?

4. Round-trip fidelity:
     encode → decode must return identical text
     (fails with broken byte fallback or bad normalization)

5. Benchmark parity:
     same task in 10 languages — token count AND model quality
     should not collapse for CJK/Indic scripts

def per_language_report(tokenizer, samples):
    """Report tokens-per-sample per language."""
    for lang, texts in samples.items():
        total_tokens = sum(len(tokenizer.encode(t)) for t in texts)
        total_chars  = sum(len(t) for t in texts)
        print(f"{lang:12s} {total_tokens:6d} tokens  "
              f"{total_chars / max(total_tokens, 1):5.1f} chars/token")

Code: Training a Multilingual SentencePiece

import sentencepiece as spm

spm.SentencePieceTrainer.train(
    input='mixed_languages.txt',   # English + Chinese + Hindi + Arabic...
    model_prefix='multilingual',
    vocab_size=64000,              # bigger than English-only
    model_type='bpe',
    character_coverage=0.9999,     # include rare scripts
    byte_fallback=True,            # never emit [UNK]
    normalization_rule_name='nfkc',  # unify full-width/half-width
    split_digits=True,             # keep digits separate
    allow_whitespace_only_pieces=True,
    # Language balancing (input format: "prefix\ttext"):
    #   add <s> and </s> style prefixes, or sample with α≈0.3
    #   spm supports 'input_sentence_size' + random sampling
)

# Quick multilingual sanity check
sp = spm.SentencePieceProcessor(model_file='multilingual.model')
for text in ["Hello world", "你好世界", "नमस्ते दुनिया", "مرحبا بالعالم"]:
    pieces = sp.encode_as_pieces(text)
    print(f"{text:20s} → {len(pieces)} tokens: {pieces}")

Key Takeaways

🌍 Multilingual ≠ bigger vocab: it's spaces, scripts, coverage, AND balanced training data
💰 Token inflation is cost: 2–4× tokens in CJK = 2–4× API cost, 60% less context
🧰 Five tools: larger vocab, character_coverage, byte_fallback, language sampling, NFKC
📊 Measure per language: overall metrics hide 3.0-tokens/word Chinese
🚫 [UNK] is not acceptable: byte-level/byte_fallback makes it nearly extinct
⚖️ Sample languages fairly: α ≈ 0.3 exponential sampling rescues low-resource languages