Google T5: Text-to-Text Transfer Transformer

Quick Facts

Attribute	Value
Released	October 2019
Organization	Google Research (Brain Team)
Architecture	Encoder-Decoder Transformer (Seq2Seq)
Sizes	Small (60M), Base (220M), Large (770M), 3B, 11B
Pre-training Data	C4 (750GB of English text)
Training Objective	Denoising Autoencoder (noise spans replaced)
Max Sequence Length	512 tokens input, 128-512 output
Vocabulary Size	32,128 (SentencePiece)
License	Open Source (Apache 2.0)
Innovation	Unified Text-to-Text Framework

---

The Core Insight: Text-to-Text

T5's revolutionary insight is treating ALL NLP tasks as text-to-text problems:

Task                    Input Format                    Output
─────────────────────────────────────────────────────────────────
Classification:   "classify: Movie review"        →  "positive"
                  [FULL REVIEW TEXT]

Summarization:    "summarize: [DOCUMENT]"         →  "[SUMMARY]"

Translation:      "translate English to French:   →  "[FRENCH TEXT]"
                  [ENGLISH TEXT]"

Question Answer:  "question: What is AI?"         →  "[ANSWER]"
                  "context: [DOCUMENT]"

Paraphrase:       "rephrase: [SENTENCE]"          →  "[PARAPHRASE]"

Sentiment:        "sentiment: [TEXT]"             →  "positive"

NER:              "extract entities: [TEXT]"      →  "person: John, org: Google"

This unified approach enables: - Single model for multiple tasks - Transfer learning across tasks - Instruction-based prompting (pre-cursor to modern LLMs)

---

Architecture

Encoder-Decoder Structure

Input Text                          Output Text
───────────────────────────────────────────────

"translate English to French:       "Bonjour, je suis"
Hello, I am"
     ↓
- ┌──────────────────────────────┐
    - ENCODER (12 transformer      │
    - layers for T5-Base)          │
    - • Bi-directional attention   │
    - • Processes full input       │
    - • Produces context vectors   │
  - ┬───────────────┘
               ↓
         Context Embeddings
         (B, T, 768)
               ↓
- ┌──────────────────────────────┐
    - DECODER (12 transformer      │
    - layers)                      │
    - • Causal (left-to-right)     │
    - • Auto-regressive            │
    - • Attends to encoder outputs │
    - • Generates one token at a   │
    - time                       │
  - ┬───────────────┘
               ↓
         Token Predictions
         (vocabulary logits)
               ↓
         "Bonjour, je suis"

Key Architectural Components

# T5 Configuration (T5-Base)
config = {
    # Encoder
    "encoder_layers": 12,
    "encoder_hidden_size": 768,
    "encoder_feed_forward_size": 3072,
    "encoder_attention_heads": 12,

    # Decoder
    "decoder_layers": 12,
    "decoder_hidden_size": 768,
    "decoder_feed_forward_size": 3072,
    "decoder_attention_heads": 12,

    # Shared
    "vocab_size": 32128,
    "max_position_embeddings": 512,
    "dropout_rate": 0.1,
    "layer_norm_epsilon": 1e-6,

    # Model size
    "total_parameters": "220M"
}

Attention Mechanisms

1. Encoder Self-Attention
   Each token attends to ALL tokens in input
   (Can look forward and backward)

2. Decoder Self-Attention
   Each token attends only to PREVIOUS tokens
   (Causal masking - cannot look forward)

3. Cross-Attention
   Decoder attends to encoder's output
   (Query from decoder, Key/Value from encoder)

Example:
Input:  "summarize: The quick brown fox"

Encoder Self-Attention:
"summarize" sees: [summarize, :, The, quick, brown, fox]
"quick"     sees: [summarize, :, The, quick, brown, fox]

Decoder produces output token-by-token:
Step 1: Generate first word "The" (attends to encoder)
Step 2: Generate "fox" (attends to encoder + previous tokens)
Step 3: Generate "[EOS]" (stop)

---

Pre-training Objective

Denoising Autoencoder

T5 uses a denoising autoencoder objective, not just MLM:

# Original text:
"The quick brown fox jumps over the lazy dog"

# Create corrupted version (15% spans):
# - 50% → Replace with [MASK]
# - 50% → Replace with sentinel tokens

"The quick [X] fox jumps [Y] lazy dog"

# Note: Unlike BERT's token-level masking,
# T5 masks SPANS (contiguous sequences)

# T5 pre-training task:
Input:   "The quick [X] fox jumps [Y] lazy dog"
Predict: "brown [X] over the [Y]"

# Sentinel tokens: [X], [Y], [Z], etc.
# Each span maps to unique sentinel
# Model learns to infill missing spans

Why Denoising is Better

# Advantages over BERT's MLM:
advantages = {
    "Contiguous spans": "Learns to handle longer corruptions",
    "Longer contexts": "Better for document-level understanding",
    "Generation-like": "Mimics actual text generation task",
    "Variable lengths": "Spans vary in length"
}

# Example effectiveness:
# BERT (token-level):    "The [MASK] brown [MASK] jumps"
# T5 (span-level):       "The [X] jumps [Y]"
#
# T5 learns more realistic corruption patterns

---

Model Variants

Official Sizes

Variant	Params	Encoder Layers	Hidden Size	Speed	Best For
T5-Small	60M	8	512	⚡⚡⚡	Edge, mobile
T5-Base	220M	12	768	⚡⚡	Standard inference
T5-Large	770M	24	1024	⚡	Stronger performance
T5-3B	3B	24	1024	❌	High-end servers
T5-11B	11B	24	1024	❌	Research, large scale

Specialized Variants

Model	Purpose	Key Features
mT5	Multilingual (75 langs)	Cross-lingual transfer
mt5-small	Smaller multilingual	300M params, 75 languages
byT5	Byte-level	Handles any Unicode, no tokenizer needed
FlanT5	Instruction following	Fine-tuned on 100+ tasks with instructions
mt0	Massively multilingual	300+ languages and language families

FlanT5: Instruction-Following Version

# FlanT5 is fine-tuned T5 on 100+ diverse tasks
# Result: Follows instructions naturally

# Example:
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model = AutoModelForSeq2SeqLM.from_pretrained("google/flan-t5-base")
tokenizer = AutoTokenizer.from_pretrained("google/flan-t5-base")

# Works without explicit task prefix!
inputs = tokenizer("What is 2+2?", return_tensors="pt")
outputs = model.generate(**inputs)
print(tokenizer.decode(outputs[0]))  # "4"

# Compare to base T5:
# Base T5 would need: "arithmetic: 2 + 2 = "
# FlanT5 understands the question naturally

---

Implementation Guide

Installation

pip install transformers torch

1. Summarization

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
import torch

model_name = "google/t5-base"
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Long document
document = """
The quick brown fox jumps over the lazy dog. 
This sentence contains every letter of the alphabet.
It's a pangram, used for testing fonts and keyboards.
The phrase is often used in typography.
It dates back to the 1880s.
"""

# Task prefix
input_text = f"summarize: {document}"
inputs = tokenizer(
    input_text,
    max_length=512,
    truncation=True,
    return_tensors="pt"
)

# Generate summary
summary_ids = model.generate(
    inputs["input_ids"],
    max_length=150,
    min_length=40,
    num_beams=4,  # Beam search
    length_penalty=2.0,
    early_stopping=True
)

summary = tokenizer.decode(summary_ids[0], skip_special_tokens=True)
print(summary)
# Output: "The quick brown fox is a pangram containing all letters. 
#          It's used for testing and dates to the 1880s."

2. Machine Translation

from transformers import pipeline

# Use pipeline for convenient access
translator = pipeline(
    "translation_en_to_de",
    model="google/t5-base"  # Or use specific translation model
)

english_text = "Hello, my name is John and I am a software engineer."
german = translator(english_text)

print(german[0]['translation_text'])
# Output: "Hallo, mein Name ist John und ich bin ein Softwareentwickler."

# Multi-language translation
source_text = "Good morning, how are you?"

# English to French
translator_en_fr = pipeline("translation_en_to_fr")
french = translator_en_fr(source_text)[0]['translation_text']

# English to Spanish
translator_en_es = pipeline("translation_en_to_es")
spanish = translator_en_es(source_text)[0]['translation_text']

3. Question Answering

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model = AutoModelForSeq2SeqLM.from_pretrained("google/t5-base")
tokenizer = AutoTokenizer.from_pretrained("google/t5-base")

context = """
Machine Learning is a subset of Artificial Intelligence.
It focuses on algorithms that can learn from data.
Deep Learning uses neural networks with multiple layers.
"""

question = "What is Machine Learning?"

# Task format
input_text = f"question: {question} context: {context}"

inputs = tokenizer(input_text, max_length=512, truncation=True, return_tensors="pt")
outputs = model.generate(**inputs, max_length=100)

answer = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(answer)
# Output: "A subset of Artificial Intelligence that focuses on algorithms 
#          that learn from data"

4. Sentiment Analysis

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model = AutoModelForSeq2SeqLM.from_pretrained("google/t5-base")
tokenizer = AutoTokenizer.from_pretrained("google/t5-base")

text = "This movie was absolutely fantastic! I loved every minute of it."

# Sentiment task
input_text = f"sentiment: {text}"

inputs = tokenizer(input_text, max_length=512, truncation=True, return_tensors="pt")
outputs = model.generate(**inputs, max_length=10)

sentiment = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(sentiment)  # Output: "positive"

5. Fine-tuning on Custom Data

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer, Seq2SeqTrainer, Seq2SeqTrainingArguments
from datasets import load_dataset, Dataset
import pandas as pd

# Load dataset
dataset = load_dataset("glue", "mrpc")  # Paraphrase detection

# Prepare data for T5
def preprocess_function(examples):
    inputs = [f"paraphrase: {ex}" for ex in examples["sentence1"]]
    targets = examples["sentence2"]

    model_inputs = tokenizer(
        inputs,
        max_length=128,
        truncation=True,
        padding="max_length"
    )

    labels = tokenizer(
        targets,
        max_length=128,
        truncation=True,
        padding="max_length"
    )

    model_inputs["labels"] = labels["input_ids"]
    return model_inputs

# Apply preprocessing
processed_dataset = dataset.map(
    preprocess_function,
    batched=True,
    remove_columns=dataset["train"].column_names
)

# Training setup
model = AutoModelForSeq2SeqLM.from_pretrained("google/t5-base")
tokenizer = AutoTokenizer.from_pretrained("google/t5-base")

training_args = Seq2SeqTrainingArguments(
    output_dir="./t5-paraphrase",
    num_train_epochs=3,
    per_device_train_batch_size=32,
    per_device_eval_batch_size=64,
    warmup_steps=500,
    weight_decay=0.01,
    save_total_limit=3,
    evaluation_strategy="epoch"
)

trainer = Seq2SeqTrainer(
    model=model,
    args=training_args,
    train_dataset=processed_dataset["train"],
    eval_dataset=processed_dataset["validation"],
    tokenizer=tokenizer
)

trainer.train()

---

Performance Comparison

Task Performance (T5-Large)

Task                     T5-Large    BERT    GPT-2
───────────────────────────────────────────────
SQuAD v1.1 (QA)          90.7 F1     88.4    -
GLUE (Classification)    87.0        88.4    -
Machine Translation      BLEU 28     -       -
Summarization            44.0 ROUGE  -       -

Inference Speed

# Latency comparison (T5-Base, single GPU)

tasks = {
    "Classification (SST-2)": "50ms",
    "Summarization": "800ms (1500 char doc)",
    "Translation": "150ms",
    "QA": "250ms"
}

# Batch efficiency
batch_latencies = {
    "Batch size 1": 50,    # ms
    "Batch size 4": 120,   # ms (30ms per item)
    "Batch size 32": 600,  # ms (18ms per item)
}

# Batch is ~3x more efficient per item

---

T5 vs Other Models

Comparison Matrix

graph TD
    A["Sequence-to-Sequence Framework"] --> B["BERT vs T5 vs GPT"]

    B --> C["BERT"]
    C --> C1["Encoder only<br/>Bidirectional<br/>Classification focus"]

    B --> D["T5"]
    D --> D1["Encoder-Decoder<br/>Any-to-any<br/>All NLP tasks"]

    B --> E["GPT"]
    E --> E1["Decoder only<br/>Unidirectional<br/>Generation focus"]

    style C1 fill:#ffcccc
    style D1 fill:#ccffcc
    style E1 fill:#ccccff

Aspect	BERT	T5	GPT
Architecture	Encoder only	Encoder-Decoder	Decoder only
Direction	Bidirectional	Both	Causal/Unidirectional
Primary Strength	Classification	Multi-task transfer	Generation
Task Prefix Needed	No	Yes	Sometimes
Can Generate Text	No	Yes	Yes
Context Understanding	⭐⭐⭐⭐⭐	⭐⭐⭐⭐⭐	⭐⭐⭐⭐
Generalization	Task-specific	Excellent	Excellent
Fine-tuning Ease	Very Easy	Easy	Moderate

---

T5 Strengths & Weaknesses

✅ Strengths

1. Unified Framework: One model for many tasks
2. Transfer Learning: Pre-training on C4 gives strong initialization
3. Instruction-Following: Especially FlanT5 variants
4. Encoder-Decoder: Ideal for seq2seq problems
5. Open Source: Fully available and well-documented
6. Multilingual: mT5 covers 75+ languages

❌ Weaknesses

1. Requires Task Prefix: Need to know task format in advance
2. Not State-of-the-Art: Newer models (GPT-4, Claude) are stronger
3. Fixed Vocabulary: Cannot handle out-of-vocabulary well
4. Slower at Scale: Sequence length affects quadratic attention
5. Smaller Context: 512 tokens vs modern 4K-128K

---

Use Cases

Ideal For T5

# 1. Multiple tasks → One model
model = T5(pretrained_weights)
for task in [summarization, translation, qa, classification]:
    predictions = model(task_prompt + input_text)

# 2. Instruction-following (FlanT5)
model = FlanT5()
output = model("Write a haiku about programming")
# Output: "Code flows like rivers\nBugs hide in the shadows\nDebugging at dawn"

# 3. Text infilling
prompt = "The capital of France is [X]. It is famous for [Y]."
output = model(prompt)

# 4. Multilingual task (mT5)
model = mT5()
summary = model("summarize: [English doc]")  # English
summary = model("summarize: [French doc]")   # French

Not Ideal For

# 1. Real-time streaming (slow generation)
# 2. Very long documents (512 token limit)
# 3. When task format unknown (GPT better)
# 4. Edge devices (too large for most)

---

FlanT5: Instruction-Following Version

What Changed?

# Base T5 requires explicit task prefix:
"summarize: The quick brown fox jumps over the lazy dog"

# FlanT5 understands natural instructions:
"Summarize this text: The quick brown fox jumps over the lazy dog"
"Can you summarize: The quick brown fox jumps over the lazy dog"
"Write a brief summary of: The quick brown fox jumps over the lazy dog"

# All produce reasonable outputs!

Key Differences

# FlanT5 is Base T5 + fine-tuned on:
# - 100+ diverse NLP tasks
# - Multiple instruction formats per task
# - Instruction-based prompting (pre-cursor to ChatGPT)

# Original paper: "Finetuned Language Models are Zero-Shot Learners"
# Result: Better generalization to unseen tasks

# Instruction diversity examples:
instructions = {
    "Sentiment": [
        "What sentiment is this text?",
        "Is this text positive or negative?",
        "Rate the sentiment of: ",
        "Determine if this is positive: "
    ],
    "Translation": [
        "Translate to French: ",
        "Convert to French: ",
        "French translation: ",
        "How do you say this in French?"
    ]
}

---

Papers & References

Core Papers

• T5: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Introduces T5 architecture and denoising objective
• Comparison of pre-training objectives

• Comprehensive evaluation across tasks

• FlanT5: Finetuned Language Models are Zero-Shot Learners

• Instruction fine-tuning methodology
• 100+ task collection (FLAN)

• Zero-shot transfer to new tasks

• mT5: mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer

• 75+ language support
• Cross-lingual transfer learning
• Multilingual evaluation

Resources

• Hugging Face T5 Documentation
• T5 Model Hub
• FlanT5 Variants
• Papers with Code: T5

---

Summary

T5 represents a paradigm shift in NLP:

1. Unified Objective: Text-to-text for all tasks
2. Strong Pre-training: C4 corpus provides excellent initialization
3. Effective Transfer: Works well for diverse downstream tasks
4. Open Source: Democratizes advanced NLP
5. Instruction-Following: FlanT5 variant enables instruction-based usage

Today's Context: While newer models (GPT-4, Claude) exceed T5 in many metrics, T5 remains valuable for: - Multi-task learning - Fine-tuning on specific tasks - Resource-constrained environments - Educational understanding of seq2seq models