Google T5

Quick Facts

Attribute	Value
Released	October 2019
Organization	Google Research (Brain Team)
Architecture	Encoder-Decoder Transformer (Seq2Seq)
Sizes	Small (60M), Base (220M), Large (770M), 3B, 11B
Pre-training Data	C4 (750GB of English text)
Training Objective	Denoising Autoencoder (noise spans replaced)
Max Sequence Length	512 tokens input, 128-512 output
Vocabulary Size	32,128 (SentencePiece)
License	Open Source (Apache 2.0)
Innovation	Unified Text-to-Text Framework

---

The Core Insight: Text-to-Text

T5's revolutionary insight is treating ALL NLP tasks as text-to-text problems:

Task Input Format Output
─────────────────────────────────────────────────────────────────
Classification: "classify: Movie review" → "positive"
 [FULL REVIEW TEXT]

Summarization: "summarize: [DOCUMENT]" → "[SUMMARY]"

Translation: "translate English to French: → "[FRENCH TEXT]"
 [ENGLISH TEXT]"

Question Answer: "question: What is AI?" → "[ANSWER]"
 "context: [DOCUMENT]"

Paraphrase: "rephrase: [SENTENCE]" → "[PARAPHRASE]"

Sentiment: "sentiment: [TEXT]" → "positive"

NER: "extract entities: [TEXT]" → "person: John, org: Google"

This unified approach enables:

• Single model for multiple tasks
• Transfer learning across tasks
• Instruction-based prompting (pre-cursor to modern LLMs)

-

Architecture

Encoder-Decoder Structure

Input Text Output Text
───────────────────────────────────────────────

"translate English to French: "Bonjour, je suis"
Hello, I am"
 ↓
- ┌──────────────────────────────┐
 - ENCODER (12 transformer │
 - layers for T5-Base) │
 - • Bi-directional attention │
 - • Processes full input │
 - • Produces context vectors │
 - ┬───────────────┘
 ↓
 Context Embeddings
 (B, T, 768)
 ↓
- ┌──────────────────────────────┐
 - DECODER (12 transformer │
 - layers) │
 - • Causal (left-to-right) │
 - • Auto-regressive │
 - • Attends to encoder outputs │
 - • Generates one token at a │
 - time │
 - ┬───────────────┘
 ↓
 Token Predictions
 (vocabulary logits)
 ↓
 "Bonjour, je suis"

Key Architectural Components

# T5 Configuration (T5-Base)
config = {
 # Encoder
 "encoder_layers": 12,
 "encoder_hidden_size": 768,
 "encoder_feed_forward_size": 3072,
 "encoder_attention_heads": 12,

 # Decoder
 "decoder_layers": 12,
 "decoder_hidden_size": 768,
 "decoder_feed_forward_size": 3072,
 "decoder_attention_heads": 12,

 # Shared
 "vocab_size": 32128,
 "max_position_embeddings": 512,
 "dropout_rate": 0.1,
 "layer_norm_epsilon": 1e-6,

 # Model size
 "total_parameters": "220M"
}

Attention Mechanisms

1. Encoder Self-Attention
 Each token attends to ALL tokens in input
 (Can look forward and backward)

2. Decoder Self-Attention
 Each token attends only to PREVIOUS tokens
 (Causal masking - cannot look forward)

3. Cross-Attention
 Decoder attends to encoder's output
 (Query from decoder, Key/Value from encoder)

Example:
Input: "summarize: The quick brown fox"

Encoder Self-Attention:
"summarize" sees: [summarize,:, The, quick, brown, fox]
"quick" sees: [summarize,:, The, quick, brown, fox]

Decoder produces output token-by-token:
Step 1: Generate first word "The" (attends to encoder)
Step 2: Generate "fox" (attends to encoder + previous tokens)
Step 3: Generate "[EOS]" (stop)

---

Pre-training Objective

Denoising Autoencoder

T5 uses a denoising autoencoder objective, not just MLM:

# Original text:
"The quick brown fox jumps over the lazy dog"

# Create corrupted version (15% spans):
# - 50% → Replace with [MASK]
# - 50% → Replace with sentinel tokens

"The quick [X] fox jumps [Y] lazy dog"

# Note
# T5 masks SPANS (contiguous sequences)

# T5 pre-training task:
Input: "The quick [X] fox jumps [Y] lazy dog"
Predict: "brown [X] over the [Y]"

# Sentinel tokens
# Each span maps to unique sentinel
# Model learns to infill missing spans

Why Denoising is Better

# Advantages over BERT's MLM:
advantages = {
 "Contiguous spans": "Learns to handle longer corruptions",
 "Longer contexts": "Better for document-level understanding",
 "Generation-like": "Mimics actual text generation task",
 "Variable lengths": "Spans vary in length"
}

# Example effectiveness:
# BERT (token-level)
# T5 (span-level)
#
# T5 learns more realistic corruption patterns

---

Model Variants

Official Sizes

Variant	Params	Encoder Layers	Hidden Size	Speed	Best For
T5-Small	60M	8	512		Edge, mobile
T5-Base	220M	12	768		Standard inference
T5-Large	770M	24	1024		Stronger performance
T5-3B	3B	24	1024		High-end servers
T5-11B	11B	24	1024		Research, large scale

Specialized Variants

Model	Purpose	Key Features
mT5	Multilingual (75 langs)	Cross-lingual transfer
mt5-small	Smaller multilingual	300M params, 75 languages
byT5	Byte-level	Handles any Unicode, no tokenizer needed
FlanT5	Instruction following	Fine-tuned on 100+ tasks with instructions
mt0	Massively multilingual	300+ languages and language families

FlanT5: Instruction-Following Version

# FlanT5 is fine-tuned T5 on 100+ diverse tasks
# Result

# Example:
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model = AutoModelForSeq2SeqLM.from_pretrained("google/flan-t5-base")
tokenizer = AutoTokenizer.from_pretrained("google/flan-t5-base")

# Works without explicit task prefix!
inputs = tokenizer("What is 2+2?", return_tensors="pt")
outputs = model.generate(**inputs)
print(tokenizer.decode(outputs[0])) # "4"

# Compare to base T5:
# Base T5 would need
# FlanT5 understands the question naturally

-

Implementation Guide

Installation

pip install transformers torch

1. Summarization

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
import torch

model_name = "google/t5-base"
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Long document
document = """
The quick brown fox jumps over the lazy dog. 
This sentence contains every letter of the alphabet.
It's a pangram, used for testing fonts and keyboards.
The phrase is often used in typography.
It dates back to the 1880s.
"""

# Task prefix
input_text = f"summarize: {document}"
inputs = tokenizer(
 input_text,
 max_length=512,
 truncation=True,
 return_tensors="pt"
)

# Generate summary
summary_ids = model.generate(
 inputs["input_ids"],
 max_length=150,
 min_length=40,
 num_beams=4, # Beam search
 length_penalty=2.0,
 early_stopping=True
)

summary = tokenizer.decode(summary_ids[0], skip_special_tokens=True)
print(summary)
# Output
# It's used for testing and dates to the 1880s."

2. Machine Translation

from transformers import pipeline

# Use pipeline for convenient access
translator = pipeline(
 "translation_en_to_de",
 model="google/t5-base" # Or use specific translation model
)

english_text = "Hello, my name is John and I am a software engineer."
german = translator(english_text)

print(german[0]['translation_text'])
# Output

# Multi-language translation
source_text = "Good morning, how are you?"

# English to French
translator_en_fr = pipeline("translation_en_to_fr")
french = translator_en_fr(source_text)[0]['translation_text']

# English to Spanish
translator_en_es = pipeline("translation_en_to_es")
spanish = translator_en_es(source_text)[0]['translation_text']

3. Question Answering

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model = AutoModelForSeq2SeqLM.from_pretrained("google/t5-base")
tokenizer = AutoTokenizer.from_pretrained("google/t5-base")

context = """
Machine Learning is a subset of Artificial Intelligence.
It focuses on algorithms that can learn from data.
Deep Learning uses neural networks with multiple layers.
"""

question = "What is Machine Learning?"

# Task format
input_text = f"question: {question} context: {context}"

inputs = tokenizer(input_text, max_length=512, truncation=True, return_tensors="pt")
outputs = model.generate(**inputs, max_length=100)

answer = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(answer)
# Output
# that learn from data"

4. Sentiment Analysis

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model = AutoModelForSeq2SeqLM.from_pretrained("google/t5-base")
tokenizer = AutoTokenizer.from_pretrained("google/t5-base")

text = "This movie was absolutely fantastic! I loved every minute of it."

# Sentiment task
input_text = f"sentiment: {text}"

inputs = tokenizer(input_text, max_length=512, truncation=True, return_tensors="pt")
outputs = model.generate(**inputs, max_length=10)

sentiment = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(sentiment) # Output: "positive"

5. Fine-tuning on Custom Data

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer, Seq2SeqTrainer, Seq2SeqTrainingArguments
from datasets import load_dataset, Dataset
import pandas as pd

# Load dataset
dataset = load_dataset("glue", "mrpc") # Paraphrase detection

# Prepare data for T5
def preprocess_function(examples):
 inputs = [f"paraphrase: {ex}" for ex in examples["sentence1"]]
 targets = examples["sentence2"]

 model_inputs = tokenizer(
 inputs,
 max_length=128,
 truncation=True,
 padding="max_length"
)

 labels = tokenizer(
 targets,
 max_length=128,
 truncation=True,
 padding="max_length"
)

 model_inputs["labels"] = labels["input_ids"]
 return model_inputs

# Apply preprocessing
processed_dataset = dataset.map(
 preprocess_function,
 batched=True,
 remove_columns=dataset["train"].column_names
)

# Training setup
model = AutoModelForSeq2SeqLM.from_pretrained("google/t5-base")
tokenizer = AutoTokenizer.from_pretrained("google/t5-base")

training_args = Seq2SeqTrainingArguments(
 output_dir="./t5-paraphrase",
 num_train_epochs=3,
 per_device_train_batch_size=32,
 per_device_eval_batch_size=64,
 warmup_steps=500,
 weight_decay=0.01,
 save_total_limit=3,
 evaluation_strategy="epoch"
)

trainer = Seq2SeqTrainer(
 model=model,
 args=training_args,
 train_dataset=processed_dataset["train"],
 eval_dataset=processed_dataset["validation"],
 tokenizer=tokenizer
)

trainer.train()

---

Performance Comparison

Task Performance (T5-Large)

Task T5-Large BERT GPT-2
───────────────────────────────────────────────
SQuAD v1.1 (QA) 90.7 F1 88.4 -
GLUE (Classification) 87.0 88.4 -
Machine Translation BLEU 28 - -
Summarization 44.0 ROUGE - -

Inference Speed

# Latency comparison (T5-Base, single GPU)

tasks = {
 "Classification (SST-2)": "50ms",
 "Summarization": "800ms (1500 char doc)",
 "Translation": "150ms",
 "QA": "250ms"
}

# Batch efficiency
batch_latencies = {
 "Batch size 1": 50, # ms
 "Batch size 4": 120, # ms (30ms per item)
 "Batch size 32": 600, # ms (18ms per item)
}

# Batch is ~3x more efficient per item

-

T5 vs Other Models

Comparison Matrix

graph TD
 A["Sequence-to-Sequence Framework"] --> B["BERT vs T5 vs GPT"]

 B --> C["BERT"]
 C --> C1["Encoder only<br/>Bidirectional<br/>Classification focus"]

 B --> D["T5"]
 D --> D1["Encoder-Decoder<br/>Any-to-any<br/>All NLP tasks"]

 B --> E["GPT"]
 E --> E1["Decoder only<br/>Unidirectional<br/>Generation focus"]

 style C1 fill:#ffcccc
 style D1 fill:#ccffcc
 style E1 fill:#ccccff

Aspect	BERT	T5	GPT
Architecture	Encoder only	Encoder-Decoder	Decoder only
Direction	Bidirectional	Both	Causal/Unidirectional
Primary Strength	Classification	Multi-task transfer	Generation
Task Prefix Needed	No	Yes	Sometimes
Can Generate Text	No	Yes	Yes
Context Understanding
Generalization	Task-specific	Excellent	Excellent
Fine-tuning Ease	Very Easy	Easy	Moderate

---

T5 Strengths & Weaknesses

Strengths

1. Unified Framework: One model for many tasks
2. Transfer Learning: Pre-training on C4 gives strong initialization
3. Instruction-Following: Especially FlanT5 variants
4. Encoder-Decoder: Ideal for seq2seq problems
5. Open Source: Fully available and well-documented
6. Multilingual: mT5 covers 75+ languages

Weaknesses

1. Requires Task Prefix: Need to know task format in advance
2. Not State-of-the-Art: Newer models (GPT-4, Claude) are stronger
3. Fixed Vocabulary: Cannot handle out-of-vocabulary well
4. Slower at Scale: Sequence length affects quadratic attention
5. Smaller Context: 512 tokens vs modern 4K-128K

---

Use Cases

Ideal For T5

# 1. Multiple tasks → One model
model = T5(pretrained_weights)
for task in [summarization, translation, qa, classification]:
 predictions = model(task_prompt + input_text)

# 2. Instruction-following (FlanT5)
model = FlanT5()
output = model("Write a haiku about programming")
# Output

# 3. Text infilling
prompt = "The capital of France is [X]. It is famous for [Y]."
output = model(prompt)

# 4. Multilingual task (mT5)
model = mT5()
summary = model("summarize: [English doc]") # English
summary = model("summarize: [French doc]") # French

Not Ideal For

# 1. Real-time streaming (slow generation)
# 2. Very long documents (512 token limit)
# 3. When task format unknown (GPT better)
# 4. Edge devices (too large for most)

---

FlanT5: Instruction-Following Version

What Changed?

# Base T5 requires explicit task prefix:
"summarize: The quick brown fox jumps over the lazy dog"

# FlanT5 understands natural instructions:
"Summarize this text: The quick brown fox jumps over the lazy dog"
"Can you summarize: The quick brown fox jumps over the lazy dog"
"Write a brief summary of: The quick brown fox jumps over the lazy dog"

# All produce reasonable outputs!

Key Differences

# FlanT5 is Base T5 + fine-tuned on:
# - 100+ diverse NLP tasks
# - Multiple instruction formats per task
# - Instruction-based prompting (pre-cursor to ChatGPT)

# Original paper
# Result

# Instruction diversity examples:
instructions = {
 "Sentiment": [
 "What sentiment is this text?",
 "Is this text positive or negative?",
 "Rate the sentiment of: ",
 "Determine if this is positive: "
],
 "Translation": [
 "Translate to French: ",
 "Convert to French: ",
 "French translation: ",
 "How do you say this in French?"
]
}

-

Papers & References

Core Papers

• T5: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Introduces T5 architecture and denoising objective
• Comparison of pre-training objectives

• Comprehensive evaluation across tasks

• FlanT5: Finetuned Language Models are Zero-Shot Learners

• Instruction fine-tuning methodology
• 100+ task collection (FLAN)

• Zero-shot transfer to new tasks

• mT5: mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer

• 75+ language support
• Cross-lingual transfer learning
• Multilingual evaluation

Resources

• Hugging Face T5 Documentation
• T5 Model Hub
• FlanT5 Variants
• Papers with Code: T5

-

Summary

T5 represents a paradigm shift in NLP:

1. Unified Objective: Text-to-text for all tasks
2. Strong Pre-training: C4 corpus provides excellent initialization
3. Effective Transfer: Works well for diverse downstream tasks
4. Open Source: Democratizes advanced NLP
5. Instruction-Following: FlanT5 variant enables instruction-based usage

Today's Context: While newer models (GPT-4, Claude) exceed T5 in many metrics, T5 remains valuable for:

• Multi-task learning
• Fine-tuning on specific tasks
• Resource-constrained environments
• Educational understanding of seq2seq models