Google BERT: Bidirectional Encoder Representations from Transformers¶

Quick Facts¶

Attribute	Value
Released	October 2018
Organization	Google AI Language
Architecture	Encoder-only Transformer
Sizes	12L/768H (110M), 24L/1024H (340M)
Pre-training Data	3.3B words (BooksCorpus 800M + Wikipedia 2.5B)
Training Objective	MLM + NSP
Max Sequence Length	512 tokens
Vocabulary Size	30,522 (WordPiece)
License	Open Source (Apache 2.0)

Architecture Overview¶

Core Design¶

BERT uses a stack of bidirectional Transformer Encoders
Instead of unidirectional (left→right or right→left)
"Bidirectional" = Can see ALL tokens in BOTH directions simultaneously

This is fundamentally different from GPT's causal/autoregressive approach

Architectural Details¶

- ┌─────────────────────────────────────────────┐
    - Input: "The capital of France is Paris"     │
  - ┬──────────────────────────┘
                   ↓
- ┌─────────────────────────────────────────────┐
    - Tokenization (WordPiece)                    │
    - [CLS] The cap ##ital of France is Paris     │
    - [SEP]                                       │
  - ┬──────────────────────────┘
                   ↓
- ┌─────────────────────────────────────────────┐
    - Token Embeddings + Segment Embeddings       │
    - + Position Embeddings (0-512)               │
  - ┬──────────────────────────┘
                   ↓
- ┌─────────────────────────────────────────────┐
    - 12 × Transformer Encoder Blocks             │
    - (For Base: 768D, 12 heads, 3072 FFN)        │
    - • Multi-head Self-Attention                 │
    - • Feed-Forward Network                      │
    - • Layer Normalization + Residuals           │
  - ┬──────────────────────────┘
                   ↓
- ┌─────────────────────────────────────────────┐
    - Output: Contextual Embeddings for ALL tokens│
    - Each token "sees" all other tokens          │
  - ┘

Key Innovation: Bidirectionality¶

# GPT-style (Causal/Unidirectional)
"The capital of France is ___"
            ↓
Only previous tokens available
Cannot look ahead to "is" or "Paris"

# BERT-style (Bidirectional)
"The capital of France is ___"
     ↑↓ ↑↓ ↑↓ ↑↓ ↑↓ ↑↓
All tokens available simultaneously
Can see what comes before AND after

Consequences: - ✅ Better contextual understanding - ✅ Excellent for classification, tagging - ❌ Cannot do causal generation (next-token prediction) - ❌ Must be fine-tuned for generation tasks

Pre-training Objectives¶

1. Masked Language Modeling (MLM)¶

# Objective: Predict masked tokens from context

Original:     "The capital of France is Paris"
Masked:       "The [MASK] of France is [MASK]"
Task:         Predict: capital, Paris

# Masking strategy (15% of tokens):
# - 80% → Replace with [MASK]
#         "The [MASK] of France is [MASK]"
# - 10% → Replace with random token
#         "The keyboard of France is dogs"
# - 10% → Keep original
#         "The capital of France is Paris"

# Why random replacement and keeping original?
# - Prevents model from "cheating" by only recognizing [MASK]
# - Forces deeper contextual understanding
# - Makes model robust to corrupted input

2. Next Sentence Prediction (NSP)¶

# Objective: Predict if second sentence follows first

Sentence A: "The cat sat on the mat"
Sentence B: "It was very comfortable"
Label: IsNext (True)

# Format in training:
"[CLS] The cat sat on the mat [SEP] It was very comfortable [SEP]"
        ↑ Sentence A (segment 0)          ↑ Sentence B (segment 1)

# Model predicts: [CLS] token → classification head → IsNext/NotNext

# NSP turns out to be less useful than MLM
# (Later work like RoBERTa removed it)

Model Variants¶

Official Variants (Google)¶

# BERT-Base
config = {
    "hidden_size": 768,
    "num_hidden_layers": 12,
    "num_attention_heads": 12,
    "intermediate_size": 3072,  # FFN hidden
    "max_position_embeddings": 512,
    "vocab_size": 30522,
    "total_params": "110M"
}

# BERT-Large
config = {
    "hidden_size": 1024,
    "num_hidden_layers": 24,
    "num_attention_heads": 16,
    "intermediate_size": 4096,
    "max_position_embeddings": 512,
    "vocab_size": 30522,
    "total_params": "340M"
}

Domain-Specific Variants¶

Variant	Domain	Training Data	Use Case
SciBERT	Scientific	1.14M papers (CS)	Paper analysis, citation prediction
FinBERT	Finance	SEC filings, earnings calls	Sentiment analysis, risk assessment
BioBERT	Biomedical	PubMed abstracts (14M)	Named entity recognition, relation extraction
LegalBERT	Legal	Large legal corpus	Contract analysis, legal classification
ClinicalBERT	Medical	Clinical notes (2M)	Clinical NLP, patient outcome prediction

Multilingual Variants¶

mBERT (Multilingual BERT): 110M params, 104 languages
XLM-RoBERTa: 270M params, 100+ languages (cross-lingual)

Implementation¶

Installation & Basic Usage¶

# Install transformers library
# pip install transformers torch

from transformers import AutoTokenizer, AutoModel
import torch

# Load pre-trained BERT
model_name = "bert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)

# Tokenization
text = "The quick brown fox jumps over the lazy dog"
tokens = tokenizer.encode(text, return_tensors="pt")
# tokens: [101, 1996, 3613, 2769, 4419, 14523, 2058, 1996, 13971, 3899, 102](/101,-1996,-3613,-2769,-4419,-14523,-2058,-1996,-13971,-3899,-102/)
#         [[CLS], The, quick, brown, fox, jumps, over, the, lazy, dog, [SEP]]

# Forward pass
with torch.no_grad():
    outputs = model(tokens)

# Extract embeddings
token_embeddings = outputs.last_hidden_state  # [1, 11, 768]
#                                              # [batch, seq_len, hidden_dim]

# CLS token embedding (often used for classification)
cls_embedding = token_embeddings[:, 0, :]  # [1, 768]

Fine-tuning for Classification¶

from transformers import AutoModelForSequenceClassification, Trainer, TrainingArguments
from datasets import load_dataset

# Load dataset (e.g., SST-2 for sentiment)
dataset = load_dataset("glue", "sst2")

# Load model for classification (adds classification head)
model = AutoModelForSequenceClassification.from_pretrained(
    "bert-base-uncased",
    num_labels=2  # Binary: positive/negative
)

# Training setup
training_args = TrainingArguments(
    output_dir="./bert-sst2",
    num_train_epochs=3,
    per_device_train_batch_size=32,
    per_device_eval_batch_size=64,
    warmup_steps=500,
    weight_decay=0.01,
    logging_steps=100,
    evaluation_strategy="epoch"
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=dataset["train"],
    eval_dataset=dataset["validation"]
)

# Fine-tune
trainer.train()

# Inference
from transformers import pipeline

classifier = pipeline(
    "sentiment-analysis",
    model="path/to/fine-tuned-model"
)

result = classifier("This movie is absolutely fantastic!")
# Output: [{'label': 'POSITIVE', 'score': 0.9998}]

Fine-tuning for Named Entity Recognition (NER)¶

from transformers import AutoModelForTokenClassification, AutoTokenizer

# Load model for token classification
model = AutoModelForTokenClassification.from_pretrained(
    "bert-base-cased",
    num_labels=9  # O, B-PER, I-PER, B-ORG, I-ORG, B-LOC, I-LOC, B-MISC, I-MISC
)

tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")

# Inference pipeline
ner_pipeline = pipeline(
    "ner",
    model=model,
    tokenizer=tokenizer,
    aggregation_strategy="simple"
)

text = "Hugging Face Inc. is located in New York City."
entities = ner_pipeline(text)

# Output:
# [
#   {'entity': 'B-ORG', 'score': 0.9995, 'word': 'Hugging Face', ...},
#   {'entity': 'B-LOC', 'score': 0.9978, 'word': 'New York City', ...}
# ]

Fine-tuning for Question Answering¶

from transformers import AutoModelForQuestionAnswering, Trainer

# Dataset format: {context, question, answers}
# "answers": {"answer_start": [int], "text": [str]}

model = AutoModelForQuestionAnswering.from_pretrained("bert-base-uncased")

training_args = TrainingArguments(
    output_dir="./bert-qa",
    num_train_epochs=2,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=64
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=qa_dataset["train"],
    eval_dataset=qa_dataset["validation"],
    data_collator=default_data_collator
)

trainer.train()

# Inference
qa_pipeline = pipeline("question-answering", model=model, tokenizer=tokenizer)

result = qa_pipeline(
    question="What is the capital of France?",
    context="The capital of France is Paris, located on the Seine river."
)
# Output: {'score': 0.9845, 'start': 26, 'end': 31, 'answer': 'Paris'}

Sentence Embeddings (Information Retrieval)¶

from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity

# Pre-trained sentence BERT (fine-tuned for semantic similarity)
model = SentenceTransformer("all-MiniLM-L6-v2")  # 22M params, distilled from BERT

# Encode sentences
sentences = [
    "This is an example sentence",
    "Each sentence is converted to a vector",
    "The dog jumped over the fence"
]

embeddings = model.encode(sentences)  # [3, 384]

# Semantic similarity
similarity = cosine_similarity([embeddings[0]], embeddings)
# [0.8234, 1.0, 0.1234]

# This is useful for:
# - Semantic search
# - Clustering documents
# - Finding duplicate content
# - Recommendation systems

Practical Applications¶

1. Text Classification¶

# Spam detection
# Sentiment analysis
# Toxicity detection
# Intent classification for chatbots

result = classifier("URGENT: YOU HAVE WON $1,000,000 CLICK HERE!")
# Output: {'label': 'SPAM', 'score': 0.9998}

2. Named Entity Recognition¶

# Extract persons, organizations, locations
# Disease/drug extraction in medical text
# Product extraction from reviews

text = "Apple CEO Tim Cook announced a partnership with IBM in New York."
entities = ner_pipeline(text)
# Entities: Apple (ORG), Tim Cook (PER), IBM (ORG), New York (LOC)

3. Semantic Search¶

# Document retrieval
# Similar question finding (for FAQ systems)
# Duplicate detection

query_embedding = model.encode("What is machine learning?")
doc_embeddings = model.encode(document_list)

similarities = cosine_similarity([query_embedding], doc_embeddings)[0]
top_k = np.argsort(similarities)[-5:]  # Top 5 similar documents

4. Semantic Similarity¶

# Paraphrase detection
# Duplicate document detection
# Sentence matching

model = SentenceTransformer('paraphrase-MiniLM-L6-v2')

sentences1 = ["The cat sat on the mat", "I love dogs"]
sentences2 = ["A cat is sitting on a mat", "I enjoy puppies"]

similarity = model.similarity(sentences1, sentences2)

5. Zero-shot Classification¶

# Classify without task-specific fine-tuning
from transformers import pipeline

clf = pipeline("zero-shot-classification", model="facebook/bart-large-mnli")

text = "I love coffee"
candidate_labels = ["positive", "negative", "neutral"]

result = clf(text, candidate_labels)
# Output: {'sequence': 'I love coffee',
#          'labels': ['positive', 'neutral', 'negative'],
#          'scores': [0.998, 0.001, 0.001]}

Performance Characteristics¶

Benchmark Results¶

GLUE Benchmark (downstream NLU tasks):
- CoLA (grammaticality): 83.6
- SST-2 (sentiment): 93.5
- MRPC (paraphrase): 89.3
- SQuAD v1.1 (reading comprehension): 88.5 F1

Inference Speed:
- Base model: ~50-100ms per sequence (single GPU)
- Batch inference: ~10ms per sequence (batch size 32)

Computational Requirements¶

# Memory footprint
config = {
    "bert-base-uncased": {
        "parameters": "110M",
        "model_size": "440MB",  # float32
        "model_size_fp16": "220MB",
        "memory_for_training": "2GB",  # batch_size=32
        "inference_latency": "50ms"  # single sequence
    },
    "bert-large-uncased": {
        "parameters": "340M",
        "model_size": "1.3GB",
        "model_size_fp16": "650MB",
        "memory_for_training": "6GB",
        "inference_latency": "150ms"
    }
}

Improvements and Successors¶

BERT Limitations¶

Cannot generate text - No language modeling head
Fixed sequence length - 512 tokens maximum
NSP objective limited - Doesn't improve downstream tasks much
Redundant masking - Random replacement/original token strategies unused

Improved Variants¶

Model	Improvement	Result
RoBERTa	Remove NSP, better pre-training	GLUE: 88.5 → 88.6
ALBERT	Parameter sharing, factorization	110M→12M params, same performance
ELECTRA	Replaced token detection (discriminator)	Better sample efficiency
DeBERTa	Disentangled attention	GLUE: 88.6 → 91.5

When to Use BERT¶

✅ Good For¶

Text classification (sentiment, intent, toxicity)
Named entity recognition (NER)
Semantic similarity/search
Question answering
Text matching/paraphrase detection
When you need pre-trained encoder

❌ Not Ideal For¶

Text generation (use GPT-style models)
Long documents (fixed 512 token limit)
Real-time applications requiring speed
Causal language modeling

Key Insights¶

Bidirectionality is powerful for understanding
Pre-training → Fine-tuning is highly effective
[CLS] token often performs well for classification
Sentence-BERT transforms embeddings into semantic vectors
Domain-specific BERT (FinBERT, SciBERT) transfers well

References¶

Original Papers¶

BERT: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding - Devlin et al., 2018

RoBERTa: RoBERTa: A Robustly Optimized BERT Pretraining Approach
Sentence-BERT: Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
ALBERT: ALBERT: A Lite BERT for Self-supervised Learning
DeBERTa: DeBERTa: Decoding-enhanced BERT with Disentangled Attention

Google BERT: Bidirectional Encoder Representations from Transformers¶

Quick Facts¶

Architecture Overview¶

Core Design¶

Architectural Details¶

Key Innovation: Bidirectionality¶

Pre-training Objectives¶

1. Masked Language Modeling (MLM)¶

2. Next Sentence Prediction (NSP)¶

Model Variants¶

Official Variants (Google)¶

Domain-Specific Variants¶

Multilingual Variants¶

Implementation¶

Installation & Basic Usage¶

Fine-tuning for Classification¶

Fine-tuning for Named Entity Recognition (NER)¶

Fine-tuning for Question Answering¶

Sentence Embeddings (Information Retrieval)¶

Practical Applications¶

1. Text Classification¶

2. Named Entity Recognition¶

3. Semantic Search¶

4. Semantic Similarity¶

5. Zero-shot Classification¶

Performance Characteristics¶

Benchmark Results¶

Computational Requirements¶

Improvements and Successors¶

BERT Limitations¶

Improved Variants¶

When to Use BERT¶

✅ Good For¶

❌ Not Ideal For¶

Key Insights¶

References¶

Original Papers¶

Related Work¶

Resources¶