Google BERT

Quick Facts

Attribute	Value
Released	October 2018
Organization	Google AI Language
Architecture	Encoder-only Transformer
Sizes	12L/768H (110M), 24L/1024H (340M)
Pre-training Data	3.3B words (BooksCorpus 800M + Wikipedia 2.5B)
Training Objective	MLM + NSP
Max Sequence Length	512 tokens
Vocabulary Size	30,522 (WordPiece)
License	Open Source (Apache 2.0)

-

Architecture Overview

Core Design

BERT uses a stack of bidirectional Transformer Encoders
Instead of unidirectional (left→right or right→left)
"Bidirectional" = Can see ALL tokens in BOTH directions simultaneously

This is fundamentally different from GPT's causal/autoregressive approach

Architectural Details

- ┌─────────────────────────────────────────────┐
 - Input: "The capital of France is Paris" │
 - ┬──────────────────────────┘
 ↓
- ┌─────────────────────────────────────────────┐
 - Tokenization (WordPiece) │
 - [CLS] The cap ##ital of France is Paris │
 - [SEP] │
 - ┬──────────────────────────┘
 ↓
- ┌─────────────────────────────────────────────┐
 - Token Embeddings + Segment Embeddings │
 - + Position Embeddings (0-512) │
 - ┬──────────────────────────┘
 ↓
- ┌─────────────────────────────────────────────┐
 - 12 × Transformer Encoder Blocks │
 - (For Base: 768D, 12 heads, 3072 FFN) │
 - • Multi-head Self-Attention │
 - • Feed-Forward Network │
 - • Layer Normalization + Residuals │
 - ┬──────────────────────────┘
 ↓
- ┌─────────────────────────────────────────────┐
 - Output: Contextual Embeddings for ALL tokens│
 - Each token "sees" all other tokens │
 - ┘

Key Innovation: Bidirectionality

# GPT-style (Causal/Unidirectional)
"The capital of France is ___"
 ↓
Only previous tokens available
Cannot look ahead to "is" or "Paris"

# BERT-style (Bidirectional)
"The capital of France is ___"
 ↑↓ ↑↓ ↑↓ ↑↓ ↑↓ ↑↓
All tokens available simultaneously
Can see what comes before AND after

Consequences:

• Better contextual understanding
• Excellent for classification, tagging
• Cannot do causal generation (next-token prediction)
• Must be fine-tuned for generation tasks

-

Pre-training Objectives

1. Masked Language Modeling (MLM)

# Objective

Original: "The capital of France is Paris"
Masked: "The [MASK] of France is [MASK]"
Task: Predict: capital, Paris

# Masking strategy (15% of tokens):
# - 80% → Replace with [MASK]
# "The [MASK] of France is [MASK]"
# - 10% → Replace with random token
# "The keyboard of France is dogs"
# - 10% → Keep original
# "The capital of France is Paris"

# Why random replacement and keeping original?
# - Prevents model from "cheating" by only recognizing [MASK]
# - Forces deeper contextual understanding
# - Makes model robust to corrupted input

2. Next Sentence Prediction (NSP)

# Objective

Sentence A: "The cat sat on the mat"
Sentence B: "It was very comfortable"
Label: IsNext (True)

# Format in training:
"[CLS] The cat sat on the mat [SEP] It was very comfortable [SEP]"
 ↑ Sentence A (segment 0) ↑ Sentence B (segment 1)

# Model predicts

# NSP turns out to be less useful than MLM
# (Later work like RoBERTa removed it)

---

Model Variants

Official Variants (Google)

# BERT-Base
config = {
 "hidden_size": 768,
 "num_hidden_layers": 12,
 "num_attention_heads": 12,
 "intermediate_size": 3072, # FFN hidden
 "max_position_embeddings": 512,
 "vocab_size": 30522,
 "total_params": "110M"
}

# BERT-Large
config = {
 "hidden_size": 1024,
 "num_hidden_layers": 24,
 "num_attention_heads": 16,
 "intermediate_size": 4096,
 "max_position_embeddings": 512,
 "vocab_size": 30522,
 "total_params": "340M"
}

Domain-Specific Variants

Variant	Domain	Training Data	Use Case
SciBERT	Scientific	1.14M papers (CS)	Paper analysis, citation prediction
FinBERT	Finance	SEC filings, earnings calls	Sentiment analysis, risk assessment
BioBERT	Biomedical	PubMed abstracts (14M)	Named entity recognition, relation extraction
LegalBERT	Legal	Large legal corpus	Contract analysis, legal classification
ClinicalBERT	Medical	Clinical notes (2M)	Clinical NLP, patient outcome prediction

Multilingual Variants

• mBERT (Multilingual BERT): 110M params, 104 languages
• XLM-RoBERTa: 270M params, 100+ languages (cross-lingual)

-

Implementation

Installation & Basic Usage

# Install transformers library
# pip install transformers torch

from transformers import AutoTokenizer, AutoModel
import torch

# Load pre-trained BERT
model_name = "bert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)

# Tokenization
text = "The quick brown fox jumps over the lazy dog"
tokens = tokenizer.encode(text, return_tensors="pt")
# tokens
# [[CLS], The, quick, brown, fox, jumps, over, the, lazy, dog, [SEP]]

# Forward pass
with torch.no_grad():
 outputs = model(tokens)

# Extract embeddings
token_embeddings = outputs.last_hidden_state # [1, 11, 768]
# # [batch, seq_len, hidden_dim]

# CLS token embedding (often used for classification)
cls_embedding = token_embeddings[:, 0,:] # [1, 768]

Fine-tuning for Classification

from transformers import AutoModelForSequenceClassification, Trainer, TrainingArguments
from datasets import load_dataset

# Load dataset (e.g., SST-2 for sentiment)
dataset = load_dataset("glue", "sst2")

# Load model for classification (adds classification head)
model = AutoModelForSequenceClassification.from_pretrained(
 "bert-base-uncased",
 num_labels=2 # Binary: positive/negative
)

# Training setup
training_args = TrainingArguments(
 output_dir="./bert-sst2",
 num_train_epochs=3,
 per_device_train_batch_size=32,
 per_device_eval_batch_size=64,
 warmup_steps=500,
 weight_decay=0.01,
 logging_steps=100,
 evaluation_strategy="epoch"
)

trainer = Trainer(
 model=model,
 args=training_args,
 train_dataset=dataset["train"],
 eval_dataset=dataset["validation"]
)

# Fine-tune
trainer.train()

# Inference
from transformers import pipeline

classifier = pipeline(
 "sentiment-analysis",
 model="path/to/fine-tuned-model"
)

result = classifier("This movie is absolutely fantastic!")
# Output

Fine-tuning for Named Entity Recognition (NER)

from transformers import AutoModelForTokenClassification, AutoTokenizer

# Load model for token classification
model = AutoModelForTokenClassification.from_pretrained(
 "bert-base-cased",
 num_labels=9 # O, B-PER, I-PER, B-ORG, I-ORG, B-LOC, I-LOC, B-MISC, I-MISC
)

tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")

# Inference pipeline
ner_pipeline = pipeline(
 "ner",
 model=model,
 tokenizer=tokenizer,
 aggregation_strategy="simple"
)

text = "Hugging Face Inc. is located in New York City."
entities = ner_pipeline(text)

# Output:
# [
# {'entity'
# {'entity'
#]

Fine-tuning for Question Answering

from transformers import AutoModelForQuestionAnswering, Trainer

# Dataset format
# "answers"

model = AutoModelForQuestionAnswering.from_pretrained("bert-base-uncased")

training_args = TrainingArguments(
 output_dir="./bert-qa",
 num_train_epochs=2,
 per_device_train_batch_size=16,
 per_device_eval_batch_size=64
)

trainer = Trainer(
 model=model,
 args=training_args,
 train_dataset=qa_dataset["train"],
 eval_dataset=qa_dataset["validation"],
 data_collator=default_data_collator
)

trainer.train()

# Inference
qa_pipeline = pipeline("question-answering", model=model, tokenizer=tokenizer)

result = qa_pipeline(
 question="What is the capital of France?",
 context="The capital of France is Paris, located on the Seine river."
)
# Output

Sentence Embeddings (Information Retrieval)

from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity

# Pre-trained sentence BERT (fine-tuned for semantic similarity)
model = SentenceTransformer("all-MiniLM-L6-v2") # 22M params, distilled from BERT

# Encode sentences
sentences = [
 "This is an example sentence",
 "Each sentence is converted to a vector",
 "The dog jumped over the fence"
]

embeddings = model.encode(sentences) # [3, 384]

# Semantic similarity
similarity = cosine_similarity([embeddings[0]], embeddings)
# [0.8234, 1.0, 0.1234]

# This is useful for:
# - Semantic search
# - Clustering documents
# - Finding duplicate content
# - Recommendation systems

-

Practical Applications

1. Text Classification

# Spam detection
# Sentiment analysis
# Toxicity detection
# Intent classification for chatbots

result = classifier("URGENT: YOU HAVE WON $1,000,000 CLICK HERE!")
# Output

2. Named Entity Recognition

# Extract persons, organizations, locations
# Disease/drug extraction in medical text
# Product extraction from reviews

text = "Apple CEO Tim Cook announced a partnership with IBM in New York."
entities = ner_pipeline(text)
# Entities

3. Semantic Search

# Document retrieval
# Similar question finding (for FAQ systems)
# Duplicate detection

query_embedding = model.encode("What is machine learning?")
doc_embeddings = model.encode(document_list)

similarities = cosine_similarity([query_embedding], doc_embeddings)[0]
top_k = np.argsort(similarities)[-5:] # Top 5 similar documents

4. Semantic Similarity

# Paraphrase detection
# Duplicate document detection
# Sentence matching

model = SentenceTransformer('paraphrase-MiniLM-L6-v2')

sentences1 = ["The cat sat on the mat", "I love dogs"]
sentences2 = ["A cat is sitting on a mat", "I enjoy puppies"]

similarity = model.similarity(sentences1, sentences2)

5. Zero-shot Classification

# Classify without task-specific fine-tuning
from transformers import pipeline

clf = pipeline("zero-shot-classification", model="facebook/bart-large-mnli")

text = "I love coffee"
candidate_labels = ["positive", "negative", "neutral"]

result = clf(text, candidate_labels)
# Output
# 'labels'
# 'scores'

---

Performance Characteristics

Benchmark Results

GLUE Benchmark (downstream NLU tasks):
- CoLA (grammaticality): 83.6
- SST-2 (sentiment): 93.5
- MRPC (paraphrase): 89.3
- SQuAD v1.1 (reading comprehension): 88.5 F1

Inference Speed:
- Base model: ~50-100ms per sequence (single GPU)
- Batch inference: ~10ms per sequence (batch size 32)

Computational Requirements

# Memory footprint
config = {
 "bert-base-uncased": {
 "parameters": "110M",
 "model_size": "440MB", # float32
 "model_size_fp16": "220MB",
 "memory_for_training": "2GB", # batch_size=32
 "inference_latency": "50ms" # single sequence
 },
 "bert-large-uncased": {
 "parameters": "340M",
 "model_size": "1.3GB",
 "model_size_fp16": "650MB",
 "memory_for_training": "6GB",
 "inference_latency": "150ms"
 }
}

-

Improvements and Successors

BERT Limitations

1. Cannot generate text - No language modeling head
2. Fixed sequence length - 512 tokens maximum
3. NSP objective limited - Doesn't improve downstream tasks much
4. Redundant masking - Random replacement/original token strategies unused

Improved Variants

Model	Improvement	Result
RoBERTa	Remove NSP, better pre-training	GLUE: 88.5 → 88.6
ALBERT	Parameter sharing, factorization	110M→12M params, same performance
ELECTRA	Replaced token detection (discriminator)	Better sample efficiency
DeBERTa	Disentangled attention	GLUE: 88.6 → 91.5

-

When to Use BERT

Good For

• Text classification (sentiment, intent, toxicity)
• Named entity recognition (NER)
• Semantic similarity/search
• Question answering
• Text matching/paraphrase detection
• When you need pre-trained encoder

Not Ideal For

• Text generation (use GPT-style models)
• Long documents (fixed 512 token limit)
• Real-time applications requiring speed
• Causal language modeling

---

Key Insights

1. Bidirectionality is powerful for understanding
2. Pre-training → Fine-tuning is highly effective
3. [CLS] token often performs well for classification
4. Sentence-BERT transforms embeddings into semantic vectors
5. Domain-specific BERT (FinBERT, SciBERT) transfers well

-

References

Original Papers

• BERT: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding - Devlin et al., 2018

Related Work

• RoBERTa: RoBERTa: A Robustly Optimized BERT Pretraining Approach
• Sentence-BERT: Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• ALBERT: ALBERT: A Lite BERT for Self-supervised Learning
• DeBERTa: DeBERTa: Decoding-enhanced BERT with Disentangled Attention

Resources

• Hugging Face BERT Documentation
• BERT Model Hub
• Sentence-BERT Pretrained Models