GLM

Quick Facts

Attribute	Value
Organization	Tsinghua University + Zhipu AI
Released	GLM (2021), ChatGLM (2023), GLM-4 (2024)
Architecture	Decoder-only Transformer with GLM Pre-training
Current Sizes	ChatGLM2: 6B, 12B; ChatGLM3: 6B-32B
Key Innovation	Unified Language Model for both NLU & NLG
Training Data	Chinese + English, 200B+ tokens
License	Open Weights (Research + Commercial)
Focus	Chinese-English Bilingual LLM

-

Historical Evolution

graph LR
 A["GLM<br/>2021"] -->|Add Seq2Seq| B["GLM-Large<br/>Encoder-Decoder"]
 B -->|Instruction Tuning| C["ChatGLM<br/>2023"]
 C -->|Better Alignment| D["ChatGLM2<br/>2023"]
 D -->|GPT-4 Level| E["ChatGLM3<br/>2024"]
 E -->|Extended Context| F["GLM-4<br/>2024"]

 style C fill:#ffcccc
 style D fill:#ffcccc
 style E fill:#ccffcc
 style F fill:#99ff99

-

Architecture: GLM Pre-training Paradigm

GLM's Unique Pre-training

Unlike GPT (causal) and BERT (masked), GLM uses auto-regressive blank infilling:

# Example

Original Text:
"The quick brown fox jumps over the lazy dog"

Randomly Select Spans (30% of tokens):
"The quick [BLANK] fox jumps [BLANK] lazy dog"

GLM Pre-training Objective:
- Input: "The quick [BLANK] fox jumps [BLANK] lazy dog"
- Output: "[BLANK] brown [BLANK] over the [BLANK]"
 (only fill blanks, auto-regressive)

Key Differences from BERT:
- BERT: Predict masked tokens from context (MLM)
 Input: "The quick [MASK] fox [MASK] over the lazy dog"
 Output: "brown", "jumps"

- GLM: Generate missing spans auto-regressively
 Input: "The quick [BLANK] fox jumps [BLANK]..."
 Output: "brown [BLANK] over the [BLANK]..." (left-to-right)

Why is this better?
More similar to actual generation task
Handles variable-length spans
Better for both understanding AND generation
Bidirectional attention for understanding
Causal attention for generation

Core Innovation: Bidirectional Context + Causal Generation

- ┌─────────────────────────────────────────┐
 - Input: "The quick [BLANK] jumps [BLANK] dog"
 - ┬───────────────────┘
 ↓
- ┌─────────────────────────────┐
 - Token Embeddings │
 - Position Embeddings │
 - ┬──────────────┘
 ↓
- ┌──────────────────────────────────────┐
 - Bidirectional Attention Block │
 - (Can attend to tokens before/after) │
 - For known tokens: The, quick, jumps, │
 - dog (full bidirectional context) │
 - ┬───────────────────────┘
 ↓
- ┌──────────────────────────────────────┐
 - Auto-regressive Generation │
 - For [BLANK] tokens: │
 - • [BLANK]₁ attends to bidirectional │
 - context │
 - • [BLANK]₂ attends to context + │
 - previous generated tokens │
 - ┬───────────────────────┘
 ↓
 Output: "brown over the"

---

GLM Model Variants

Generation Progression

Model	Year	Size	Focus	Key Features
GLM	2021	110M-10B	Research	Auto-regressive blank infilling
GLM-Large	2021	10B	NLU+NLG	Encoder-decoder hybrid
ChatGLM	2023	6B-13B	Chat	Instruction-tuned, Chinese optimized
ChatGLM2	2023	6B-12B	Production	Better performance, wider context
ChatGLM3	2024	6B-32B	Advanced	GPT-4 competitive, better reasoning
GLM-4	2024	1.3T MoE	Frontier	Extended context (128K), multimodal

ChatGLM2 Technical Details

# ChatGLM2-6B Configuration
config = {
 "hidden_size": 4096,
 "num_hidden_layers": 28,
 "num_attention_heads": 32,
 "vocab_size": 65024,
 "max_sequence_length": 32768, # Extended from 2K
 "attention_type": "rotary_position_embedding",
 "parameters": "6B"
}

# Key Improvements:
improvements = {
 "Context Window": "4K → 32K tokens",
 "Accuracy": "Better on benchmarks",
 "Speed": "Faster inference",
 "Alignment": "Better instruction following",
 "Chat": "Multi-turn conversation support"
}

ChatGLM3 Advancements

# ChatGLM3 = ChatGLM2 + Stronger Alignment
config_gpt_comparable = {
 "sizes": ["6B", "32B"],
 "improvements": {
 "Reasoning": "Chain-of-thought capabilities",
 "Coding": "Near GPT-4 performance on HumanEval",
 "Math": "Improved mathematical reasoning",
 "Tool Use": "Better function calling",
 "Instruction": "More flexible instruction formats"
 },
 "context_window": "32K tokens"
}

-

Use Cases and Applications

Primary Markets

1. China-Focused: Optimized for Chinese NLP
2. Production Deployment: Smaller sizes (6B-12B)
3. Chat Applications: Instruction-tuned variants
4. Local Inference: Can run on consumer hardware

# Why use ChatGLM over GPT?

reasons = {
 "Cost": "Self-hosted, no API fees",
 "Privacy": "Data stays on your servers",
 "Speed": "Fast local inference (6B model)",
 "Chinese": "Better Chinese understanding",
 "Legal": "No data sharing with OpenAI/Google",
 "Control": "Can fine-tune on proprietary data"
}

-

Implementation

Installation & Setup

# Basic installation
pip install transformers torch accelerate

# GPU support (optional but recommended)
pip install torch --index-url https://download.pytorch.org/whl/cu118

Basic Usage

import torch
from transformers import AutoTokenizer, AutoModel

# Load model
model_name = "THUDM/chatglm-6b"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModel.from_pretrained(model_name, trust_remote_code=True)

# Move to GPU if available
device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)

# Generate response
prompt = "你好，请你介绍一下自己" # "Hello, please introduce yourself"
response, history = model.chat(tokenizer, prompt, history=[])

print(response)

Conversation with History

# Multi-turn conversation
history = []

# Turn 1
prompt1 = "What is machine learning?"
response1, history = model.chat(
 tokenizer,
 prompt1,
 history=history
)
history.append((prompt1, response1))
print(f"User: {prompt1}")
print(f"Assistant: {response1}\n")

# Turn 2 (model remembers context)
prompt2 = "Can you provide some examples?"
response2, history = model.chat(
 tokenizer,
 prompt2,
 history=history
)
history.append((prompt2, response2))
print(f"User: {prompt2}")
print(f"Assistant: {response2}\n")

# Turn 3
prompt3 = "How is it different from deep learning?"
response3, history = model.chat(
 tokenizer,
 prompt3,
 history=history # Model has full context
)
print(f"User: {prompt3}")
print(f"Assistant: {response3}")

Using ChatGLM3 (Newest)

from transformers import AutoTokenizer, AutoModel

model_name = "THUDM/chatglm3-6b"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModel.from_pretrained(model_name, trust_remote_code=True).half().cuda()
model = model.eval()

# ChatGLM3 uses OpenAI-style messages
response = model.chat(tokenizer, [
 {"role": "user", "content": "Explain the GLM pre-training objective"}
])

print(response)

Quantization for Efficiency

import torch
from transformers import AutoTokenizer, AutoModel

# 4-bit quantization (loads in ~3.5GB instead of 12GB)
model_name = "THUDM/chatglm-6b-int4"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModel.from_pretrained(model_name, trust_remote_code=True).cuda()

# Now fits on consumer GPU!

# Generate with quantized model
response, _ = model.chat(tokenizer, "你好", history=[])

Batch Inference (Production)

import torch
from transformers import AutoTokenizer, AutoModel

model_name = "THUDM/chatglm-6b"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModel.from_pretrained(model_name, trust_remote_code=True).eval()

# Batch processing for efficiency
prompts = [
 "What is AI?",
 "Explain machine learning",
 "Tell me about deep learning"
]

# Note
# We process sequentially but cache for efficiency
responses = []
for prompt in prompts:
 with torch.no_grad():
 response, _ = model.chat(tokenizer, prompt, history=[])
 responses.append(response)

for prompt, response in zip(prompts, responses):
 print(f"Q: {prompt}")
 print(f"A: {response}\n")

---

Performance Metrics

Benchmarks

# ChatGLM2-6B Performance
benchmarks = {
 "MMLU": 47.9, # General knowledge (6B model)
 "HumanEval": 40.9, # Code generation
 "GSM8K": 28.7, # Mathematical reasoning
 "CEVAL": 52.0, # Chinese college entrance exam
 "GAOKAO": 44.0, # Chinese high school exam
}

# ChatGLM3-6B Performance (significant improvement)
benchmarks_v3 = {
 "MMLU": 61.4, # +13.5 points!
 "HumanEval": 71.9, # +31 points (code)
 "GSM8K": 52.1, # +23.4 points (math)
 "CEVAL": 69.0, # +17 points
 "Chinese": "Near GPT-3.5"
}

# ChatGLM3-32B
benchmarks_32b = {
 "MMLU": 74.1,
 "HumanEval": 81.0,
 "GSM8K": 82.0,
 "Chinese": "Near GPT-4 level"
}

Inference Speed

# Latency on consumer GPU (RTX 3090)
latencies = {
 "ChatGLM2-6B": {
 "First token": "200ms", # Prompt processing
 "Per token": "50ms", # Generation
 "Full response": "1-3s" # Typical conversational
 },
 "ChatGLM2-6B-int4": {
 "First token": "150ms",
 "Per token": "45ms",
 "Full response": "1-2s"
 },
 "ChatGLM3-32B": {
 "First token": "500ms",
 "Per token": "100ms",
 "Full response": "3-5s"
 }
}

-

GLM vs Competitors

Comparison with GPT-3.5, Claude, Llama

graph TD
 A["LLM Comparison"] --> B["Open Weights"]
 A --> C["API Only"]

 B --> B1["ChatGLM<br/>- Open source<br/>- Free inference<br/>- Chinese optimized<br/>- Smaller models"]
 B --> B2["Llama<br/>- Larger models<br/>- English optimized<br/>- Strong reasoning"]

 C --> C1["GPT-4<br/>- SOTA<br/>- Expensive<br/>- Closed weights"]
 C --> C2["Claude<br/>- Safe/Aligned<br/>- Good reasoning<br/>- Expensive"]

Aspect	ChatGLM3	GPT-3.5	Claude 3	Llama 2 70B
Cost	Free (local)	$0.005/1K tokens	$0.015/1K	Free (local)
Speed	Fast	Fast (API)	Moderate	Moderate
Chinese
English
Reasoning
Privacy
Fine-tune
Deployment	Easy	API	API	Medium

-

Fine-tuning GLM

Instruction Fine-tuning

from transformers import AutoTokenizer, AutoModel, TextGenerationPipeline
from peft import get_peft_model, LoraConfig, TaskType

# Load base model
model_name = "THUDM/chatglm-6b"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModel.from_pretrained(model_name, trust_remote_code=True)

# Setup LoRA (Low-Rank Adaptation) for efficient fine-tuning
lora_config = LoraConfig(
 task_type=TaskType.CAUSAL_LM,
 r=8,
 lora_alpha=32,
 lora_dropout=0.1,
 target_modules=["query_key_value"],
 bias="none"
)

model = get_peft_model(model, lora_config)

# Your training data
training_data = [
 {
 "instruction": "Translate to French",
 "input": "Hello, how are you?",
 "output": "Bonjour, comment allez-vous?"
 },
 #... more examples
]

# Training loop (simplified)
from torch.utils.data import Dataset, DataLoader
import torch.optim as optim

class CustomDataset(Dataset):
 def __init__(self, data, tokenizer):
 self.data = data
 self.tokenizer = tokenizer

 def __len__(self):
 return len(self.data)

 def __getitem__(self, idx):
 item = self.data[idx]
 prompt = f"{item['instruction']}\n{item['input']}"
 response = item['output']

 # Tokenize
 encoded = self.tokenizer(
 f"{prompt} {response}",
 max_length=512,
 truncation=True,
 return_tensors="pt"
)
 return encoded

# Create dataset and loader
dataset = CustomDataset(training_data, tokenizer)
dataloader = DataLoader(dataset, batch_size=4)

# Fine-tune (example)
optimizer = optim.AdamW(model.parameters(), lr=1e-4)
model.train()

for batch in dataloader:
 outputs = model(**batch)
 loss = outputs.loss

 optimizer.zero_grad()
 loss.backward()
 optimizer.step()

# Save fine-tuned model
model.save_pretrained("./chatglm-finetuned")

-

Strengths & Weaknesses

Strengths

1. Chinese Excellence: Best-in-class for Chinese NLP
2. Open Source: Full weights available
3. Local Deployment: Runs on consumer hardware
4. Cost-Effective: No API fees
5. Privacy: Data stays on your servers
6. Bilingual: Handles English + Chinese well
7. Production Ready: ChatGLM2/3 mature for deployment

Weaknesses

1. English Performance: Still behind GPT-4, Claude
2. Context Limit: 32K (less than some competitors)
3. Reasoning: Weaker chain-of-thought than GPT-4
4. Model Size: 6B/12B/32B available, no 70B+
5. Community: Smaller community than Llama
6. Documentation: Less comprehensive than GPT-4

-

When to Use GLM

Use GLM When

# 1. Chinese language processing is critical
text = "机器学习是人工智能的重要分支" # Chinese
# ChatGLM is best choice here

# 2. Cost is a concern (self-hosted)
# Avoid API costs

# 3. Privacy/Data sovereignty required
# Government/enterprise with strict data policies

# 4. Running on consumer hardware
# ChatGLM-6B on RTX 3090

# 5. Need fine-tuning on proprietary data
# Open weights enable custom training

Use Alternatives When

# 1. Maximum English performance needed
# → GPT-4, Claude 3, Llama 70B

# 2. Long document processing (200K+ tokens)
# → GPT-4 Turbo, Claude 3

# 3. Reasoning and planning critical
# → GPT-4, Claude 3 Opus

# 4. Need extremely fast inference
# → Smaller models

-

References

Core Papers

• GLM: GLM: General Language Model Pretraining with Autoregressive Blank Infilling
• Introduces GLM pre-training objective

• Bidirectional attention with causal generation

• ChatGLM: ChatGLM: An Open Bilingual Dialogue Language Model

• Instruction tuning for chat

• Bilingual optimization

• GLM-4: GLM-4 Technical Report (if released)

• Extended context, improved reasoning

Resources

• ChatGLM GitHub
• Hugging Face ChatGLM Models
• Chinese LLM Leaderboard
• Zhipu AI Official

-

Summary

ChatGLM is the best choice for:

• Chinese-English bilingual applications
• Cost-conscious production deployments
• Privacy-sensitive use cases
• Organizations needing data sovereignty

GLM Pre-training innovation: Auto-regressive blank infilling combines BERT's bidirectionality with GPT's generation capability—the best of both worlds.

Current State: ChatGLM3 achieves near-GPT-4 performance for Chinese understanding, making it competitive for multilingual applications where Chinese is important.