GLM: General Language Model Family

Quick Facts

Attribute	Value
Organization	Tsinghua University + Zhipu AI
Released	GLM (2021), ChatGLM (2023), GLM-4 (2024)
Architecture	Decoder-only Transformer with GLM Pre-training
Current Sizes	ChatGLM2: 6B, 12B; ChatGLM3: 6B-32B
Key Innovation	Unified Language Model for both NLU & NLG
Training Data	Chinese + English, 200B+ tokens
License	Open Weights (Research + Commercial)
Focus	Chinese-English Bilingual LLM

---

Historical Evolution

graph LR
    A["GLM<br/>2021"] -->|Add Seq2Seq| B["GLM-Large<br/>Encoder-Decoder"]
    B -->|Instruction Tuning| C["ChatGLM<br/>2023"]
    C -->|Better Alignment| D["ChatGLM2<br/>2023"]
    D -->|GPT-4 Level| E["ChatGLM3<br/>2024"]
    E -->|Extended Context| F["GLM-4<br/>2024"]

    style C fill:#ffcccc
    style D fill:#ffcccc
    style E fill:#ccffcc
    style F fill:#99ff99

---

Architecture: GLM Pre-training Paradigm

GLM's Unique Pre-training

Unlike GPT (causal) and BERT (masked), GLM uses auto-regressive blank infilling:

# Example: Auto-regressive Blank Infilling

Original Text:
"The quick brown fox jumps over the lazy dog"

Randomly Select Spans (30% of tokens):
"The quick [BLANK] fox jumps [BLANK] lazy dog"

GLM Pre-training Objective:
- Input:  "The quick [BLANK] fox jumps [BLANK] lazy dog"
- Output: "[BLANK] brown [BLANK] over the [BLANK]"
         (only fill blanks, auto-regressive)

Key Differences from BERT:
- BERT: Predict masked tokens from context (MLM)
  Input:  "The quick [MASK] fox [MASK] over the lazy dog"
  Output: "brown", "jumps"

- GLM: Generate missing spans auto-regressively
  Input:  "The quick [BLANK] fox jumps [BLANK]..."
  Output: "brown [BLANK] over the [BLANK]..." (left-to-right)

Why is this better?
✓ More similar to actual generation task
✓ Handles variable-length spans
✓ Better for both understanding AND generation
✓ Bidirectional attention for understanding
✓ Causal attention for generation

Core Innovation: Bidirectional Context + Causal Generation

- ┌─────────────────────────────────────────┐
  - Input: "The quick [BLANK] jumps [BLANK] dog"
  - ┬───────────────────┘
                      ↓
- ┌─────────────────────────────┐
    - Token Embeddings            │
    - Position Embeddings         │
  - ┬──────────────┘
                       ↓
- ┌──────────────────────────────────────┐
    - Bidirectional Attention Block        │
    - (Can attend to tokens before/after)  │
    - For known tokens: The, quick, jumps, │
    - dog (full bidirectional context)     │
  - ┬───────────────────────┘
                   ↓
- ┌──────────────────────────────────────┐
    - Auto-regressive Generation           │
    - For [BLANK] tokens:                  │
    - • [BLANK]₁ attends to bidirectional  │
    - context                           │
    - • [BLANK]₂ attends to context +      │
    - previous generated tokens          │
  - ┬───────────────────────┘
                   ↓
         Output: "brown over the"

---

GLM Model Variants

Generation Progression

Model	Year	Size	Focus	Key Features
GLM	2021	110M-10B	Research	Auto-regressive blank infilling
GLM-Large	2021	10B	NLU+NLG	Encoder-decoder hybrid
ChatGLM	2023	6B-13B	Chat	Instruction-tuned, Chinese optimized
ChatGLM2	2023	6B-12B	Production	Better performance, wider context
ChatGLM3	2024	6B-32B	Advanced	GPT-4 competitive, better reasoning
GLM-4	2024	1.3T MoE	Frontier	Extended context (128K), multimodal

ChatGLM2 Technical Details

# ChatGLM2-6B Configuration
config = {
    "hidden_size": 4096,
    "num_hidden_layers": 28,
    "num_attention_heads": 32,
    "vocab_size": 65024,
    "max_sequence_length": 32768,  # Extended from 2K
    "attention_type": "rotary_position_embedding",
    "parameters": "6B"
}

# Key Improvements:
improvements = {
    "Context Window": "4K → 32K tokens",
    "Accuracy": "Better on benchmarks",
    "Speed": "Faster inference",
    "Alignment": "Better instruction following",
    "Chat": "Multi-turn conversation support"
}

ChatGLM3 Advancements

# ChatGLM3 = ChatGLM2 + Stronger Alignment
config_gpt_comparable = {
    "sizes": ["6B", "32B"],
    "improvements": {
        "Reasoning": "Chain-of-thought capabilities",
        "Coding": "Near GPT-4 performance on HumanEval",
        "Math": "Improved mathematical reasoning",
        "Tool Use": "Better function calling",
        "Instruction": "More flexible instruction formats"
    },
    "context_window": "32K tokens"
}

---

Use Cases and Applications

Primary Markets

1. China-Focused: Optimized for Chinese NLP
2. Production Deployment: Smaller sizes (6B-12B)
3. Chat Applications: Instruction-tuned variants
4. Local Inference: Can run on consumer hardware

# Why use ChatGLM over GPT?

reasons = {
    "Cost": "Self-hosted, no API fees",
    "Privacy": "Data stays on your servers",
    "Speed": "Fast local inference (6B model)",
    "Chinese": "Better Chinese understanding",
    "Legal": "No data sharing with OpenAI/Google",
    "Control": "Can fine-tune on proprietary data"
}

---

Implementation

Installation & Setup

# Basic installation
pip install transformers torch accelerate

# GPU support (optional but recommended)
pip install torch --index-url https://download.pytorch.org/whl/cu118

Basic Usage

import torch
from transformers import AutoTokenizer, AutoModel

# Load model
model_name = "THUDM/chatglm-6b"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModel.from_pretrained(model_name, trust_remote_code=True)

# Move to GPU if available
device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)

# Generate response
prompt = "你好，请你介绍一下自己"  # "Hello, please introduce yourself"
response, history = model.chat(tokenizer, prompt, history=[])

print(response)

Conversation with History

# Multi-turn conversation
history = []

# Turn 1
prompt1 = "What is machine learning?"
response1, history = model.chat(
    tokenizer,
    prompt1,
    history=history
)
history.append((prompt1, response1))
print(f"User: {prompt1}")
print(f"Assistant: {response1}\n")

# Turn 2 (model remembers context)
prompt2 = "Can you provide some examples?"
response2, history = model.chat(
    tokenizer,
    prompt2,
    history=history
)
history.append((prompt2, response2))
print(f"User: {prompt2}")
print(f"Assistant: {response2}\n")

# Turn 3
prompt3 = "How is it different from deep learning?"
response3, history = model.chat(
    tokenizer,
    prompt3,
    history=history  # Model has full context
)
print(f"User: {prompt3}")
print(f"Assistant: {response3}")

Using ChatGLM3 (Newest)

from transformers import AutoTokenizer, AutoModel

model_name = "THUDM/chatglm3-6b"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModel.from_pretrained(model_name, trust_remote_code=True).half().cuda()
model = model.eval()

# ChatGLM3 uses OpenAI-style messages
response = model.chat(tokenizer, [
    {"role": "user", "content": "Explain the GLM pre-training objective"}
])

print(response)

Quantization for Efficiency

import torch
from transformers import AutoTokenizer, AutoModel

# 4-bit quantization (loads in ~3.5GB instead of 12GB)
model_name = "THUDM/chatglm-6b-int4"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModel.from_pretrained(model_name, trust_remote_code=True).cuda()

# Now fits on consumer GPU!

# Generate with quantized model
response, _ = model.chat(tokenizer, "你好", history=[])

Batch Inference (Production)

import torch
from transformers import AutoTokenizer, AutoModel

model_name = "THUDM/chatglm-6b"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModel.from_pretrained(model_name, trust_remote_code=True).eval()

# Batch processing for efficiency
prompts = [
    "What is AI?",
    "Explain machine learning",
    "Tell me about deep learning"
]

# Note: ChatGLM doesn't natively support batch generation
# We process sequentially but cache for efficiency
responses = []
for prompt in prompts:
    with torch.no_grad():
        response, _ = model.chat(tokenizer, prompt, history=[])
    responses.append(response)

for prompt, response in zip(prompts, responses):
    print(f"Q: {prompt}")
    print(f"A: {response}\n")

---

Performance Metrics

Benchmarks

# ChatGLM2-6B Performance
benchmarks = {
    "MMLU": 47.9,        # General knowledge (6B model)
    "HumanEval": 40.9,   # Code generation
    "GSM8K": 28.7,       # Mathematical reasoning
    "CEVAL": 52.0,       # Chinese college entrance exam
    "GAOKAO": 44.0,      # Chinese high school exam
}

# ChatGLM3-6B Performance (significant improvement)
benchmarks_v3 = {
    "MMLU": 61.4,        # +13.5 points!
    "HumanEval": 71.9,   # +31 points (code)
    "GSM8K": 52.1,       # +23.4 points (math)
    "CEVAL": 69.0,       # +17 points
    "Chinese": "Near GPT-3.5"
}

# ChatGLM3-32B
benchmarks_32b = {
    "MMLU": 74.1,
    "HumanEval": 81.0,
    "GSM8K": 82.0,
    "Chinese": "Near GPT-4 level"
}

Inference Speed

# Latency on consumer GPU (RTX 3090)
latencies = {
    "ChatGLM2-6B": {
        "First token": "200ms",  # Prompt processing
        "Per token": "50ms",     # Generation
        "Full response": "1-3s"  # Typical conversational
    },
    "ChatGLM2-6B-int4": {
        "First token": "150ms",
        "Per token": "45ms",
        "Full response": "1-2s"
    },
    "ChatGLM3-32B": {
        "First token": "500ms",
        "Per token": "100ms",
        "Full response": "3-5s"
    }
}

---

GLM vs Competitors

Comparison with GPT-3.5, Claude, Llama

graph TD
    A["LLM Comparison"] --> B["Open Weights"]
    A --> C["API Only"]

    B --> B1["ChatGLM<br/>- Open source<br/>- Free inference<br/>- Chinese optimized<br/>- Smaller models"]
    B --> B2["Llama<br/>- Larger models<br/>- English optimized<br/>- Strong reasoning"]

    C --> C1["GPT-4<br/>- SOTA<br/>- Expensive<br/>- Closed weights"]
    C --> C2["Claude<br/>- Safe/Aligned<br/>- Good reasoning<br/>- Expensive"]

Aspect	ChatGLM3	GPT-3.5	Claude 3	Llama 2 70B
Cost	Free (local)	$0.005/1K tokens	$0.015/1K	Free (local)
Speed	Fast	Fast (API)	Moderate	Moderate
Chinese	⭐⭐⭐⭐⭐	⭐⭐⭐	⭐⭐⭐⭐	⭐⭐⭐
English	⭐⭐⭐⭐	⭐⭐⭐⭐⭐	⭐⭐⭐⭐⭐	⭐⭐⭐⭐⭐
Reasoning	⭐⭐⭐⭐	⭐⭐⭐⭐	⭐⭐⭐⭐⭐	⭐⭐⭐⭐
Privacy	⭐⭐⭐⭐⭐	❌	❌	⭐⭐⭐⭐⭐
Fine-tune	✅	⚠️	⚠️	✅
Deployment	Easy	API	API	Medium

---

Fine-tuning GLM

Instruction Fine-tuning

from transformers import AutoTokenizer, AutoModel, TextGenerationPipeline
from peft import get_peft_model, LoraConfig, TaskType

# Load base model
model_name = "THUDM/chatglm-6b"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModel.from_pretrained(model_name, trust_remote_code=True)

# Setup LoRA (Low-Rank Adaptation) for efficient fine-tuning
lora_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=8,
    lora_alpha=32,
    lora_dropout=0.1,
    target_modules=["query_key_value"],
    bias="none"
)

model = get_peft_model(model, lora_config)

# Your training data
training_data = [
    {
        "instruction": "Translate to French",
        "input": "Hello, how are you?",
        "output": "Bonjour, comment allez-vous?"
    },
    # ... more examples
]

# Training loop (simplified)
from torch.utils.data import Dataset, DataLoader
import torch.optim as optim

class CustomDataset(Dataset):
    def __init__(self, data, tokenizer):
        self.data = data
        self.tokenizer = tokenizer

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
        item = self.data[idx]
        prompt = f"{item['instruction']}\n{item['input']}"
        response = item['output']

        # Tokenize
        encoded = self.tokenizer(
            f"{prompt} {response}",
            max_length=512,
            truncation=True,
            return_tensors="pt"
        )
        return encoded

# Create dataset and loader
dataset = CustomDataset(training_data, tokenizer)
dataloader = DataLoader(dataset, batch_size=4)

# Fine-tune (example)
optimizer = optim.AdamW(model.parameters(), lr=1e-4)
model.train()

for batch in dataloader:
    outputs = model(**batch)
    loss = outputs.loss

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

# Save fine-tuned model
model.save_pretrained("./chatglm-finetuned")

---

Strengths & Weaknesses

✅ Strengths

1. Chinese Excellence: Best-in-class for Chinese NLP
2. Open Source: Full weights available
3. Local Deployment: Runs on consumer hardware
4. Cost-Effective: No API fees
5. Privacy: Data stays on your servers
6. Bilingual: Handles English + Chinese well
7. Production Ready: ChatGLM2/3 mature for deployment

❌ Weaknesses

1. English Performance: Still behind GPT-4, Claude
2. Context Limit: 32K (less than some competitors)
3. Reasoning: Weaker chain-of-thought than GPT-4
4. Model Size: 6B/12B/32B available, no 70B+
5. Community: Smaller community than Llama
6. Documentation: Less comprehensive than GPT-4

---

When to Use GLM

✅ Use GLM When

# 1. Chinese language processing is critical
text = "机器学习是人工智能的重要分支"  # Chinese
# ChatGLM is best choice here

# 2. Cost is a concern (self-hosted)
# Avoid API costs: $0 vs $0.005+/1K tokens

# 3. Privacy/Data sovereignty required
# Government/enterprise with strict data policies

# 4. Running on consumer hardware
# ChatGLM-6B on RTX 3090/4090

# 5. Need fine-tuning on proprietary data
# Open weights enable custom training

❌ Use Alternatives When

# 1. Maximum English performance needed
# → GPT-4, Claude 3, Llama 70B

# 2. Long document processing (200K+ tokens)
# → GPT-4 Turbo, Claude 3

# 3. Reasoning and planning critical
# → GPT-4, Claude 3 Opus

# 4. Need extremely fast inference
# → Smaller models: Mistral 7B, Llama 13B

---

References

Core Papers

• GLM: GLM: General Language Model Pretraining with Autoregressive Blank Infilling
• Introduces GLM pre-training objective

• Bidirectional attention with causal generation

• ChatGLM: ChatGLM: An Open Bilingual Dialogue Language Model

• Instruction tuning for chat

• Bilingual optimization

• GLM-4: GLM-4 Technical Report (if released)

• Extended context, improved reasoning

Resources

• ChatGLM GitHub
• Hugging Face ChatGLM Models
• Chinese LLM Leaderboard
• Zhipu AI Official

---

Summary

ChatGLM is the best choice for: - Chinese-English bilingual applications - Cost-conscious production deployments - Privacy-sensitive use cases - Organizations needing data sovereignty

GLM Pre-training innovation: Auto-regressive blank infilling combines BERT's bidirectionality with GPT's generation capability—the best of both worlds.

Current State: ChatGLM3 achieves near-GPT-4 performance for Chinese understanding, making it competitive for multilingual applications where Chinese is important.