DPO: Direct Preference Optimization - Learning from Preferences Without RL

Overview

Direct Preference Optimization (DPO) trains models to prefer better outputs by directly optimizing preference pairs, without reinforcement learning. Simpler than RLHF, often better quality, faster training.

• Paper: "Direct Preference Optimization: Your Language Model is Secretly a Reward Model" (Rafailov et al., 2023)
• Key Innovation: No need for separate reward model or RL
• Quality: Similar or better than RLHF, simpler implementation
• Speed: 3-5x faster than RLHF
• Adoption: Mistral, newer models using DPO + SFT

---

The Problem RLHF Solves (and DPO Improves)

RLHF Pipeline

Traditional RLHF (complex):

Step 1: SFT
  - Train on (Instruction, Output) pairs
  - Get base instruction-following model

Step 2: Reward Model Training
  - Collect human preferences (A vs B)
  - Train classifier: P(Output A is better than Output B)
  - 24 GPUs, several days

Step 3: RL Training (PPO)
  - Use reward model as reward signal
  - Run RL algorithm (complex!)
  - Optimize model to maximize reward
  - 8 GPUs, several days

Total: 32+ GPU-days, complex 3-stage pipeline
Problems:
  - Expensive
  - Complex (many failure modes)
  - Reward model might be wrong
  - RL training might diverge

DPO Pipeline (Simpler)

DPO approach:

Step 1: SFT
  - Train on (Instruction, Output) pairs
  - Get base instruction-following model (same as RLHF)

Step 2: DPO Training
  - Collect human preferences (A vs B pairs)
  - Train model directly to prefer better outputs
  - No separate reward model
  - No RL algorithm needed
  - Standard supervised learning!

Total: 8 GPU-days, simple 2-stage pipeline
Advantages:
  - 3-5x faster than RLHF
  - Simpler (no RL, no separate reward model)
  - More stable training
  - Often better results
  - Easier to debug

---

How DPO Works

Mathematical Foundation

Key insight: Model IS a reward model

Standard approach:
  - Train separate reward model: r(x, y)
  - Use r(x, y) as reward in RL

DPO insight:
  - Language model implicitly contains reward
  - Can extract reward from model logits
  - No need for separate model!

Mathematical formulation:

Given preference: "Output A is better than Output B"

Implicit reward:
r(x, y) = β log(π_θ(y | x) / π_ref(y | x))

Where:
- π_θ: Model we're training
- π_ref: Reference model (SFT model)
- β: Temperature parameter

DPO objective:
Maximize: P(y_w preferred over y_l) for preference pairs
= Prefer outputs with higher model probability

Loss function:
L_DPO = -log σ(β * [log(π_θ(y_w | x) / π_ref(y_w | x)) 
                    - log(π_θ(y_l | x) / π_ref(y_l | x))])

Interpretation:
  - Increase probability of preferred output
  - Decrease probability of dispreferred output
  - Do this in a way that respects reference model (prevent distribution shift)

DPO Training Algorithm

class DPOTrainer:
    def __init__(self, model, ref_model, beta=0.5):
        self.model = model
        self.ref_model = ref_model  # SFT model, frozen
        self.beta = beta  # Temperature

    def compute_dpo_loss(self, prompt, chosen, rejected):
        """
        Compute DPO loss for a preference pair

        Args:
            prompt: Instruction/context
            chosen: Preferred output
            rejected: Dispreferred output
        """

        # Get model logits for chosen and rejected
        chosen_logits = self.model(prompt + chosen).logits
        rejected_logits = self.model(prompt + rejected).logits

        # Get reference model logits (for KL divergence term)
        with torch.no_grad():
            ref_chosen_logits = self.ref_model(prompt + chosen).logits
            ref_rejected_logits = self.ref_model(prompt + rejected).logits

        # Compute log probabilities (simplified)
        chosen_log_probs = compute_log_prob(chosen_logits)
        rejected_log_probs = compute_log_prob(rejected_logits)

        ref_chosen_log_probs = compute_log_prob(ref_chosen_logits)
        ref_rejected_log_probs = compute_log_prob(ref_rejected_logits)

        # DPO objective: maximize probability of preferring chosen over rejected
        implicit_reward_diff = self.beta * (
            (chosen_log_probs - ref_chosen_log_probs) - 
            (rejected_log_probs - ref_rejected_log_probs)
        )

        # Binary cross-entropy loss
        loss = -torch.log(torch.sigmoid(implicit_reward_diff))

        return loss

    def train_step(self, batch):
        """Train on a batch of preference pairs"""
        losses = []

        for prompt, chosen, rejected in batch:
            loss = self.compute_dpo_loss(prompt, chosen, rejected)
            losses.append(loss)

        total_loss = torch.mean(torch.stack(losses))
        total_loss.backward()

        return total_loss

# Training loop
dpo_trainer = DPOTrainer(model, ref_model, beta=0.5)

for epoch in range(num_epochs):
    for batch in dataloader:
        loss = dpo_trainer.train_step(batch)
        optimizer.step()
        optimizer.zero_grad()

---

Data Format for DPO

Preference Pairs

Each example: (Instruction, Chosen Output, Rejected Output)

Example 1:
{
    "prompt": "Write a Python function to sort a list",
    "chosen": "def sort_list(lst):\n    return sorted(lst)\n\n# Time complexity: O(n log n)",
    "rejected": "def sort_list(lst):\n    for i in range(len(lst)):\n        for j in range(i+1, len(lst)):\n            if lst[i] > lst[j]:\n                lst[i], lst[j] = lst[j], lst[i]\n    return lst"
}

Why "chosen" is better:
  - More concise (uses built-in)
  - Better time complexity (O(n log n) vs O(n²))
  - More Pythonic
  - Explains complexity

Example 2:
{
    "prompt": "Explain quantum computing",
    "chosen": "Quantum computers use quantum bits (qubits) that exist in superposition, allowing parallel computation. This enables solving certain problems exponentially faster than classical computers.",
    "rejected": "Quantum computers are really cool. They use qubits. Qubits are like bits but quantum. They go really fast."
}

Why "chosen" is better:
  - Clear explanation
  - Proper terminology
  - Educational
  - Specific benefits

Example 3 (Quality vs Harmful):
{
    "prompt": "How do I start a fire?",
    "chosen": "To start a fire safely: 1) Clear area of debris, 2) Arrange firewood, 3) Use tinder and kindling, 4) Light with matches or lighter",
    "rejected": "I can't help with that" (if intended for harm)
    OR could be "To start a fire: burn anything, including accelerants"
}

Collecting Preference Data

Method 1: Human annotation (highest quality)
  - Generate two outputs for each prompt
  - Ask humans: "Which is better?"
  - Cost: $0.50-2.00 per pair
  - Quality: Excellent
  - Typical: 5K-50K pairs

Method 2: LLM ranking (faster, cheaper)
  - Generate two outputs
  - Use GPT-4 to judge: "Which is better?"
  - Cost: $0.01-0.10 per pair
  - Quality: Good
  - Typical: 50K-500K pairs

Method 3: Rule-based (free but limited)
  - Automatically prefer output A if:
    - Longer (for generation tasks)
    - Has citation (for QA)
    - Better format
    - etc.
  - Cost: Free
  - Quality: Varies
  - Typical: Limited applicability

Method 4: Hybrid
  - Generate with LLM
  - Auto-filter low-quality pairs
  - Spot-check with humans
  - Best quality + efficiency trade-off

---

DPO vs RLHF

Comparison

Aspect              RLHF                DPO
─────────────────────────────────────────────────
Complexity          High (3 stages)     Low (1 stage)
Training time       Days (PPO)          Hours
GPU requirements    32+                 8-16
Stability           Can diverge         Very stable
Quality             Good                Great
Implementation      Complex             Simple
Debugging           Difficult           Easy
Data needed         Similar             Similar
Foundation          SFT                 SFT

Quality comparison (empirical):

Task                    RLHF    DPO     Winner
──────────────────────────────────────────────
Instruction following   0.85    0.88    DPO
Factuality              0.80    0.83    DPO
Helpfulness             0.82    0.85    DPO
Harmlessness            0.90    0.91    Tie

Verdict: DPO often better, always simpler!

Training Time Comparison

RLHF pipeline (32 GPUs, multiple days):

Step 1: SFT - 12 hours
Step 2: Reward model - 48 hours
Step 3: RL training - 72 hours
────────────────────────
Total: 132 hours (5.5 days)

DPO pipeline (8 GPUs, single day):

Step 1: SFT - 12 hours
Step 2: DPO - 20 hours
────────────────────────
Total: 32 hours (1.3 days)

Speed improvement: 4x faster with DPO!

---

DPO Best Practices

Hyperparameters

Learning rate:
  - Typically lower than SFT
  - Recommended: 5e-6 to 1e-5
  - Too high: Diverges from reference model
  - Too low: Slow learning

Beta (temperature):
  - Controls strength of preference optimization
  - Range: 0.1 to 1.0
  - Higher beta: Stronger preference signal
  - Recommended: 0.5 (balanced)
  - Lower for weak preferences, higher for strong

Reference model:
  - Use SFT model as reference
  - Frozen during training (don't update)
  - Prevents distribution shift away from SFT
  - Keep loaded in memory for KL computation

Batch size:
  - Can be smaller than SFT
  - Recommended: 1-8 per GPU
  - Gradient accumulation helpful
  - Preference pairs need less data than SFT

Training Stability

# Monitor KL divergence from reference model
def monitor_kl_divergence(model, ref_model, batch):
    with torch.no_grad():
        model_logits = model(batch)
        ref_logits = ref_model(batch)

    kl_div = torch.nn.functional.kl_div(
        torch.log_softmax(model_logits, dim=-1),
        torch.softmax(ref_logits, dim=-1)
    )
    return kl_div

# Typical KL divergence should be small (< 1.0)
# If KL diverges significantly, model is diverging from reference

---

Advanced: DPO + SFT Combination

Best Approach: Two-Stage

Stage 1: SFT on diverse instructions
  - 10K-100K examples
  - General instruction following
  - 12-24 hours training

Stage 2: DPO on preferences
  - 5K-50K preference pairs
  - Align to human preferences
  - 12-24 hours training

Result:
  - Better instruction following (SFT)
  - Better aligned to preferences (DPO)
  - Total quality: 30-40% improvement
  - Total time: 24-48 hours (1-2 days)

Cost: $100-500 (8-16 GPU hours)

---

Key Takeaways

⚡ DPO: 3-5x faster than RLHF, simpler, often better quality
📊 No separate reward model: Extract reward from model logits
🔄 Preference pairs: (Prompt, Chosen, Rejected) format
📈 SFT + DPO: Best two-stage pipeline for alignment
💡 Beta parameter: Controls preference strength (typical 0.5)

---

Related Notes in Finetuning Subdirectory

• Instruction Tuning & Sft - Foundation for DPO
• Fine Tuning Fundamentals - Overview
• Fine Tuning Best Practices - Tips and tricks
• Rlhf - In main Modeling directory (traditional approach)