Instruction Tuning & SFT: Teaching Models to Follow Instructions¶

Overview¶

Instruction Tuning (Supervised Fine-Tuning/SFT) teaches models to follow instructions through supervised learning. SFT is the foundation for most fine-tuning, used before RLHF and DPO.

Goal: Model learns to understand and follow instructions
Data: (Instruction, Input, Output) triplets
Result: Better instruction following, improved generalization
Foundation: Base for RLHF, DPO, and other alignment techniques

Why Instruction Tuning Matters¶

Pre-trained vs Instruction-Tuned¶

Pre-trained model (e.g., raw LLaMA):
  - Optimized for: Next-token prediction
  - Training data: Diverse internet text
  - Behavior: Generates continuations, not responses

Problem:
Query: "Classify the sentiment: This movie is great!"
Output: "This movie is great! It's one of the best..."
  - Continues text, doesn't classify!

Instruction-tuned model (e.g., Llama 2-Chat):
  - Optimized for: Following instructions
  - Training data: (Instruction, Input, Output) pairs
  - Behavior: Responds to instructions

Solution:
Query: "Classify the sentiment: This movie is great!"
Output: "Positive"
  - Follows instruction correctly!

Quality comparison:
Pre-trained → Instruction-tuned: 30-50% improvement typical

Data Format for Instruction Tuning¶

Standard Format¶

Each example: (Instruction, Input, Output)

Example 1:
{
  "instruction": "Classify the sentiment of the following text",
  "input": "This movie was amazing! I loved every minute.",
  "output": "Positive"
}

Example 2:
{
  "instruction": "Translate the following English text to French",
  "input": "Hello, how are you?",
  "output": "Bonjour, comment allez-vous?"
}

Example 3:
{
  "instruction": "Summarize the following text",
  "input": "Long article about climate change...",
  "output": "Climate change is accelerating due to greenhouse gas emissions..."
}

Format in training:

Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

Instruction:¶

Classify the sentiment of the following text

Input:¶

This movie was amazing!

Response:¶

Positive

Flexibility:
  - No input: Just instruction + output
  - Multiple inputs: Complex multi-step tasks
  - Role-playing: "You are a Python expert..."
  - Whatever works for your task!

Data Collection Strategies¶

Strategy 1: Human annotation (best quality)
  - Hire annotators
  - Write instructions, get responses
  - High quality but expensive ($0.10-1.00 per example)
  - Typical: 1K-10K examples
  - Best for: Production systems

Strategy 2: LLM generation (quick, cheap)
  - Use GPT-4 to generate (Instruction, Output) pairs
  - Example prompt: "Generate 100 classification tasks"
  - Cost: $0.01-0.10 per example
  - Typical: 10K-100K examples
  - Quality: Good, sometimes hallucinated
  - Best for: Initial prototyping

Strategy 3: Hybrid
  - Generate with LLM
  - Filter low quality (use reward model)
  - Keep high-quality examples
  - Verify sample with humans
  - Best for: Production

Strategy 4: Existing data repurposing
  - Take datasets (Alpaca, Cleaned GPTQ, etc.)
  - Format into (Instruction, Input, Output)
  - Cost: Free
  - Typical: 50K-100K examples
  - Best for: Research

SFT Training¶

Training Process¶

from transformers import AutoModelForCausalLM, TrainingArguments, Trainer
from datasets import load_dataset
from peft import LoraConfig, get_peft_model

# Step 1: Load model
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")

# Step 2: (Optional) Add LoRA for efficiency
lora_config = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05)
model = get_peft_model(model, lora_config)

# Step 3: Load and format data
dataset = load_dataset("json", data_files="instructions.jsonl")

def formatting_func(example):
    text = f"Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\n{example['instruction']}\n\n### Input:\n{example['input']}\n\n### Response:\n{example['output']}"
    return {"text": text}

dataset = dataset.map(formatting_func)

# Step 4: Training
training_args = TrainingArguments(
    output_dir="./results",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    learning_rate=2e-4,
    warmup_steps=100,
    weight_decay=0.01,
    save_total_limit=3,
    save_steps=500,
    eval_steps=500,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=dataset["train"],
    eval_dataset=dataset["validation"],
)

trainer.train()

Key Hyperparameters¶

Learning rate:
  - Too high (1e-3): Unstable, diverges
  - Too low (1e-6): Slow learning
  - Sweet spot: 1e-5 to 5e-5 (depends on model)
  - Default: 2e-4 for 7B models

Batch size:
  - Larger batch: Stabler gradients, better convergence
  - Memory trade-off: Larger batch needs more memory
  - Sweet spot: 4-32 (depending on GPU)
  - Gradient accumulation can help

Epochs:
  - Too few (<1): Underfitting
  - Too many (>10): Overfitting
  - Sweet spot: 2-4 epochs
  - Use early stopping to find optimal

Warmup:
  - Helps stabilize early training
  - Typical: 5-10% of training steps
  - Example: 100-500 steps for 1K-10K examples

Weight decay:
  - Prevents overfitting
  - Typical: 0.01
  - Smaller values for small datasets

Quality Improvement Through SFT¶

Capability Gains¶

Instruction following:
  - Before: Model ignores/misunderstands instructions
  - After: Model follows instructions consistently
  - Improvement: 20-30%

Output format:
  - Before: Random formats (lists, paragraphs, etc.)
  - After: Consistent formatting
  - Improvement: 10-20%

Reasoning:
  - Before: Shallow responses
  - After: Step-by-step reasoning
  - Improvement: 10-15%

Truthfulness:
  - Before: Hallucinations common
  - After: More grounded responses
  - Improvement: 5-10%

Total quality gain: 30-50%

Measuring SFT Quality¶

Metrics:

1. Instruction following accuracy
  - Does output match instruction?
  - Human evaluation: 0-1
  - Target: >0.9 (90%+)

2. Output format correctness
  - Is format as specified?
  - Can be automated
  - Target: >0.95

3. Content quality
  - Is answer correct/helpful?
  - Human evaluation needed
  - Target: Varies by task

4. Hallucination rate
  - False statements / total statements
  - Factuality check
  - Target: <0.05 (5% or less)

Typical benchmark:
Task           Before SFT   After SFT   Improvement
─────────────────────────────────────────────────
Instruction    65%          90%         +25%
Format         70%          95%         +25%
Correctness    60%          80%         +20%
Hallucination  20%          8%          -12%

Common SFT Strategies¶

Strategy 1: Pure SFT¶

Train on (Instruction, Output) pairs only

Pros:
✅ Simple
✅ Works well
✅ Improves instruction following

Cons:
❌ Doesn't optimize for preference
❌ Model might prefer wrong answer if trained on it

Strategy 2: Multi-task SFT¶

Train on diverse instructions across tasks

Benefits:
  - Generalization to new tasks
  - Robustness
  - Transfer learning

Example:
  - 20% classification tasks
  - 20% summarization tasks
  - 20% question answering
  - 20% reasoning tasks
  - 20% creative writing

Result:
  - Model learns diverse instructions
  - Better generalization than single-task

Strategy 3: Progressive SFT¶

Train on progressively harder examples

Idea: Curriculum learning

Phase 1: Simple tasks (easy instructions)
  - 1,000 examples
  - Basic instructions
  - Model learns fundamentals

Phase 2: Medium tasks (moderate instructions)
  - 5,000 examples
  - More complex instructions
  - Model builds on fundamentals

Phase 3: Hard tasks (complex instructions)
  - 10,000 examples
  - Challenging instructions
  - Model learns advanced reasoning

Result:
  - Better convergence than training on all at once
  - 5-10% quality improvement

SFT Foundation for Alignment¶

RLHF Pipeline¶

Step 1: SFT (base instruction following)
  - Train on (Instruction, Output) pairs
  - Model learns basic instruction following
  - Foundation for next steps

Step 2: Reward Model Training
  - Train on (Output A, Output B, Winner) triplets
  - Model learns to judge which output is better
  - Used for RL training

Step 3: RLHF (Optimize with RL)
  - Use reward model as reward signal
  - RL algorithm improves model
  - Model learns human preferences

Result: Better aligned model

DPO Alternative¶

Step 1: SFT (same as above)
  - Train on (Instruction, Output) pairs
  - Foundation

Step 2: DPO (Direct Preference Optimization)
  - Train on preference pairs directly
  - Combines SFT + preference learning
  - Simpler than RLHF, similarly effective

Result: Better aligned model (simpler than RLHF)

SFT vs Full Fine-tuning¶

When to Use Each¶

SFT (Instruction Tuning):
✅ Goal: Better instruction following
✅ Data: Diverse (Instruction, Output) pairs
✅ Cost: Moderate ($100-1000)
✅ Quality gain: 20-30%
✅ Generalization: Good to new tasks

Task-Specific Fine-tuning:
✅ Goal: Perfect performance on specific task
✅ Data: Task-specific examples
✅ Cost: Low to moderate ($50-500)
✅ Quality gain: 10-20%
✅ Generalization: Poor to other tasks

Combined (Best):
  - SFT on diverse instructions
  - Task-specific fine-tuning on target data
  - Quality: Best (30-40% improvement)
  - Cost: Higher ($200-1500)

Key Takeaways¶

📚 Instruction tuning: Teaches following instructions, foundation for alignment
📊 Data format: (Instruction, Input, Output) triplets
🎯 SFT improves quality 20-30% with good instruction data
🔄 Foundation for RLHF/DPO: Start with SFT before preference learning
💡 Multi-task SFT: Better generalization than single-task

Fine Tuning Fundamentals - Overview
Dpo (Direct Preference Optimization) - Preference-based improvement
Domain Specific Fine Tuning - Task-specific adaptation
Fine Tuning Best Practices - Tips and tricks
Rlhf - In main Modeling directory