Instruction Tuning & SFT

Overview

Instruction Tuning (Supervised Fine-Tuning/SFT) teaches models to follow instructions through supervised learning. SFT is the foundation for most fine-tuning, used before RLHF and DPO.

• Goal: Model learns to understand and follow instructions
• Data: (Instruction, Input, Output) triplets
• Result: Better instruction following, improved generalization
• Foundation: Base for RLHF, DPO, and other alignment techniques

---

Why Instruction Tuning Matters

Pre-trained vs Instruction-Tuned

Pre-trained model (e.g., raw LLaMA):
 - Optimized for: Next-token prediction
 - Training data: Diverse internet text
 - Behavior: Generates continuations, not responses

Problem:
Query: "Classify the sentiment: This movie is great!"
Output: "This movie is great! It's one of the best..."
 - Continues text, doesn't classify!

Instruction-tuned model (e.g., Llama 2-Chat):
 - Optimized for: Following instructions
 - Training data: (Instruction, Input, Output) pairs
 - Behavior: Responds to instructions

Solution:
Query: "Classify the sentiment: This movie is great!"
Output: "Positive"
 - Follows instruction correctly!

Quality comparison:
Pre-trained → Instruction-tuned: 30-50% improvement typical

---

Data Format for Instruction Tuning

Standard Format

Each example: (Instruction, Input, Output)

Example 1:
{
 "instruction": "Classify the sentiment of the following text",
 "input": "This movie was amazing! I loved every minute.",
 "output": "Positive"
}

Example 2:
{
 "instruction": "Translate the following English text to French",
 "input": "Hello, how are you?",
 "output": "Bonjour, comment allez-vous?"
}

Example 3:
{
 "instruction": "Summarize the following text",
 "input": "Long article about climate change...",
 "output": "Climate change is accelerating due to greenhouse gas emissions..."
}

Format in training:

Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

Instruction

Classify the sentiment of the following text

Input

This movie was amazing!

Response

Positive

Flexibility:
 - No input: Just instruction + output
 - Multiple inputs: Complex multi-step tasks
 - Role-playing: "You are a Python expert..."
 - Whatever works for your task!

Data Collection Strategies

Strategy 1: Human annotation (best quality)
 - Hire annotators
 - Write instructions, get responses
 - High quality but expensive ($0.10-1.00 per example)
 - Typical: 1K-10K examples
 - Best for: Production systems

Strategy 2: LLM generation (quick, cheap)
 - Use GPT-4 to generate (Instruction, Output) pairs
 - Example prompt: "Generate 100 classification tasks"
 - Cost: $0.01-0.10 per example
 - Typical: 10K-100K examples
 - Quality: Good, sometimes hallucinated
 - Best for: Initial prototyping

Strategy 3: Hybrid
 - Generate with LLM
 - Filter low quality (use reward model)
 - Keep high-quality examples
 - Verify sample with humans
 - Best for: Production

Strategy 4: Existing data repurposing
 - Take datasets (Alpaca, Cleaned GPTQ, etc.)
 - Format into (Instruction, Input, Output)
 - Cost: Free
 - Typical: 50K-100K examples
 - Best for: Research

-

SFT Training

Training Process

from transformers import AutoModelForCausalLM, TrainingArguments, Trainer
from datasets import load_dataset
from peft import LoraConfig, get_peft_model

# Step 1
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")

# Step 2
lora_config = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05)
model = get_peft_model(model, lora_config)

# Step 3
dataset = load_dataset("json", data_files="instructions.jsonl")

def formatting_func(example):
 text = f"Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\n{example['instruction']}\n\n### Input:\n{example['input']}\n\n### Response:\n{example['output']}"
 return {"text": text}

dataset = dataset.map(formatting_func)

# Step 4
training_args = TrainingArguments(
 output_dir="./results",
 num_train_epochs=3,
 per_device_train_batch_size=4,
 learning_rate=2e-4,
 warmup_steps=100,
 weight_decay=0.01,
 save_total_limit=3,
 save_steps=500,
 eval_steps=500,
)

trainer = Trainer(
 model=model,
 args=training_args,
 train_dataset=dataset["train"],
 eval_dataset=dataset["validation"],
)

trainer.train()

Key Hyperparameters

Learning rate:
 - Too high (1e-3): Unstable, diverges
 - Too low (1e-6): Slow learning
 - Sweet spot: 1e-5 to 5e-5 (depends on model)
 - Default: 2e-4 for 7B models

Batch size:
 - Larger batch: Stabler gradients, better convergence
 - Memory trade-off: Larger batch needs more memory
 - Sweet spot: 4-32 (depending on GPU)
 - Gradient accumulation can help

Epochs:
 - Too few (<1): Underfitting
 - Too many (>10): Overfitting
 - Sweet spot: 2-4 epochs
 - Use early stopping to find optimal

Warmup:
 - Helps stabilize early training
 - Typical: 5-10% of training steps
 - Example: 100-500 steps for 1K-10K examples

Weight decay:
 - Prevents overfitting
 - Typical: 0.01
 - Smaller values for small datasets

-

Quality Improvement Through SFT

Capability Gains

Instruction following:
 - Before: Model ignores/misunderstands instructions
 - After: Model follows instructions consistently
 - Improvement: 20-30%

Output format:
 - Before: Random formats (lists, paragraphs, etc.)
 - After: Consistent formatting
 - Improvement: 10-20%

Reasoning:
 - Before: Shallow responses
 - After: Step-by-step reasoning
 - Improvement: 10-15%

Truthfulness:
 - Before: Hallucinations common
 - After: More grounded responses
 - Improvement: 5-10%

Total quality gain: 30-50%

Measuring SFT Quality

Metrics:

1. Instruction following accuracy
 - Does output match instruction?
 - Human evaluation: 0-1
 - Target: >0.9 (90%+)

2. Output format correctness
 - Is format as specified?
 - Can be automated
 - Target: >0.95

3. Content quality
 - Is answer correct/helpful?
 - Human evaluation needed
 - Target: Varies by task

4. Hallucination rate
 - False statements / total statements
 - Factuality check
 - Target: <0.05 (5% or less)

Typical benchmark:
Task Before SFT After SFT Improvement
─────────────────────────────────────────────────
Instruction 65% 90% +25%
Format 70% 95% +25%
Correctness 60% 80% +20%
Hallucination 20% 8% -12%

---

Common SFT Strategies

Strategy 1: Pure SFT

Train on (Instruction, Output) pairs only

Pros:
Simple
Works well
Improves instruction following

Cons:
Doesn't optimize for preference
Model might prefer wrong answer if trained on it

Strategy 2: Multi-task SFT

Train on diverse instructions across tasks

Benefits:
 - Generalization to new tasks
 - Robustness
 - Transfer learning

Example:
 - 20% classification tasks
 - 20% summarization tasks
 - 20% question answering
 - 20% reasoning tasks
 - 20% creative writing

Result:
 - Model learns diverse instructions
 - Better generalization than single-task

Strategy 3: Progressive SFT

Train on progressively harder examples

Idea: Curriculum learning

Phase 1: Simple tasks (easy instructions)
 - 1,000 examples
 - Basic instructions
 - Model learns fundamentals

Phase 2: Medium tasks (moderate instructions)
 - 5,000 examples
 - More complex instructions
 - Model builds on fundamentals

Phase 3: Hard tasks (complex instructions)
 - 10,000 examples
 - Challenging instructions
 - Model learns advanced reasoning

Result:
 - Better convergence than training on all at once
 - 5-10% quality improvement

---

SFT Foundation for Alignment

RLHF Pipeline

Step 1: SFT (base instruction following)
 - Train on (Instruction, Output) pairs
 - Model learns basic instruction following
 - Foundation for next steps

Step 2: Reward Model Training
 - Train on (Output A, Output B, Winner) triplets
 - Model learns to judge which output is better
 - Used for RL training

Step 3: RLHF (Optimize with RL)
 - Use reward model as reward signal
 - RL algorithm improves model
 - Model learns human preferences

Result: Better aligned model

DPO Alternative

Step 1: SFT (same as above)
 - Train on (Instruction, Output) pairs
 - Foundation

Step 2: DPO (Direct Preference Optimization)
 - Train on preference pairs directly
 - Combines SFT + preference learning
 - Simpler than RLHF, similarly effective

Result: Better aligned model (simpler than RLHF)

---

SFT vs Full Fine-tuning

When to Use Each

SFT (Instruction Tuning):
Goal: Better instruction following
Data: Diverse (Instruction, Output) pairs
Cost: Moderate ($100-1000)
Quality gain: 20-30%
Generalization: Good to new tasks

Task-Specific Fine-tuning:
Goal: Perfect performance on specific task
Data: Task-specific examples
Cost: Low to moderate ($50-500)
Quality gain: 10-20%
Generalization: Poor to other tasks

Combined (Best):
 - SFT on diverse instructions
 - Task-specific fine-tuning on target data
 - Quality: Best (30-40% improvement)
 - Cost: Higher ($200-1500)

-

Key Takeaways

Instruction tuning: Teaches following instructions, foundation for alignment Data format: (Instruction, Input, Output) triplets SFT improves quality 20-30% with good instruction data Foundation for RLHF/DPO: Start with SFT before preference learning Multi-task SFT: Better generalization than single-task

-

Related Notes in Finetuning Subdirectory

• Fine Tuning Fundamentals - Overview
• Dpo (Direct Preference Optimization) - Preference-based improvement
• Domain Specific Fine Tuning - Task-specific adaptation
• Fine Tuning Best Practices - Tips and tricks
• Rlhf - In main Modeling directory