TRL: Transformer Reinforcement Learning

Quick Facts

Aspect	Details
Organization	Hugging Face
Purpose	Fine-tuning and RLHF training
License	Apache 2.0 (Open Source)
Installation	pip install trl
Best For	RLHF workflows, production training
Community	Large, well-maintained

What It Does

TRL (Trainer Reinforcement Learning) is Hugging Face's complete library for training LLMs with: - Supervised Fine-Tuning (SFT) - Reward Model Training - PPO (Proximal Policy Optimization) - DPO (Direct Preference Optimization) - QLoRA integration for memory efficiency

Installation

pip install trl
pip install transformers datasets peft torch

Core Features

1. Supervised Fine-Tuning (SFT)

from trl import SFTTrainer
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments
from datasets import load_dataset

# Load model
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")

# Load dataset
dataset = load_dataset("imdb", split="train[:1000]")

# Configure training
training_args = TrainingArguments(
    output_dir="./sft-model",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    save_steps=100,
    logging_steps=10,
)

# Create trainer
trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    tokenizer=tokenizer,
    dataset_text_field="text",
    max_seq_length=512,
    args=training_args,
)

# Train
trainer.train()

# Save
trainer.save_model("./my-sft-model")

2. Reward Model Training

from trl import RewardTrainer
from transformers import AutoModelForSequenceClassification, AutoTokenizer

# Load base model
model = AutoModelForSequenceClassification.from_pretrained(
    "meta-llama/Llama-2-7b-hf", 
    num_labels=1
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")

# Create trainer
reward_trainer = RewardTrainer(
    model=model,
    tokenizer=tokenizer,
    args=TrainingArguments(...),
    train_dataset=reward_dataset,
)

# Train
reward_trainer.train()

3. PPO Training

from trl import PPOTrainer, PPOConfig
from transformers import AutoModelForCausalLM, AutoTokenizer

config = PPOConfig(
    model_name="meta-llama/Llama-2-7b-hf",
    learning_rate=1e-5,
    num_ppo_epochs=4,
    mini_batch_size=4,
    batch_size=32,
)

model = AutoModelForCausalLM.from_pretrained(config.model_name)
tokenizer = AutoTokenizer.from_pretrained(config.model_name)
reward_model = AutoModelForSequenceClassification.from_pretrained("./reward-model")

ppo_trainer = PPOTrainer(
    config=config,
    model=model,
    ref_model=None,
    tokenizer=tokenizer,
    dataset=dataset,
    data_collator=data_collator,
)

# Training loop
for epoch in range(10):
    for batch in ppo_trainer.dataloader:
        query_tensors = batch["input_ids"]
        response_tensors = ppo_trainer.generate(query_tensors)

        # Get reward from reward model
        rewards = reward_model(response_tensors)["score"]

        # Update policy with PPO
        stats = ppo_trainer.step(query_tensors, response_tensors, rewards)
        ppo_trainer.log_stats(stats, batch, rewards)

4. DPO (Direct Preference Optimization)

from trl import DPOTrainer, DPOConfig
from transformers import AutoModelForCausalLM, AutoTokenizer

config = DPOConfig(
    output_dir="./dpo-model",
    per_device_train_batch_size=4,
    learning_rate=5e-5,
    num_train_epochs=3,
    beta=0.1,  # Temperature parameter
)

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")

trainer = DPOTrainer(
    model=model,
    args=config,
    train_dataset=dpo_dataset,  # Dataset with preferred/rejected pairs
    tokenizer=tokenizer,
    peft_config=peft_config,
)

trainer.train()

Memory Efficiency with QLoRA

from peft import LoraConfig, get_peft_model
from transformers import BitsAndBytesConfig
import torch

# 4-bit quantization
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)

# Load quantized model
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    quantization_config=bnb_config,
    device_map="auto",
)

# Add LoRA
lora_config = LoraConfig(
    r=8,
    lora_alpha=16,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
)

model = get_peft_model(model, lora_config)

# Now train with SFTTrainer
# Memory usage: ~4GB (vs 28GB for full training)

Strengths

✅ RLHF Support - Complete pipeline for preference training
✅ Proven - Used by Hugging Face and industry leaders
✅ Flexible - Supports multiple training paradigms
✅ Well-Documented - Extensive examples and tutorials
✅ Active Development - Regular updates and improvements

Weaknesses

❌ Memory Usage - Without QLoRA, needs substantial GPU memory
❌ Complexity - PPO training has many hyperparameters
❌ Speed - Slower than optimized alternatives like Unsloth

Best Practices

1. Start with SFT - Verify data quality before RLHF
2. Use QLoRA - Reduces memory 80% with minimal quality loss
3. Validate Rewards - Reward model quality critical for PPO
4. Monitor Training - Use Weights & Biases integration for tracking

Integration

# With Weights & Biases
from trl import SFTTrainer
from transformers import TrainingArguments

args = TrainingArguments(
    report_to="wandb",  # Log to Weights & Biases
    run_name="my-sft-run",
)

trainer = SFTTrainer(..., args=args)
trainer.train()

When to Use TRL

Scenario	Recommendation
RLHF training	✅ Best choice
Production fine-tuning	✅ Good choice
Speed critical	❌ Use Unsloth
No coding	❌ Use Ludwig
Multi-method flexibility	⚠️ Use Axolotl

Resources

• Official Docs
• GitHub
• Examples

---

Next: Compare with 02 Axolotl for flexible multi-method training