Unsloth¶

Quick Facts¶

Aspect	Details
Organization	Open Source
Purpose	2-5x faster LoRA training
License	Apache 2.0
Installation	`pip install unsloth[colab-new] @ git+...`
Best For	Speed-critical projects, budget-conscious
Hardware	Any GPU, optimized for RTX 4090

What It Does¶

Unsloth is a drop-in replacement for Hugging Face that:

2-5x faster LoRA training
80% less VRAM consumption
Same quality output as standard training
Simple API - minimal code changes

Installation¶

# For Colab
pip install unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git

# For local setup
pip install unsloth @ git+https://github.com/unslothai/unsloth.git

Basic Usage¶

from unsloth import FastLanguageModel
import torch

# Load model with Unsloth
model, tokenizer = FastLanguageModel.from_pretrained(
 model_name="unsloth/llama-2-7b-bnb-4bit",
 max_seq_length=2048,
 dtype=torch.float16,
 load_in_4bit=True,
)

# Add LoRA - 2-5x faster!
model = FastLanguageModel.get_peft_model(
 model,
 r=16,
 lora_alpha=32,
 lora_dropout=0.05,
 bias="none",
 use_gradient_checkpointing="unsloth",
 random_state=3407,
)

# Standard Hugging Face trainer (unchanged!)
from transformers import TrainingArguments, SFTTrainer

trainer = SFTTrainer(
 model=model,
 tokenizer=tokenizer,
 train_dataset=dataset,
 dataset_text_field="text",
 max_seq_length=2048,
 args=TrainingArguments(
 output_dir="./output",
 num_train_epochs=1,
 per_device_train_batch_size=4,
 gradient_accumulation_steps=4,
 warmup_steps=5,
 learning_rate=2e-4,
 fp16=True,
 logging_steps=1,
 optim="adamw_8bit",
 weight_decay=0.01,
 lr_scheduler_type="linear",
 seed=3407,
),
)

trainer.train()

Memory Comparison¶

Training Llama 2 7B:

Method	VRAM	Speed	Quality
Standard LoRA	18GB	1x	100%
Unsloth LoRA	3.5GB	3x	100%
TRL QLoRA	4GB	1x	98%
Unsloth QLoRA	2GB	4x	98%

Speed Benchmarks¶

Model: Llama 2 7B, Batch Size: 4, Seq Len: 2048

Standard LoRA:
 Forward pass: 2.5s
 Backward pass: 2.5s
 Total: 5s per step

Unsloth LoRA:
 Forward pass: 0.6s
 Backward pass: 0.7s
 Total: 1.3s per step
 Speedup: 3.8x

Inference with Trained Model¶

# Load trained model
from unsloth import FastLanguageModel
model, tokenizer = FastLanguageModel.from_pretrained(
 model_name="./output",
 max_seq_length=2048,
 dtype=torch.float16,
 load_in_4bit=True,
)

# Prepare for inference (minor optimization)
FastLanguageModel.for_inference(model)

# Generate
inputs = tokenizer("What is AI?", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_length=100)
print(tokenizer.decode(outputs[0]))

Merging LoRA¶

# Merge LoRA adapters into base model
model.save_pretrained_merged(
 "merged_model",
 tokenizer,
 save_method="merged_16bit" # or "merged_4bit"
)

# Now model is a standard Hugging Face model
# Can be used with any framework!

Supported Models¶

Llama 2 (7B, 13B, 70B)
Mistral 7B
Phi series
Gemma series
Many others (check repo)

Strengths¶

Speed - 2-5x faster than standard training Memory - 80% reduction in VRAM Simplicity - Drop-in replacement, no code changes Quality - Exact same output quality Cost - Train on RTX 4090 instead of A100

Weaknesses¶

Limited Methods - LoRA only, no full training RLHF - Not optimized for PPO/preference training Bleeding Edge - Actively developed, might have issues

Best Practices¶

Use Quantization - 4-bit further reduces memory
Gradient Checkpointing - Already enabled by default
Merge After Training - For deployment simplicity
Batch Size Tuning - Memory savings enable larger batches

Typical Training Time¶

Dataset: 10K examples, Llama 2 7B, 1 epoch

Hardware:
 RTX 4090 (24GB): ~2-3 hours with Unsloth
 A100 (80GB): ~1.5-2 hours standard LoRA

Result: Unsloth on RTX 4090 ≈ A100 standard training

When to Use Unsloth¶

Scenario	Recommendation
Speed critical	Best choice
Budget limited	Best choice
Quick iteration	Best choice
Consumer GPU	Best choice
RLHF training	Use TRL
Full fine-tuning	Use Axolotl

Resources¶

-

Next: Compare with 04 Ludwig for no-code training