LoRA: Low-Rank Adaptation - Complete Technical Guide

Overview

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning technique that adapts large language models by adding small trainable matrices (adapters) to existing weights, rather than fine-tuning all parameters. It reduces memory usage and training time by 60-90% while maintaining comparable performance.

• Paper: "LoRA: Low-Rank Adaptation of Large Language Models"
• Authors: Hu et al., Microsoft (2021)
• Key Innovation: Add trainable low-rank matrices instead of modifying all weights
• Impact: 60-90% parameter reduction, 3-5x faster training
• Adoption: Becomes industry standard (HuggingFace, vLLM, Ollama)

---

The Problem: Full Fine-Tuning is Expensive

Standard Fine-Tuning Challenges

Full Fine-Tuning (Traditional):

Model: Llama 2 7B
  - Total parameters: 7 billion
  - In float32: 28 GB
  - Batch size: 8
  - Sequence length: 2048

Memory required:
  - Model weights: 28 GB
  - Gradients (backward): 28 GB (same size)
  - Optimizer states (Adam): 56 GB (2x for momentum + variance)
  - Activations (forward): ~8 GB
  - Total: ~120 GB ❌ Needs 8x A100 GPUs!

Cost:
  - 8x A100 GPU: $600/hour
  - 24-hour training: $14,400
  - Plus setup, cooling, power
  - Total project cost: $20,000+

Training time:
  - For 1 epoch on 50K examples: 24 hours
  - Multiple epochs needed: 48-72 hours
  - Total: 2-3 days (slow!)

Comparison: Llama 2 13B
  - 13 billion parameters
  - Memory: 26 × (13/7) ≈ 52 GB per parameter
  - Total: 260 GB! ❌ Needs 20+ A100s (not practical!)

Why Fine-Tuning All Parameters Doesn't Work

Insight from research:
"Pre-trained models have intrinsic dimensionality!"

When fine-tuning, changes to weights aren't random:
  - Change matrix: ΔW (size: d_out × d_in)
  - But ΔW has low rank!
  - Intrinsic rank: ~4-8 (much less than d_out or d_in)

Example:
Weight matrix: 4096 × 4096 (16M parameters)
Change during fine-tuning: 4096 × 4096 (same size)
But effective rank of change: ~6!

This means:
  - We're using 16M parameters
  - But only ~6 × (4096 + 4096) = 49K effective parameters!
  - We're being wasteful!

LoRA exploits this insight!

---

LoRA: The Solution

Core Idea: Low-Rank Decomposition

Standard fine-tuning:
W_new = W_original + ΔW  (all parameters trainable)

LoRA approach:
W_new = W_original + B × A (only B, A trainable!)

Where:
  - W_original: Pre-trained weight matrix (d_out × d_in), frozen
  - A: Small trainable matrix (d_in × r), where r ≪ d_in
  - B: Small trainable matrix (d_out × r), where r ≪ d_out
  - r: Rank (typically 8-64)

Forward pass:
h = W_original @ x  (original, frozen)
h_lora = B @ (A @ x)  (adapter, trainable)
output = h + α/r * h_lora  (combined)

Parameter count:
Standard: d_out × d_in
LoRA: (d_out × r) + (d_in × r) = r × (d_out + d_in)

Reduction:
For d_out = d_in = 4096, r = 8:
  - Standard: 16M parameters
  - LoRA: 8 × (4096 + 4096) = 64K parameters
  - Reduction: 16M / 64K = 250x fewer parameters!

Visual Representation

Standard Weight Matrix (4096 × 4096):
- ┌────────────────────────────────────┐
│                                    │
    - W_original (frozen)               │
    - 16 million parameters             │
│                                    │
  - ┘
                 ↓
          Fine-tuning updates
                 ↓
- ┌────────────────────────────────────┐
    - W_original + ΔW (all trainable)   │
    - 16 million new parameters!        │
  - ┘


LoRA Approach:
- ┌────────────────────────────────────┐
    - W_original (frozen)               │
    - 16 million parameters (not touched)│
  - ┘
                 +
- ┌──────────────────┐
    - A (4096 × 8)     │ 32K params
    - 256×256 image    │
  - ┘
              ↓
         Matrix multiply
              ↓
- ┌──────────────────┐
    - B (8 × 4096)     │ 32K params
    - 8×256 image      │
  - ┘
                 =
        B @ A (output 4096×4096)
           Only 64K trainable!

---

Mathematical Foundation

Rank in Matrix Decomposition

Any matrix can be decomposed:
W ≈ U × S × V^T

Where:
  - U: d_out × r (left singular vectors)
  - S: r × r (singular values)
  - V: d_in × r (right singular vectors)
  - r: rank (number of significant singular values)

Low-rank approximation:
Keep only top r singular values/vectors
Discard small singular values (correspond to noise)

Example:
Original matrix: 4096 × 4096 (many singular values)
Low-rank approx with r=8:
  - Keep: Top 8 singular values
  - Approximate: ≈ U_r @ diag(S_r) @ V_r^T
  - Error: Small if original has low intrinsic rank
  - Parameters: r × (4096 + 4096) instead of 4096²

LoRA implementation:
B @ A approximates: U × S × V^T
  - B ≈ U × sqrt(S)
  - A ≈ sqrt(S) × V^T
  - Or just: B @ A (learned directly)

Why Low-Rank Works for Fine-Tuning

Hypothesis (verified empirically):
"Weight changes during fine-tuning have low rank"

Evidence:
1. Intrinsic dimensionality is small
  - Changes concentrate in few directions
  - Not uniformly distributed

2. Task adaptation is constrained
  - Different tasks share similar structure
  - Changes are along few principal directions
  - Don't need full rank to capture changes

3. Pre-training captures general knowledge
  - Task-specific changes are secondary
  - Only need to adjust projection
  - Doesn't require changing entire manifold

Empirical validation:
For Llama 7B fine-tuned on different tasks:
  - Rank 1: Captures 15-20% of change
  - Rank 4: Captures 50-60% of change
  - Rank 8: Captures 75-85% of change
  - Rank 16: Captures 90-95% of change
  - Rank 32+: Diminishing returns

Conclusion: Rank 8-16 usually sufficient!

---

LoRA in Practice: Implementation

Training Setup

import torch
from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers import Trainer, TrainingArguments

# 1. Load model
model_name = "meta-llama/Llama-2-7b-hf"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.float16,
    device_map="auto"
)

# 2. Configure LoRA
lora_config = LoraConfig(
    r=8,                          # Rank
    lora_alpha=16,                # Scaling factor
    target_modules=["q_proj", "v_proj"],  # Which modules to adapt
    lora_dropout=0.05,            # Dropout for regularization
    bias="none",                  # "none", "all", or "lora_only"
    task_type="CAUSAL_LM"         # Task type
)

# 3. Apply LoRA
model = get_peft_model(model, lora_config)

# 4. Check parameters
print(model.print_trainable_parameters())
# Expected output:
# trainable params: 4,194,304 || all params: 6,738,415,616 || trainable%: 0.06

# 5. Prepare data
tokenizer = AutoTokenizer.from_pretrained(model_name)

dataset = load_dataset("your_dataset")  # Your fine-tuning data

def tokenize_function(examples):
    return tokenizer(
        examples["text"],
        padding="max_length",
        truncation=True,
        max_length=512
    )

tokenized_dataset = dataset.map(tokenize_function, batched=True)

# 6. Training configuration
training_args = TrainingArguments(
    output_dir="./llama-7b-lora",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    per_device_eval_batch_size=4,
    warmup_steps=100,
    weight_decay=0.01,
    logging_steps=10,
    learning_rate=1e-4,
    save_steps=500,
    eval_steps=500,
)

# 7. Trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset["train"],
    eval_dataset=tokenized_dataset["validation"],
)

# 8. Train!
trainer.train()

# 9. Save LoRA weights
model.save_pretrained("./llama-7b-lora-final")

Memory Comparison

Llama 2 7B Full Fine-Tuning vs LoRA:

                          Full FT      LoRA      Reduction
─────────────────────────────────────────────────────────
Model weights            28 GB        28 GB     0%
Gradients                28 GB        0.01 GB   99.96%
Optimizer states         56 GB        0.02 GB   99.96%
LoRA weights             -            0.1 GB    -
Activations              8 GB         8 GB      0%
─────────────────────────────────────────────────────────
Total peak memory        ~120 GB      ~37 GB    69% reduction!
Training time            24 hours     8 hours   3x faster
GPU required             8×A100       1×A100    8x fewer!

---

LoRA Configuration Details

Rank Selection

Rank (r) determines adapter size:

r = 2:
  - Parameters: 2 × (4096 + 4096) = 16K
  - Memory: 64 KB (float16)
  - Capacity: Very limited
  - Use case: Mobile, extreme constraints
  - Accuracy: Lower (~2-3% loss)

r = 8 (common default):
  - Parameters: 8 × 8192 = 64K
  - Memory: 256 KB (float16)
  - Capacity: Good balance
  - Use case: Most fine-tuning tasks
  - Accuracy: Good (~0.5-1% loss)

r = 16:
  - Parameters: 16 × 8192 = 128K
  - Memory: 512 KB (float16)
  - Capacity: High
  - Use case: Complex domain adaptation
  - Accuracy: Better (~0.1-0.5% loss)

r = 32:
  - Parameters: 32 × 8192 = 256K
  - Memory: 1 MB (float16)
  - Capacity: Very high
  - Use case: Specialized tasks
  - Accuracy: Excellent (≈0% loss)

Recommendation:
  - Start with r=8
  - If accuracy insufficient → r=16
  - If still insufficient → r=32
  - Only rarely need r > 32

Alpha (Scaling Factor)

Purpose: Controls LoRA contribution strength

Formula:
output = original_output + (alpha / r) * lora_output

Effects:
alpha = 1:
  - Minimal contribution
  - LoRA has little effect
  - Safe but might underfit

alpha = 16 (lora_alpha=16, typical):
  - Scaling: (16 / 8) = 2x amplification
  - Good balance
  - Works well for most tasks

alpha = 32:
  - Scaling: (32 / 8) = 4x amplification
  - Strong LoRA contribution
  - Good for domain-specific tasks

Rule of thumb:
lora_alpha ≈ 2 × r

So:
  - r=4 → alpha=8
  - r=8 → alpha=16
  - r=16 → alpha=32
  - r=32 → alpha=64

Target Modules

Which modules to apply LoRA to:

Full application (most parameters):
  - q_proj: Query projection
  - k_proj: Key projection
  - v_proj: Value projection
  - o_proj: Output projection
  - ff: Feedforward layers
Result: Biggest LoRA adapters, most flexibility

Common selection (good balance):
  - q_proj, v_proj: 60-70% of benefit
  - ff (sometimes): Additional benefit
Result: 70% parameter reduction while maintaining quality

Minimal (smallest adapters):
  - v_proj only: Minimum necessary
  - Result: Smallest size but least flexible

Strategic selection:
  - Task-specific: Adapt based on task
  - Attention-heavy: Focus on attention modules
  - Dense-heavy: Focus on feedforward
  - Balanced: Use all modules

Recommendation: Start with [q_proj, v_proj]

LoRA Dropout

Dropout probability for LoRA (not model dropout)

lora_dropout = 0.0 (no dropout):
  - Maximum capacity
  - Might overfit
  - Use for large datasets

lora_dropout = 0.05 (default):
  - Light regularization
  - Good balance
  - Recommended for most cases

lora_dropout = 0.1 (strong regularization):
  - Helps with small datasets
  - Reduces overfitting
  - Use for few-shot learning

---

LoRA Usage: Inference and Merging

Inference with LoRA

from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer

# Method 1: Load LoRA directly
model = AutoPeftModelForCausalLM.from_pretrained(
    "./llama-7b-lora-final",
    device_map="auto",
    torch_dtype=torch.float16
)

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")

# Inference (LoRA weights automatically applied)
inputs = tokenizer("Explain quantum computing", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0]))

# Method 2: Load base model + LoRA weights
from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    device_map="auto",
    torch_dtype=torch.float16
)

model = PeftModel.from_pretrained(
    base_model,
    "./llama-7b-lora-final"
)

# Same inference
outputs = model.generate(**inputs, max_new_tokens=256)

Merging LoRA Weights

from peft import AutoPeftModelForCausalLM

# Load LoRA model
model = AutoPeftModelForCausalLM.from_pretrained(
    "./llama-7b-lora-final"
)

# Merge LoRA weights into base model
merged_model = model.merge_and_unload()

# Save merged model (full size again, but with LoRA adaptations)
merged_model.save_pretrained("./llama-7b-merged")

# Use merged model (no dependency on LoRA anymore)
outputs = merged_model.generate(**inputs, max_new_tokens=256)

Benefits of Merging

When to merge:

Merging pros:
✓ Single model file (no base + adapter)
✓ Faster inference (no adapter computation)
✓ Easy distribution
✓ Compatible with any framework

Merging cons:
✗ Loses original model (LoRA not separate)
✗ Larger file size (combined)
✗ Can't switch adapters
✗ Training overhead in merged form

Keep separate pros:
✓ Easy to share/distribute
✓ Small adapter files (easy transfer)
✓ Can load multiple adapters
✓ Can switch between adapters
✓ Can keep original model unchanged

Keep separate cons:
✗ Need base model + adapter
✗ Slightly slower inference
✗ More complex deployment

Recommendation:
  - Development: Keep separate
  - Production (single adapter): Merge
  - Production (multiple adapters): Keep separate

---

Multi-LoRA: Combining Multiple Adapters

Stacking LoRA Adapters

from peft import LoraConfig, PeftModel, get_peft_model

# Start with base model
base_model = AutoModelForCausalLM.from_pretrained(model_name)

# Apply first LoRA (domain adaptation)
lora_config_1 = LoraConfig(
    r=8,
    lora_alpha=16,
    target_modules=["q_proj", "v_proj"],
    task_type="CAUSAL_LM",
    bias="none"
)
model = get_peft_model(base_model, lora_config_1)

# Train on domain data
# ... training code ...
model.save_pretrained("./domain-lora")

# Load model with first LoRA
model = AutoPeftModelForCausalLM.from_pretrained("./domain-lora")

# Apply second LoRA (task-specific)
lora_config_2 = LoraConfig(
    r=8,
    lora_alpha=16,
    target_modules=["q_proj", "v_proj"],
    task_type="CAUSAL_LM",
    bias="none"
)

# Stack second adapter
model = PeftModel.from_pretrained(
    model.base_model,  # Get base model
    "./domain-lora",
    adapter_name="domain"
)
model.add_adapter("task", lora_config_2)

# Train second adapter on task data
# ... training code ...
model.train_adapter("task")
model.save_pretrained("./domain-task-lora")

# Use either adapter
model.set_active_adapters(["domain"])  # Just domain
model.set_active_adapters(["task"])    # Just task
model.set_active_adapters(["domain", "task"])  # Combined!

---

LoRA Variations

QLoRA (Quantized LoRA)

Idea: Combine LoRA with quantization

Typical LoRA (full model in float32):
  - Base model: 28 GB (not trainable)
  - LoRA adapters: 0.1 GB (trainable)
  - Total: 28 GB in memory

QLoRA (quantized model + LoRA):
  - Base model: 7 GB (int4 quantized, not trainable)
  - LoRA adapters: 0.1 GB (float16, trainable)
  - Total: 7 GB in memory!

Result: 4x memory reduction!

Implementation:
```python
from peft import prepare_model_for_kbit_training
from transformers import BitsAndBytesConfig

# Quantize model
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
)

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=bnb_config,
)

# Prepare for training
model = prepare_model_for_kbit_training(model)

# Apply LoRA as normal
lora_config = LoraConfig(
    r=8,
    lora_alpha=16,
    target_modules=["q_proj", "v_proj"],
    task_type="CAUSAL_LM"
)
model = get_peft_model(model, lora_config)

# Train!
trainer = Trainer(model=model, ...)
trainer.train()

Memory usage: Llama 7B + LoRA + INT4 - Without QLoRA: 28 + 0.1 = 28.1 GB (needs 2×A100s) - With QLoRA: 7 + 0.1 = 7.1 GB (fits on 1×A100!) - Reduction: 75%!

### DoRA (Decomposed LoRA)

Improvement on LoRA: Separate magnitude and direction

Standard LoRA: W' = W + BA (combined change)

DoRA: W' = (m / ||W||) × W + BA

Where: - m: Magnitude scalar (learnable) - W: Normalized original weight - BA: Direction adaptation

Benefits: ✓ Better expressiveness ✓ Faster convergence ✓ Higher accuracy - Cost: Slightly more parameters (just scalars)

When to use: - If LoRA accuracy insufficient - Complex domain adaptation - Multi-task learning - Extra 0.01% parameters worth benefit

---

## Practical Examples

### Example 1: Fine-tune on Custom Dataset

```python
# Complete fine-tuning example

from datasets import load_dataset, DatasetDict
from transformers import AutoModelForCausalLM, AutoTokenizer, Trainer, TrainingArguments
from peft import LoraConfig, get_peft_model
import torch

# 1. Load model and tokenizer
model_name = "meta-llama/Llama-2-7b-hf"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.float16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# 2. Prepare dataset
# Assume you have train.jsonl, validation.jsonl with "text" field
dataset = load_dataset("json", data_files={
    "train": "train.jsonl",
    "validation": "validation.jsonl"
})

def tokenize_fn(examples):
    outputs = tokenizer(
        examples["text"],
        truncation=True,
        max_length=1024,
        padding="max_length"
    )
    return outputs

dataset = dataset.map(tokenize_fn, batched=True)

# 3. Configure LoRA
lora_config = LoraConfig(
    r=8,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

# 4. Apply LoRA
model = get_peft_model(model, lora_config)
print(model.print_trainable_parameters())

# 5. Training args
training_args = TrainingArguments(
    output_dir="./results",
    num_train_epochs=3,
    per_device_train_batch_size=8,
    per_device_eval_batch_size=8,
    warmup_steps=100,
    weight_decay=0.01,
    logging_steps=10,
    save_steps=500,
    eval_steps=500,
    learning_rate=2e-4,
)

# 6. Train
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=dataset["train"],
    eval_dataset=dataset["validation"]
)

trainer.train()

# 7. Save
model.save_pretrained("./my-lora-model")

Example 2: Few-Shot Adaptation with QLoRA

# Quick adaptation with only 100 examples

from peft import prepare_model_for_kbit_training
from transformers import BitsAndBytesConfig

# Quantize for memory efficiency
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

# Load quantized model
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    quantization_config=bnb_config,
    device_map="auto"
)

# Prepare for training
model = prepare_model_for_kbit_training(model)

# Apply LoRA
lora_config = LoraConfig(
    r=16,  # Higher rank for few-shot
    lora_alpha=32,
    target_modules=["q_proj", "v_proj", "o_proj"],  # More modules
    lora_dropout=0.1,  # Higher dropout for small data
    bias="none",
    task_type="CAUSAL_LM"
)
model = get_peft_model(model, lora_config)

# Small dataset fine-tuning
small_dataset = load_dataset("json", data_files="few_shot.jsonl")

training_args = TrainingArguments(
    output_dir="./few-shot-lora",
    num_train_epochs=10,  # More epochs for small data
    per_device_train_batch_size=2,  # Small batch
    learning_rate=1e-4,
    save_steps=50,  # Frequent saves
    logging_steps=5,
)

trainer = Trainer(model=model, args=training_args, train_dataset=small_dataset["train"])
trainer.train()

model.save_pretrained("./few-shot-lora")

---

Comparison: LoRA vs Other Methods

Method              Parameters  Memory  Speed  Accuracy  Ease
─────────────────────────────────────────────────────────────
Full Fine-Tuning    100%        100%    1x     100%      Easy
Prefix Tuning       ~1%         80%     0.8x   95%       Medium
Adapter Layers      ~5%         60%     0.7x   98%       Medium
Prompt Tuning       ~0.01%      99%     1x     90%       Hard
LoRA (r=8)          0.06%       40%     1x     99%       Easy
LoRA (r=16)         0.13%       50%     1x     99.5%     Easy
QLoRA (r=8)         0.06%       10%     0.9x   98%       Easy

Best choice:
  - Speed + Simplicity: LoRA
  - Extreme memory: QLoRA
  - Best accuracy: Full Fine-Tuning (if resources allow)
  - Production: LoRA (best tradeoff)

---

Real-World Performance

Case Study 1: Domain Adaptation

Task: Adapt Llama 2 7B to Legal Domain
Dataset: 10K legal documents (~2M tokens)
Hardware: Single RTX 3090 (24GB)

Without LoRA (Full Fine-Tuning):
  - Not possible! (needs 28GB)
  - Would require 2 GPUs

With LoRA (r=8):
  - Training time: 2 hours
  - Peak memory: 18 GB
  - LoRA size: 4 MB
  - Accuracy on legal tasks: 92%

With QLoRA (r=8):
  - Training time: 3 hours
  - Peak memory: 10 GB (fits!)
  - LoRA size: 4 MB
  - Accuracy: 90% (slight loss)
  - Cost: $1-2 (vs $50+ for full FT)

Result: QLoRA enables fine-tuning on single consumer GPU!

Case Study 2: Multi-Task Learning

Task: Create specialized chat model
  - Base: Llama 2 7B
  - Task 1: Customer support (5K conversations)
  - Task 2: Technical help (3K docs)
  - Task 3: Product info (2K docs)

Approach: Multi-LoRA

Setup:
  - Base model loaded once: 28 GB
  - LoRA 1 (support): 4 MB
  - LoRA 2 (tech): 4 MB
  - LoRA 3 (product): 4 MB
  - Total trainable: 12 MB (vs 28 GB for 3 full models!)

Usage:
  - Customer support question → use LoRA 1
  - Technical question → use LoRA 2
  - Product question → use LoRA 3
  - Or combine adapters for hybrid

Benefits:
✓ 3x specialized adapters with same memory as 1 full model
✓ Easy to mix and match
✓ Each trained in parallel (12 GPU-hours vs 72)

---

Best Practices

✅ Do's

1. Start with r=8 (good default for most tasks)
2. Use QLoRA for constrained memory (enables fine-tuning on consumer GPUs)
3. Apply LoRA to q_proj and v_proj (captures 70% of benefit)
4. Monitor validation loss (watch for overfitting with small data)
5. Use learning rate 1e-4 to 5e-4 (lower than base model training)
6. Save LoRA weights separately (small files, easy distribution)
7. Test on small data first (verify before large runs)
8. Merge for production (if single adapter, merge for efficiency)

❌ Don'ts

1. ❌ Use LoRA for training from scratch (only for fine-tuning!)
2. ❌ Apply to too many modules (diminishing returns after attention+ff)
3. ❌ Use very high rank (r>32 rarely needed)
4. ❌ Train base model + LoRA together (freeze base model!)
5. ❌ Use batch size too small (< 2 for 7B models)
6. ❌ Forget to set proper learning rate (too high → divergence)
7. ❌ Mix LoRA versions (stick to one adapter per base model)
8. ❌ Ignore data quality (garbage in, garbage out applies to LoRA too)

---

Memory Estimation

Quick Calculator

For model of size M GB:

Full Fine-Tuning needs:
  - Model: M GB
  - Gradients: M GB
  - Optimizer states: 2M GB
  - Activations: ~0.2M GB
  - Total: ~4.2M GB

LoRA with rank r needs:
  - Model: M GB (frozen)
  - LoRA adapters: M × r / d × 2 / 32 × 2 (float16)
  - ≈ M × 0.01 GB (for r=8, d=4096)
  - Gradients for LoRA: 0.01 MB
  - Optimizer states: 0.02 MB
  - Activations: ~0.2M GB
  - Total: ~M + 0.22M ≈ 1.2M GB

QLoRA needs:
  - Model: M/4 GB (int4 quantized)
  - LoRA adapters: 0.01 MB
  - Activations: ~0.2M GB
  - Total: ~0.25M + 0.22M ≈ 0.47M GB

Example (Llama 7B, M=28 GB):

Full FT:   28 × 4.2 = 117.6 GB (needs 4×A100s)
LoRA:      28 × 1.2 = 33.6 GB (needs 1×A100)
QLoRA:     28 × 0.47 = 13.2 GB (fits on RTX 4090!)

---

Key Takeaways

🔑 LoRA reduces trainable parameters by 200-1000x
💾 LoRA memory usage: 70% reduction
⚡ Training 3-5x faster than full fine-tuning
📊 Accuracy: 99-99.5% of full fine-tuning
🎯 QLoRA enables consumer GPU training
🚀 Industry standard for LLM adaptation

---

Comparison Summary

Metric	Full FT	LoRA	QLoRA
Params	100%	0.06%	0.06%
Memory	120 GB	37 GB	11 GB
Time	24h	8h	10h
Accuracy	100%	99%	98%
GPU	8×A100	1×A100	RTX 4090
Cost	$600/h	$75/h	$0.5/h

---

Further Reading

• LoRA Paper: "LoRA: Low-Rank Adaptation of Large Language Models"
• QLoRA Paper: "QLoRA: Efficient Finetuning of Quantized LLMs"
• PEFT Library: github.com/huggingface/peft
• Comprehensive Guide: HuggingFace documentation on PEFT

---

Related Notes

• Quantization Gptq - Combine with LoRA for QLoRA
• Flash Attention - Speed up LoRA training
• Kv Cache - Speeds inference of LoRA-adapted models
• Model Optimization - Other efficiency techniques