Adapter Methods Beyond LoRA: Efficient Fine-tuning Alternatives

Overview

Adapter Methods enable parameter-efficient fine-tuning of large models without modifying core weights. LoRA is most popular, but alternatives offer different trade-offs in efficiency, quality, and flexibility.

• Approaches: LoRA, Adapters, Prefix Tuning, IA³, Compacter
• Trade-off: Trainable parameters vs. fine-tuning quality
• Use Case: Multi-task learning, personalization, domain adaptation
• Goal: Exploit pre-trained knowledge while adapting to specific tasks

---

Why Adapters Matter

The Problem: Full Fine-tuning is Expensive

Fine-tuning LLaMA 7B on specific task:

Full fine-tuning:
  - Trainable parameters: 7B (all)
  - Memory needed: 140GB (weights + optimizer + gradients)
  - GPU memory: 8 × H100 (80GB each) minimum
  - Time: Hours to days
  - Cost: $10,000+ for single task

LoRA fine-tuning:
  - Trainable parameters: 4.2M (0.06% of 7B)
  - Memory needed: 4GB (LoRA layers + optimizer)
  - GPU memory: 1 × A100 (80GB) sufficient
  - Time: ~10 minutes
  - Cost: $1

Trade-off:
  - LoRA: 0.06% parameters, 90% quality
  - Full: 100% parameters, 100% quality
  - LoRA is 10,000x cheaper! But 10% quality loss for some tasks

---

Low-Rank Adaptation (LoRA): Quick Review

Architecture:
Weight update: ΔW = B × A  (r << original)

                Input (h)
                    ↓
- ┌─────────────────────────────┐
    - Original layer               │
    - W: (d_out, d_in)            │
    - W × h                        │ (expensive to train)
  - ┬────────────────┘
                 ↓ (shared, frozen)
- ┌─────────────────────────────┐
    - LoRA adapter                 │
    - A: (r, d_in)                │ (small, trainable)
    - B: (d_out, r)               │ (small, trainable)
    - (B × A) × h                 │ (efficient to train)
  - ┬────────────────┘
                 ↓ (add)
    Output = W×h + (B×A)×h

Why efficient:
  - Only train r×d_in + d_out×r parameters
  - r is small (8-16 typical)
  - 100-1000x parameter reduction

---

Alternative Adapter Methods

1. Bottleneck Adapters

Classical adapter approach (before LoRA):

Architecture:
                Input (h)
                    ↓
- ┌─────────────────────────────┐
    - FC down: (d, r)             │ (compress)
    - GELU activation             │
    - Dropout                     │
  - ┤
    - FC up: (r, d)               │ (expand)
    - Add & LayerNorm             │
  - ┬────────────────┘
                 ↓
                 Output

Difference from LoRA:
  - LoRA: Pure low-rank A × B (no nonlinearity)
  - Adapters: Two FC layers with activation (nonlinear!)
  - Adapters: More expressive but larger
  - LoRA: Simpler, smaller, works as well or better

Comparison:
                Parameters  Memory  Quality
──────────────────────────────────────────
LoRA(r=8)      5M          Minimal  95%
Bottleneck     10M         Small    98%
Full tune      7000M       Huge     100%

Verdict:
  - LoRA better trade-off (simpler, smaller, comparable quality)

2. Prefix Tuning

Idea: Add trainable prefix tokens to model input

Architecture:
    Prefix tokens (trainable)     Normal input
           ↓                           ↓
- ┌──────────────────────────────────────────┐
    - Model processes [prefix + input]         │
    - Prefix learns task-specific information  │
    - Model weights stay frozen               │
  - ┘

Example:
Model: "What is the capital of France?"
Prefix (learned): [task_token_1, task_token_2, ...]
  - Only these are trainable!

Advantage:
  - Very parameter-efficient (prefix is small)
  - Can add/remove tasks with different prefixes
  - Easy to switch between tasks

Disadvantage:
  - Less expressive than LoRA
  - Can't adapt mid-network (only at start)
  - Quality sometimes lower (5-10% loss vs LoRA's <5%)

3. IA³ (Infused Adapter by Inhibiting and Amplifying Activations)

Idea: Element-wise scaling of activations instead of adding parameters

Architecture:
              Input (h)
                  ↓
- ┌─────────────────────────────┐
    - Original layer (frozen)     │
    - W × h  = hidden state       │
  - ┬────────────────┘
                 ↓ (element-wise multiply)
- ┌─────────────────────────────┐
    - IA³ scaling vector (s)      │
  - Trainable: s_1, s_2, ..., s_d
  - s ⊙ (W × h)  ⊙ = hadamard product
  - ┬────────────────┘
                 ↓
                 Output

Why parameter-efficient:
  - Only d parameters (same as hidden dimension)
  - Tiny adapter! (vs LoRA which needs r×d_in)
  - Good for massive models

Quality trade-off:
  - IA³: 3M parameters, 92% quality
  - LoRA(r=8): 5M parameters, 95% quality
  - IA³ smaller but lower quality

4. Compacter (Parameterized Hypercomplex Multiplication)

Mathematical foundation: Hypercomplex numbers

Idea: Use hypercomplex structure for ultra-compact adapters

Architecture:
                Input (h)
                    ↓
- ┌─────────────────────────────────────────┐
    - Hypercomplex transformation             │
    - (Based on quaternion / octonion algebra)│
    - More compact than matrix operations     │
  - ┬────────────────────────────┘
                 ↓
                 Output

Why ultra-compact:
  - Hypercomplex math: More structure, fewer parameters
  - Compacter(shared): <1M parameters for 7B model
  - But: More complex implementation
  - Trade: 0.1-0.2% additional quality loss vs LoRA

Use case:
  - When parameters must be absolute minimum
  - Extreme parameter budgets

---

Comparison Matrix

Method          Parameters   Memory   Quality   Expressiveness   Implementation
────────────────────────────────────────────────────────────────────────────────
LoRA            0.1-1%       Low      95%       Good             Simple
Bottleneck      0.2-0.5%     Medium   98%       Excellent        Medium
Prefix Tuning   0.05-0.1%    Very Low 90%       Fair              Simple
IA³             <0.01%       Minimal  92%       Limited           Simple
Compacter       <0.01%       Minimal  93%       Limited           Complex
Full Fine-tune  100%         Huge     100%      Excellent         Complex

Recommendation by use case:
  - Balanced: LoRA (industry standard)
  - High quality: Bottleneck adapters
  - Simplicity: Prefix tuning
  - Ultra-compact: IA³ or Compacter
  - Best possible: Full fine-tuning (if resources allow)

---

Combining Adapters

Multi-Adapter Fine-tuning

Single LLM, multiple adapters:

        LLaMA 7B (frozen)
             ↑
- ┌────────┼────────┐
    ↓        ↓        ↓
  LoRA₁    LoRA₂    LoRA₃
  (Task1)  (Task2)  (Task3)

Benefits:
  - Share base model (save memory)
  - Task-specific adapters (small)
  - Switch adapters at inference
  - Multi-task in single model!

Implementation:
  - Train LoRA₁ on task 1
  - Train LoRA₂ on task 2 (base model still frozen)
  - Train LoRA₃ on task 3
  - At inference: Load relevant LoRA
  - Total model: 7B + (0.06% × 3) ≈ 7B memory
  - vs 21B if separate models!

Merging Adapters

After training multiple adapters, optionally merge for efficiency:

Merge LoRA into weights:
  - W_merged = W + B × A  (add LoRA to base)
  - At inference: Use W_merged directly
  - No overhead! (LoRA layers gone)
  - Can't switch tasks anymore

Use case:
  - Single-task deployment
  - Inference speed important
  - Training time amortized (merge once, use many times)

---

When to Use Which Adapter

Use LoRA When

✅ Standard approach (default choice)
✅ Good quality/efficiency balance needed
✅ Multi-task learning
✅ Cost-sensitive training
✅ Parameter budget 0.1-1% of model size

Use Bottleneck Adapters When

✅ Maximum quality important
✅ Can afford 2-5x more parameters than LoRA
✅ Complex adaptation needed
✅ Non-linear transformations help

Use Prefix Tuning When

✅ Minimal parameter budget (<0.05%)
✅ Simplicity critical
✅ Task can be expressed in prefix
✅ Few inference tasks

Use IA³ When

✅ Extreme parameter budgets
✅ Scaling-based adaptation sufficient
✅ Large models where 0.1% still matters
✅ Memory is hardest constraint

---

Key Takeaways

🎯 LoRA: Best overall choice for efficiency + quality
📊 Bottleneck: Better quality, 2-5x more parameters
🎪 Prefix Tuning: Simplest, smallest budget
⚙️ IA³/Compacter: Extreme minimalism, quality trade-off
🔀 Multi-adapter: Share base, task-specific adapters

---

Related Notes

• Lora - Detailed LoRA guide
• Distributed Training - Train multiple adapters in parallel
• Model Merging & Ensemble Methods - Merge adapters with base model
• Llm Inference Optimization - Adapter inference costs