Adapter Methods Beyond LoRA

Overview

Adapter Methods enable parameter-efficient fine-tuning of large models without modifying core weights. LoRA is most popular, but alternatives offer different trade-offs in efficiency, quality, and flexibility.

• Approaches: LoRA, Adapters, Prefix Tuning, IA³, Compacter
• Trade-off: Trainable parameters vs. fine-tuning quality
• Use Case: Multi-task learning, personalization, domain adaptation
• Goal: Exploit pre-trained knowledge while adapting to specific tasks

---

Why Adapters Matter

The Problem: Full Fine-tuning is Expensive

Fine-tuning LLaMA 7B on specific task:

Full fine-tuning:
 - Trainable parameters: 7B (all)
 - Memory needed: 140GB (weights + optimizer + gradients)
 - GPU memory: 8 × H100 (80GB each) minimum
 - Time: Hours to days
 - Cost: $10,000+ for single task

LoRA fine-tuning:
 - Trainable parameters: 4.2M (0.06% of 7B)
 - Memory needed: 4GB (LoRA layers + optimizer)
 - GPU memory: 1 × A100 (80GB) sufficient
 - Time: ~10 minutes
 - Cost: $1

Trade-off:
 - LoRA: 0.06% parameters, 90% quality
 - Full: 100% parameters, 100% quality
 - LoRA is 10,000x cheaper! But 10% quality loss for some tasks

---

Low-Rank Adaptation (LoRA): Quick Review

Architecture:
Weight update: ΔW = B × A (r << original)

 Input (h)
 ↓
- ┌─────────────────────────────┐
 - Original layer │
 - W: (d_out, d_in) │
 - W × h │ (expensive to train)
 - ┬────────────────┘
 ↓ (shared, frozen)
- ┌─────────────────────────────┐
 - LoRA adapter │
 - A: (r, d_in) │ (small, trainable)
 - B: (d_out, r) │ (small, trainable)
 - (B × A) × h │ (efficient to train)
 - ┬────────────────┘
 ↓ (add)
 Output = W×h + (B×A)×h

Why efficient:
 - Only train r×d_in + d_out×r parameters
 - r is small (8-16 typical)
 - 100-1000x parameter reduction

-

Alternative Adapter Methods

1. Bottleneck Adapters

Classical adapter approach (before LoRA):

Architecture:
 Input (h)
 ↓
- ┌─────────────────────────────┐
 - FC down: (d, r) │ (compress)
 - GELU activation │
 - Dropout │
 - ┤
 - FC up: (r, d) │ (expand)
 - Add & LayerNorm │
 - ┬────────────────┘
 ↓
 Output

Difference from LoRA:
 - LoRA: Pure low-rank A × B (no nonlinearity)
 - Adapters: Two FC layers with activation (nonlinear!)
 - Adapters: More expressive but larger
 - LoRA: Simpler, smaller, works as well or better

Comparison:
 Parameters Memory Quality
──────────────────────────────────────────
LoRA(r=8) 5M Minimal 95%
Bottleneck 10M Small 98%
Full tune 7000M Huge 100%

Verdict:
 - LoRA better trade-off (simpler, smaller, comparable quality)

2. Prefix Tuning

Idea: Add trainable prefix tokens to model input

Architecture:
 Prefix tokens (trainable) Normal input
 ↓ ↓
- ┌──────────────────────────────────────────┐
 - Model processes [prefix + input] │
 - Prefix learns task-specific information │
 - Model weights stay frozen │
 - ┘

Example:
Model: "What is the capital of France?"
Prefix (learned): [task_token_1, task_token_2,...]
 - Only these are trainable!

Advantage:
 - Very parameter-efficient (prefix is small)
 - Can add/remove tasks with different prefixes
 - Easy to switch between tasks

Disadvantage:
 - Less expressive than LoRA
 - Can't adapt mid-network (only at start)
 - Quality sometimes lower (5-10% loss vs LoRA's <5%)

3. IA³ (Infused Adapter by Inhibiting and Amplifying Activations)

Idea: Element-wise scaling of activations instead of adding parameters

Architecture:
 Input (h)
 ↓
- ┌─────────────────────────────┐
 - Original layer (frozen) │
 - W × h = hidden state │
 - ┬────────────────┘
 ↓ (element-wise multiply)
- ┌─────────────────────────────┐
 - IA³ scaling vector (s) │
 - Trainable: s_1, s_2,..., s_d
 - s ⊙ (W × h) ⊙ = hadamard product
 - ┬────────────────┘
 ↓
 Output

Why parameter-efficient:
 - Only d parameters (same as hidden dimension)
 - Tiny adapter! (vs LoRA which needs r×d_in)
 - Good for massive models

Quality trade-off:
 - IA³: 3M parameters, 92% quality
 - LoRA(r=8): 5M parameters, 95% quality
 - IA³ smaller but lower quality

4. Compacter (Parameterized Hypercomplex Multiplication)

Mathematical foundation: Hypercomplex numbers

Idea: Use hypercomplex structure for ultra-compact adapters

Architecture:
 Input (h)
 ↓
- ┌─────────────────────────────────────────┐
 - Hypercomplex transformation │
 - (Based on quaternion / octonion algebra)│
 - More compact than matrix operations │
 - ┬────────────────────────────┘
 ↓
 Output

Why ultra-compact:
 - Hypercomplex math: More structure, fewer parameters
 - Compacter(shared): <1M parameters for 7B model
 - But: More complex implementation
 - Trade: 0.1-0.2% additional quality loss vs LoRA

Use case:
 - When parameters must be absolute minimum
 - Extreme parameter budgets

---

Comparison Matrix

Method Parameters Memory Quality Expressiveness Implementation
────────────────────────────────────────────────────────────────────────────────
LoRA 0.1-1% Low 95% Good Simple
Bottleneck 0.2-0.5% Medium 98% Excellent Medium
Prefix Tuning 0.05-0.1% Very Low 90% Fair Simple
IA³ <0.01% Minimal 92% Limited Simple
Compacter <0.01% Minimal 93% Limited Complex
Full Fine-tune 100% Huge 100% Excellent Complex

Recommendation by use case:
 - Balanced: LoRA (industry standard)
 - High quality: Bottleneck adapters
 - Simplicity: Prefix tuning
 - Ultra-compact: IA³ or Compacter
 - Best possible: Full fine-tuning (if resources allow)

---

Combining Adapters

Multi-Adapter Fine-tuning

Single LLM, multiple adapters:

 LLaMA 7B (frozen)
 ↑
- ┌────────┼────────┐
 ↓ ↓ ↓
 LoRA₁ LoRA₂ LoRA₃
 (Task1) (Task2) (Task3)

Benefits:
 - Share base model (save memory)
 - Task-specific adapters (small)
 - Switch adapters at inference
 - Multi-task in single model!

Implementation:
 - Train LoRA₁ on task 1
 - Train LoRA₂ on task 2 (base model still frozen)
 - Train LoRA₃ on task 3
 - At inference: Load relevant LoRA
 - Total model: 7B + (0.06% × 3) ≈ 7B memory
 - vs 21B if separate models!

Merging Adapters

After training multiple adapters, optionally merge for efficiency:

Merge LoRA into weights:
 - W_merged = W + B × A (add LoRA to base)
 - At inference: Use W_merged directly
 - No overhead! (LoRA layers gone)
 - Can't switch tasks anymore

Use case:
 - Single-task deployment
 - Inference speed important
 - Training time amortized (merge once, use many times)

-

When to Use Which Adapter

Use LoRA When

Standard approach (default choice)
Good quality/efficiency balance needed
Multi-task learning
Cost-sensitive training
Parameter budget 0.1-1% of model size

Use Bottleneck Adapters When

Maximum quality important
Can afford 2-5x more parameters than LoRA
Complex adaptation needed
Non-linear transformations help

Use Prefix Tuning When

Minimal parameter budget (<0.05%)
Simplicity critical
Task can be expressed in prefix
Few inference tasks

Use IA³ When

Extreme parameter budgets
Scaling-based adaptation sufficient
Large models where 0.1% still matters
Memory is hardest constraint

-

Key Takeaways

LoRA: Best overall choice for efficiency + quality Bottleneck: Better quality, 2-5x more parameters Prefix Tuning: Simplest, smallest budget IA³/Compacter: Extreme minimalism, quality trade-off Multi-adapter: Share base, task-specific adapters

-

Related Notes

• Lora - Detailed LoRA guide
• Distributed Training - Train multiple adapters in parallel
• [Model Merging & Ensemble Methods](/01-modeling/04-production/02-operations/(model-merging-ensemble-methods/) - Merge adapters with base model
• Llm Inference Optimization - Adapter inference costs