Model Merging & Ensemble Methods: Combining Models for Better Performance¶
Overview¶
Model Merging combines multiple trained models (or LoRA adapters) into a single model that performs better than any individual. Ensemble Methods run multiple models and combine outputs. Different trade-offs in speed, quality, and complexity.
- Merging: Average weights, interpolation, TIES, DARE
- Ensemble: Voting, averaging, distillation
- Use Case: Combine specialized models, improve robustness, multi-task
- Trade-off: Larger model size (merging) or inference cost (ensemble)
The Motivation¶
Problem: Task-Specific Overfitting¶
Scenario 1: Multiple fine-tuned models
Task 1: "Code generation"
- Model A (fine-tuned on code)
- Size: 7B
- Quality on task 1: 95%
Task 2: "Math reasoning"
- Model B (fine-tuned on math)
- Size: 7B
- Quality on task 2: 95%
Problem:
- Model A on task 2: 60% (poor!)
- Model B on task 1: 60% (poor!)
- Total model size: 14B (two copies)
- Can't handle both tasks well
Solution 1: Ensemble (run both, average)
- Quality: 80-90% on both
- Size: 14B (both loaded)
- Speed: 2x slower (must run both)
Solution 2: Merge models
- Quality: 85-92% on both
- Size: 7.1B (slightly larger, merged)
- Speed: 1x (run once)
Model Merging Techniques¶
1. Simple Weight Averaging¶
Technique: Average weights directly
W_merged = (W_A + W_B + W_C) / 3
Pseudocode:
```python
def average_models(model_a, model_b, model_c):
merged_model = type(model_a)() # Empty model
for name, param in merged_model.named_parameters():
# Average weights from all models
avg_weight = (
model_a.state_dict()[name] +
model_b.state_dict()[name] +
model_c.state_dict()[name]
) / 3
param.data = avg_weight
return merged_model
Pros: ✅ Simple to implement ✅ Fast (one-time computation) ✅ Linear combination
Cons: ❌ Often fails catastrophically! ❌ Weights from different tasks conflict ❌ Result: Model worse than any individual ❌ 50-70% quality (vs 80-90% expected)
### 2. Task-Specific Merging (TIES)
Algorithm: Step 1: Compute parameter importance per model - Find which weights matter for each task - Weights not important to task → set to zero
Step 2: Merge only "important" weights - Average important weights - Keep task-specific weights unchanged - Result: Model specialized for both tasks!
Step 3: Resolve conflicts - If weight important in both: Vote on value - Otherwise: Keep the important weight
Code:
def ties_merge(model_a, model_b, mask_a, mask_b, ratio=0.5):
"""
TIES: Task-relevant parameter Identification and Ensemble-ing
mask_a, mask_b: Boolean masks of which weights matter for each task
ratio: How much A vs B to average
"""
merged_model = type(model_a)()
for name, param in merged_model.named_parameters():
w_a = model_a.state_dict()[name]
w_b = model_b.state_dict()[name]
mask_a_w = mask_a[name]
mask_b_w = mask_b[name]
# Initialize with average (safe start)
w_merged = ratio * w_a + (1 - ratio) * w_b
# If task_a-specific: keep w_a
if mask_a_w and not mask_b_w:
w_merged = w_a
# If task_b-specific: keep w_b
elif mask_b_w and not mask_a_w:
w_merged = w_b
# If both important: weighted average
elif mask_a_w and mask_b_w:
w_merged = ratio * w_a + (1 - ratio) * w_b
# If neither important: average (least impact)
else:
w_merged = ratio * w_a + (1 - ratio) * w_b
param.data = w_merged
return merged_model
Pros: ✅ Much better than naive averaging ✅ Maintains task specialization ✅ 85-92% quality (significantly better!)
Cons: ❌ Requires computing parameter importance ❌ More complex to implement ❌ Still some conflicts to resolve
### 3. DARE (Domain-Agnostic Rank-Encoding)
Similar to LoRA merging: W = W_base + ΔW_A + ΔW_B
Where ΔW_A, ΔW_B are low-rank updates (like LoRA)
Advantage: - Works specifically for LoRA adapters - Preserves structure - Better quality than weight averaging - 87-93% quality
Implementation: Just average the LoRA weights!
def merge_lora_adapters(base_model, lora_a, lora_b):
"""Merge LoRA adapters"""
# Average LoRA matrices
merged_lora_a = (lora_a.lora_a + lora_b.lora_a) / 2
merged_lora_b = (lora_a.lora_b + lora_b.lora_b) / 2
# Apply to base model
merged_model = copy(base_model)
merged_model.lora_a = merged_lora_a
merged_model.lora_b = merged_lora_b
return merged_model
---
## Ensemble Methods (Without Merging)
### 1. Output Averaging
Models: - Model A: Output logits [2.1, -0.5, 1.2] - Model B: Output logits [2.0, -0.3, 1.4] - Model C: Output logits [2.2, -0.4, 1.1]
Average: - Result: [(2.1+2.0+2.2)/3, (-0.5-0.3-0.4)/3, (1.2+1.4+1.1)/3] - → [2.1, -0.4, 1.2]
Quality: - Ensemble: 90-95% - Individual: 82-87% - Improvement: 3-8%
Speed: - Must run 3 models - Inference time: 3x slower - Not practical for real-time
### 2. Voting Ensemble
Models vote on most likely token: - Model A: Predicts "yes" (argmax) - Model B: Predicts "yes" - Model C: Predicts "no" - Ensemble: "yes" (2 out of 3 vote yes)
Advantage: - Robust to outliers - More interpretable - Works for classification
Disadvantage: - Loses confidence information - Less effective for generation (many valid answers)
### 3. Mixture of Experts (Soft Ensemble)
Trade-off: Between full ensemble (slower) and single model (lower quality)
---
## Merging vs Ensemble Trade-off
When to use: Merging: - Inference speed critical - Model size acceptable (7-13B) - Quality 85%+ acceptable
Ensemble: - Speed not critical - Batch processing OK - Maximum quality needed - Have compute resources
---
## Practical Merging for LoRA
### Multi-Task LoRA Merging
```python
class MultiTaskLoRA:
def __init__(self, base_model, tasks):
self.base_model = base_model
self.adapters = {} # Task → LoRA weights
self.tasks = tasks
def train_adapter(self, task_name, data):
"""Train LoRA for specific task"""
lora = create_lora_adapter(self.base_model)
train_lora(lora, self.base_model, data)
self.adapters[task_name] = lora
def merge_all_adapters(self, weights=None):
"""Merge all task-specific adapters"""
if weights is None:
weights = {task: 1/len(self.tasks) for task in self.tasks}
merged_lora_a = None
merged_lora_b = None
for task, weight in weights.items():
lora = self.adapters[task]
if merged_lora_a is None:
merged_lora_a = weight * lora.lora_a
merged_lora_b = weight * lora.lora_b
else:
merged_lora_a += weight * lora.lora_a
merged_lora_b += weight * lora.lora_b
# Create merged model
merged_model = copy(self.base_model)
merged_model.lora_a = merged_lora_a
merged_model.lora_b = merged_lora_b
return merged_model
# Usage:
multi_lora = MultiTaskLoRA(base_model, ['code', 'math', 'writing'])
multi_lora.train_adapter('code', code_data)
multi_lora.train_adapter('math', math_data)
multi_lora.train_adapter('writing', writing_data)
merged = multi_lora.merge_all_adapters()
# Single 7B model handles all 3 tasks!
Advanced: Model Interpolation¶
Beyond simple averaging: learn interpolation weights
W_merged = α₁ W_A + α₂ W_B + α₃ W_C
Where α values learned (not fixed at 1/3)
Method:
1. Create merged model with learnable α values
2. Validate on held-out data
3. Optimize α to maximize performance
4. Set α values and freeze
Result:
- Better than equal weights
- Learns task importance automatically
- Quality: 87-93% (better than uniform!)
Key Takeaways¶
🔀 Simple averaging often fails; need sophisticated merging
🎯 TIES/DARE: Better merging methods (~90% quality)
⚖️ Merging vs Ensemble: Speed vs. quality trade-off
📊 Multi-task merging: Combine specialization with generalization
🚀 Interpolation: Learn optimal merge weights
Related Notes¶
- Lora - Adapter merging details
- Mixture Of Experts (Moe) - Related gating concept
- Llm Inference Optimization - Inference considerations