Model Merging & Ensemble Methods

Overview

Model Merging combines multiple trained models (or LoRA adapters) into a single model that performs better than any individual. Ensemble Methods run multiple models and combine outputs. Different trade-offs in speed, quality, and complexity.

• Merging: Average weights, interpolation, TIES, DARE
• Ensemble: Voting, averaging, distillation
• Use Case: Combine specialized models, improve robustness, multi-task
• Trade-off: Larger model size (merging) or inference cost (ensemble)

---

The Motivation

Problem: Task-Specific Overfitting

Scenario 1: Multiple fine-tuned models

Task 1: "Code generation"
 - Model A (fine-tuned on code)
 - Size: 7B
 - Quality on task 1: 95%

Task 2: "Math reasoning"
 - Model B (fine-tuned on math)
 - Size: 7B
 - Quality on task 2: 95%

Problem:
 - Model A on task 2: 60% (poor!)
 - Model B on task 1: 60% (poor!)
 - Total model size: 14B (two copies)
 - Can't handle both tasks well

Solution 1: Ensemble (run both, average)
 - Quality: 80-90% on both
 - Size: 14B (both loaded)
 - Speed: 2x slower (must run both)

Solution 2: Merge models
 - Quality: 85-92% on both
 - Size: 7.1B (slightly larger, merged)
 - Speed: 1x (run once)

-

Model Merging Techniques

1. Simple Weight Averaging

Technique: Average weights directly

W_merged = (W_A + W_B + W_C) / 3

Pseudocode:
```python
def average_models(model_a, model_b, model_c):
 merged_model = type(model_a)() # Empty model

 for name, param in merged_model.named_parameters():
 # Average weights from all models
 avg_weight = (
 model_a.state_dict()[name] +
 model_b.state_dict()[name] +
 model_c.state_dict()[name]
) / 3
 param.data = avg_weight

 return merged_model

Pros: Simple to implement Fast (one-time computation) Linear combination

Cons: Often fails catastrophically! Weights from different tasks conflict Result: Model worse than any individual 50-70% quality (vs 80-90% expected)

### 2. Task-Specific Merging (TIES)

Idea: Don't average all weights equally Only merge weights that help, discard task-specific weights

Algorithm: Step 1: Compute parameter importance per model

• Find which weights matter for each task
• Weights not important to task → set to zero

Step 2: Merge only "important" weights

• Average important weights
• Keep task-specific weights unchanged
• Result: Model specialized for both tasks!

Step 3: Resolve conflicts

• If weight important in both: Vote on value
• Otherwise: Keep the important weight

Code:

def ties_merge(model_a, model_b, mask_a, mask_b, ratio=0.5):
 """
 TIES: Task-relevant parameter Identification and Ensemble-ing

 mask_a, mask_b: Boolean masks of which weights matter for each task
 ratio: How much A vs B to average
 """
 merged_model = type(model_a)()

 for name, param in merged_model.named_parameters():
 w_a = model_a.state_dict()[name]
 w_b = model_b.state_dict()[name]
 mask_a_w = mask_a[name]
 mask_b_w = mask_b[name]

 # Initialize with average (safe start)
 w_merged = ratio * w_a + (1 - ratio) * w_b

 # If task_a-specific: keep w_a
 if mask_a_w and not mask_b_w:
 w_merged = w_a
 # If task_b-specific: keep w_b
 elif mask_b_w and not mask_a_w:
 w_merged = w_b
 # If both important: weighted average
 elif mask_a_w and mask_b_w:
 w_merged = ratio * w_a + (1 - ratio) * w_b
 # If neither important: average (least impact)
 else:
 w_merged = ratio * w_a + (1 - ratio) * w_b

 param.data = w_merged

 return merged_model

Pros: Much better than naive averaging Maintains task specialization 85-92% quality (significantly better!)

Cons: Requires computing parameter importance More complex to implement Still some conflicts to resolve

### 3. DARE (Domain-Agnostic Rank-Encoding)

Idea: Use low-rank structure to merge efficiently

Similar to LoRA merging: W = W_base + ΔW_A + ΔW_B

Where ΔW_A, ΔW_B are low-rank updates (like LoRA)

Advantage:

• Works specifically for LoRA adapters
• Preserves structure
• Better quality than weight averaging
• 87-93% quality

Implementation: Just average the LoRA weights!

def merge_lora_adapters(base_model, lora_a, lora_b):
 """Merge LoRA adapters"""

 # Average LoRA matrices
 merged_lora_a = (lora_a.lora_a + lora_b.lora_a) / 2
 merged_lora_b = (lora_a.lora_b + lora_b.lora_b) / 2

 # Apply to base model
 merged_model = copy(base_model)
 merged_model.lora_a = merged_lora_a
 merged_model.lora_b = merged_lora_b

 return merged_model

-

## Ensemble Methods (Without Merging)

### 1. Output Averaging

Run multiple models, average predictions

Models:

• Model A: Output logits [2.1, -0.5, 1.2]
• Model B: Output logits [2.0, -0.3, 1.4]
• Model C: Output logits [2.2, -0.4, 1.1]

Average:

• Result: [(2.1+2.0+2.2)/3, (-0.5-0.3-0.4)/3, (1.2+1.4+1.1)/3]
• → [2.1, -0.4, 1.2]

Quality:

• Ensemble: 90-95%
• Individual: 82-87%
• Improvement: 3-8%

Speed:

• Must run 3 models
• Inference time: 3x slower
• Not practical for real-time

### 2. Voting Ensemble

Instead of averaging logits, run majority vote

Models vote on most likely token:

• Model A: Predicts "yes" (argmax)
• Model B: Predicts "yes"
• Model C: Predicts "no"
• Ensemble: "yes" (2 out of 3 vote yes)

Advantage:

• Robust to outliers
• More interpretable
• Works for classification

Disadvantage:

• Loses confidence information
• Less effective for generation (many valid answers)

### 3. Mixture of Experts (Soft Ensemble)

Learnable routing:

• Input
• Router learns: "Use model A for X, model B for Y"
• Output from selected model(s)
• Combines speed (select few) + quality (ensemble)

Trade-off: Between full ensemble (slower) and single model (lower quality)

---

## Merging vs Ensemble Trade-off

Aspect Merging Ensemble ───────────────────────────────────────────────── Model Size Larger (7B+) Multiple copies (14B+) Inference Speed Fast (1x) Slow (Nx) Quality Good (85-92%) Excellent (90-95%) Implementation Medium Medium Training Cost Low (one-time) Low (use existing) Deployment Single model Multiple models Memory Reasonable High

When to use: Merging:

• Inference speed critical
• Model size acceptable (7-13B)
• Quality 85%+ acceptable

Ensemble:

• Speed not critical
• Batch processing OK
• Maximum quality needed
• Have compute resources

-

## Practical Merging for LoRA

### Multi-Task LoRA Merging

```python
class MultiTaskLoRA:
 def __init__(self, base_model, tasks):
 self.base_model = base_model
 self.adapters = {} # Task → LoRA weights
 self.tasks = tasks

 def train_adapter(self, task_name, data):
 """Train LoRA for specific task"""
 lora = create_lora_adapter(self.base_model)
 train_lora(lora, self.base_model, data)
 self.adapters[task_name] = lora

 def merge_all_adapters(self, weights=None):
 """Merge all task-specific adapters"""
 if weights is None:
 weights = {task: 1/len(self.tasks) for task in self.tasks}

 merged_lora_a = None
 merged_lora_b = None

 for task, weight in weights.items():
 lora = self.adapters[task]

 if merged_lora_a is None:
 merged_lora_a = weight * lora.lora_a
 merged_lora_b = weight * lora.lora_b
 else:
 merged_lora_a += weight * lora.lora_a
 merged_lora_b += weight * lora.lora_b

 # Create merged model
 merged_model = copy(self.base_model)
 merged_model.lora_a = merged_lora_a
 merged_model.lora_b = merged_lora_b

 return merged_model

# Usage:
multi_lora = MultiTaskLoRA(base_model, ['code', 'math', 'writing'])
multi_lora.train_adapter('code', code_data)
multi_lora.train_adapter('math', math_data)
multi_lora.train_adapter('writing', writing_data)

merged = multi_lora.merge_all_adapters()
# Single 7B model handles all 3 tasks!

-

Advanced: Model Interpolation

Beyond simple averaging: learn interpolation weights

W_merged = α₁ W_A + α₂ W_B + α₃ W_C

Where α values learned (not fixed at 1/3)

Method:
1. Create merged model with learnable α values
2. Validate on held-out data
3. Optimize α to maximize performance
4. Set α values and freeze

Result:
 - Better than equal weights
 - Learns task importance automatically
 - Quality: 87-93% (better than uniform!)

-

Key Takeaways

Simple averaging often fails; need sophisticated merging TIES/DARE: Better merging methods (~90% quality) Merging vs Ensemble: Speed vs. quality trade-off Multi-task merging: Combine specialization with generalization Interpolation: Learn optimal merge weights

-

Related Notes

• Lora - Adapter merging details
• Mixture Of Experts (Moe) - Related gating concept
• Llm Inference Optimization - Inference considerations