Pruning & Sparsity

Overview

Pruning removes less important weights or neurons from models. Sparsity enables efficient computation by skipping zero/low-value operations. Combined: 2-10x model size reduction with minimal quality loss.

• Approaches: Weight pruning, neuron pruning, structured vs unstructured
• Sparsity Types: Weight sparsity (80%+), activation sparsity (90%+)
• Speedup: 2-5x for unstructured, 1.5-3x for structured (hardware-dependent)
• Quality Loss: 1-5% typical with careful pruning
• Trade-off: Model size reduction vs inference speed complexity

---

The Insight: Most Parameters Aren't Important

Weight Importance Analysis

Research finding: Deep networks are over-parameterized

Typical distribution of weight magnitudes:

 Probability
 ↑
| ▁▂▃▄▅▆▇█ (many weights)
| ▇▅▄▃▂▁ (few large weights)
|
 - → Weight magnitude

Observation:
 - ~10-20% of weights are large (important)
 - ~80-90% of weights are small (less important)
 - Removing small weights: Model barely changes!
 - Pruning opportunity: Remove redundant small weights

Example (ResNet-50):
 - Total parameters: 25.5M
 - After 90% pruning: 2.5M parameters
 - Quality loss: 1-2%
 - Speedup: 3x (depends on hardware)

-

Pruning Strategies

1. Magnitude-Based Pruning (Simplest)

Idea: Remove weights below certain magnitude threshold

Algorithm:
1. Train model normally
2. Compute magnitude of each weight
3. Remove weights where|w| < threshold
4. (Optional) Fine-tune on remaining weights

Code:
```python
def magnitude_prune(model, sparsity_ratio=0.9):
 """
 Remove weights with smallest magnitudes

 Args:
 sparsity_ratio: Fraction of weights to remove (0.9 = remove 90%)
 """
 for name, param in model.named_parameters():
 if 'weight' in name: # Only prune weights, not biases
 # Compute magnitude
 magnitude = torch.abs(param.data)

 # Find threshold
 k = int(param.numel() * sparsity_ratio)
 threshold = torch.kthvalue(magnitude.view(-1), k)[0]

 # Create mask
 mask = magnitude > threshold

 # Apply mask (set small weights to zero)
 param.data = param.data * mask.float()

# Usage:
model = load_pretrained_model()
magnitude_prune(model, sparsity_ratio=0.9)
# Model now has 90% weights set to zero!

Pros: Simple to implement Works with any model No retraining necessary

Cons: Unstructured (not all hardware optimized for sparse computation) Quality drops without fine-tuning Threshold selection is manual

2. Structured Pruning (More Efficient)

Instead of removing individual weights, remove entire channels/filters

Standard pruning (unstructured):
 - Remove scattered weights
 - Model still has same dimensions
 - Hardware must skip scattered computations
 - Not all hardware supports (complexity!)

Structured pruning:
 - Remove entire filters/channels
 - Actual model size and speed improve
 - Works with any hardware
 - But: Must remove full structures (less flexible)

Example: Prune filter from CNN

Before:
Input: (batch, 64, H, W)
 ↓
Filter layer (64 → 128 filters): each 3×3×64
 ↓
Output: (batch, 128, H, W)

After pruning (remove 32 filters):
Input: (batch, 64, H, W)
 ↓
Filter layer (64 → 96 filters): each 3×3×64
 ↓
Output: (batch, 96, H, W)

Benefit:
 - Actual computation reduced (matrix multiply on 96 not 128)
 - Memory reduced (96 filters < 128 filters)
 - All hardware supports (just smaller matrix)

3. Iterative Magnitude Pruning (Lottery Ticket)

Key finding: Pruning gradually better than one-shot

Approach:
1. Train model to convergence
2. Prune 20% of weights (magnitude-based)
3. Reset remaining weights to initial values (!)
4. Train again to convergence
5. Prune another 20% of remaining
6. Repeat until target sparsity

Result:
 - Iterative: 50% quality loss at 99% sparsity
 - One-shot: 90% quality loss at 99% sparsity
 - Iterative is MUCH better!

Why it works:
- Early iteration: Remove clearly unimportant weights
- Later iteration: Remove more nuanced patterns
- Gradual reduction better than aggressive one-shot

Code:
```python
def iterative_pruning(model, data, target_sparsity=0.9, steps=5):
 current_sparsity = 0.0
 sparsity_per_step = target_sparsity / steps

 for step in range(steps):
 # Train current model
 train_model(model, data, epochs=10)

 # Prune
 current_sparsity += sparsity_per_step
 magnitude_prune(model, current_sparsity)

 # Fine-tune
 train_model(model, data, epochs=5)

---

## Activation Sparsity

Different from weight sparsity!

Weight sparsity: Remove weights from model Activation sparsity: Skip computation when activations are zero

Example: Model with ReLU:

• ReLU sets negative values to zero
• Many activations are zero!
• Can skip multiply-accumulate operations for zero activations
• Potential: Skip 50-90% of compute!

Hardware support:

• Traditional GPUs: No special support
• Sparse tensor libraries: TensorRT, cuSPARSE
• Some speedup but not proportional to sparsity
• Challenge: Controlling sparsity patterns

---

## Knowledge Distillation vs Pruning

### Comparison

Technique Size Red. Speed Quality Loss Effort ────────────────────────────────────────────────────── Pruning 3-10x 2-5x 2-5% Medium Distillation 10-50x 10-100x 5-15% High Quantization 2-4x 1.5-2x 1-3% Low Combined 20-100x 20-50x 5-10% Very High

Pruning strategy:

• Simpler than distillation
• Works with existing model
• Gradual quality loss
• Good baseline optimization

Distillation strategy:

• Requires teacher model
• More aggressive compression
• Better final quality
• Use after pruning for maximum compression

---

## Sparsity-Aware Hardware

### Tensor Engines

Modern GPUs:

• NVIDIA A100: Tensor cores with structured sparsity support
• 2x speedup for 50% structured sparsity
• NVIDIA H100: Better sparse support
• More efficient sparse operations
• AMD MI300: Sparse tensor support

CPUs:

• Can utilize sparsity via special libraries
• Smaller speedup than GPUs
• Better for 5-10% sparsity

Specialized hardware:

• SambaNova: Sparse tensor processor
• Cerebras: Sparse network support
• Groq: Optimized for inference sparsity
• Custom chips: Designed for specific sparsity patterns

-

## When to Use Pruning

### Use Pruning When

Model size is critical (mobile/edge) Hardware supports sparse operations Can accept 2-5% quality loss Inference speed important Model was over-trained / over-parameterized

### Avoid Pruning When

Model is already optimized Hardware doesn't support sparsity Quality cannot degrade Simplicity is important

---

## Practical Pruning Pipeline

1. Train model to convergence

2. Get baseline quality

3. Evaluate structured vs unstructured

4. Structured: Check if hardware supports

5. Unstructured: Requires sparse tensor support

6. Usually: Start with structured

7. Apply iterative pruning

8. Step 1: Prune 20%, fine-tune

9. Step 2: Prune 20% more, fine-tune

10. Repeat until target sparsity

11. Measure quality and speed

12. Quality: MMLU, benchmark on your task

13. Speed: Profile inference on target hardware

14. If unsatisfactory, reduce sparsity or stop

15. Deploy with sparse kernels

16. Use sparse tensor library for inference

17. Verify speedup on actual hardware
18. Monitor quality in production

```

---

Key Takeaways

Most parameters redundant: 90% can be pruned with care Iterative pruning much better than one-shot Structured pruning more hardware-friendly 2-5x compression with 2-5% quality loss typical Combine with quantization and distillation for max compression

-

Related Notes

• Quantization - Combine for maximum compression
• Model Distillation - More aggressive compression
• Llm Inference Optimization - Complete optimization stack