Pruning & Sparsity: Removing Redundant Parameters and Activations¶
Overview¶
Pruning removes less important weights or neurons from models. Sparsity enables efficient computation by skipping zero/low-value operations. Combined: 2-10x model size reduction with minimal quality loss.
- Approaches: Weight pruning, neuron pruning, structured vs unstructured
- Sparsity Types: Weight sparsity (80%+), activation sparsity (90%+)
- Speedup: 2-5x for unstructured, 1.5-3x for structured (hardware-dependent)
- Quality Loss: 1-5% typical with careful pruning
- Trade-off: Model size reduction vs inference speed complexity
The Insight: Most Parameters Aren't Important¶
Weight Importance Analysis¶
Research finding: Deep networks are over-parameterized
Typical distribution of weight magnitudes:
Probability
↑
| ▁▂▃▄▅▆▇█ (many weights)
| ▇▅▄▃▂▁ (few large weights)
|
- → Weight magnitude
Observation:
- ~10-20% of weights are large (important)
- ~80-90% of weights are small (less important)
- Removing small weights: Model barely changes!
- Pruning opportunity: Remove redundant small weights
Example (ResNet-50):
- Total parameters: 25.5M
- After 90% pruning: 2.5M parameters
- Quality loss: 1-2%
- Speedup: 3x (depends on hardware)
Pruning Strategies¶
1. Magnitude-Based Pruning (Simplest)¶
Idea: Remove weights below certain magnitude threshold
Algorithm:
1. Train model normally
2. Compute magnitude of each weight
3. Remove weights where |w| < threshold
4. (Optional) Fine-tune on remaining weights
Code:
```python
def magnitude_prune(model, sparsity_ratio=0.9):
"""
Remove weights with smallest magnitudes
Args:
sparsity_ratio: Fraction of weights to remove (0.9 = remove 90%)
"""
for name, param in model.named_parameters():
if 'weight' in name: # Only prune weights, not biases
# Compute magnitude
magnitude = torch.abs(param.data)
# Find threshold
k = int(param.numel() * sparsity_ratio)
threshold = torch.kthvalue(magnitude.view(-1), k)[0]
# Create mask
mask = magnitude > threshold
# Apply mask (set small weights to zero)
param.data = param.data * mask.float()
# Usage:
model = load_pretrained_model()
magnitude_prune(model, sparsity_ratio=0.9)
# Model now has 90% weights set to zero!
Pros: ✅ Simple to implement ✅ Works with any model ✅ No retraining necessary
Cons: ❌ Unstructured (not all hardware optimized for sparse computation) ❌ Quality drops without fine-tuning ❌ Threshold selection is manual
2. Structured Pruning (More Efficient)¶
Instead of removing individual weights, remove entire channels/filters
Standard pruning (unstructured):
- Remove scattered weights
- Model still has same dimensions
- Hardware must skip scattered computations
- Not all hardware supports (complexity!)
Structured pruning:
- Remove entire filters/channels
- Actual model size and speed improve
- Works with any hardware
- But: Must remove full structures (less flexible)
Example: Prune filter from CNN
Before:
Input: (batch, 64, H, W)
↓
Filter layer (64 → 128 filters): each 3×3×64
↓
Output: (batch, 128, H, W)
After pruning (remove 32 filters):
Input: (batch, 64, H, W)
↓
Filter layer (64 → 96 filters): each 3×3×64
↓
Output: (batch, 96, H, W)
Benefit:
- Actual computation reduced (matrix multiply on 96 not 128)
- Memory reduced (96 filters < 128 filters)
- All hardware supports (just smaller matrix)
3. Iterative Magnitude Pruning (Lottery Ticket)¶
Key finding: Pruning gradually better than one-shot
Approach:
1. Train model to convergence
2. Prune 20% of weights (magnitude-based)
3. Reset remaining weights to initial values (!)
4. Train again to convergence
5. Prune another 20% of remaining
6. Repeat until target sparsity
Result:
- Iterative: 50% quality loss at 99% sparsity
- One-shot: 90% quality loss at 99% sparsity
- Iterative is MUCH better!
Why it works:
- Early iteration: Remove clearly unimportant weights
- Later iteration: Remove more nuanced patterns
- Gradual reduction better than aggressive one-shot
Code:
```python
def iterative_pruning(model, data, target_sparsity=0.9, steps=5):
current_sparsity = 0.0
sparsity_per_step = target_sparsity / steps
for step in range(steps):
# Train current model
train_model(model, data, epochs=10)
# Prune
current_sparsity += sparsity_per_step
magnitude_prune(model, current_sparsity)
# Fine-tune
train_model(model, data, epochs=5)
---
## Activation Sparsity
Weight sparsity: Remove weights from model Activation sparsity: Skip computation when activations are zero
Example: Model with ReLU: - ReLU sets negative values to zero - Many activations are zero! - Can skip multiply-accumulate operations for zero activations - Potential: Skip 50-90% of compute!
Hardware support: - Traditional GPUs: No special support - Sparse tensor libraries: TensorRT, cuSPARSE - Some speedup but not proportional to sparsity - Challenge: Controlling sparsity patterns
---
## Knowledge Distillation vs Pruning
### Comparison
Pruning strategy: - Simpler than distillation - Works with existing model - Gradual quality loss - Good baseline optimization
Distillation strategy: - Requires teacher model - More aggressive compression - Better final quality - Use after pruning for maximum compression
---
## Sparsity-Aware Hardware
### Tensor Engines
CPUs: - Can utilize sparsity via special libraries - Smaller speedup than GPUs - Better for 5-10% sparsity
Specialized hardware: - SambaNova: Sparse tensor processor - Cerebras: Sparse network support - Groq: Optimized for inference sparsity - Custom chips: Designed for specific sparsity patterns
---
## When to Use Pruning
### Use Pruning When
### Avoid Pruning When
---
## Practical Pruning Pipeline
- Evaluate structured vs unstructured
- Structured: Check if hardware supports
- Unstructured: Requires sparse tensor support
-
Usually: Start with structured
-
Apply iterative pruning
- Step 1: Prune 20%, fine-tune
- Step 2: Prune 20% more, fine-tune
-
Repeat until target sparsity
-
Measure quality and speed
- Quality: MMLU, benchmark on your task
- Speed: Profile inference on target hardware
-
If unsatisfactory, reduce sparsity or stop
-
Deploy with sparse kernels
- Use sparse tensor library for inference
- Verify speedup on actual hardware
- Monitor quality in production ```
Key Takeaways¶
✂️ Most parameters redundant: 90% can be pruned with care
🎯 Iterative pruning much better than one-shot
⚙️ Structured pruning more hardware-friendly
📊 2-5x compression with 2-5% quality loss typical
🔄 Combine with quantization and distillation for max compression
Related Notes¶
- Quantization - Combine for maximum compression
- Model Distillation - More aggressive compression
- Llm Inference Optimization - Complete optimization stack