Pruning & Sparsity: Removing Redundant Parameters and Activations

Overview

Pruning removes less important weights or neurons from models. Sparsity enables efficient computation by skipping zero/low-value operations. Combined: 2-10x model size reduction with minimal quality loss.

• Approaches: Weight pruning, neuron pruning, structured vs unstructured
• Sparsity Types: Weight sparsity (80%+), activation sparsity (90%+)
• Speedup: 2-5x for unstructured, 1.5-3x for structured (hardware-dependent)
• Quality Loss: 1-5% typical with careful pruning
• Trade-off: Model size reduction vs inference speed complexity

---

The Insight: Most Parameters Aren't Important

Weight Importance Analysis

Research finding: Deep networks are over-parameterized

Typical distribution of weight magnitudes:

                    Probability
                    ↑
                    |  ▁▂▃▄▅▆▇█ (many weights)
                    |  ▇▅▄▃▂▁   (few large weights)
                    |
  - → Weight magnitude

Observation:
  - ~10-20% of weights are large (important)
  - ~80-90% of weights are small (less important)
  - Removing small weights: Model barely changes!
  - Pruning opportunity: Remove redundant small weights

Example (ResNet-50):
  - Total parameters: 25.5M
  - After 90% pruning: 2.5M parameters
  - Quality loss: 1-2%
  - Speedup: 3x (depends on hardware)

---

Pruning Strategies

1. Magnitude-Based Pruning (Simplest)

Idea: Remove weights below certain magnitude threshold

Algorithm:
1. Train model normally
2. Compute magnitude of each weight
3. Remove weights where |w| < threshold
4. (Optional) Fine-tune on remaining weights

Code:
```python
def magnitude_prune(model, sparsity_ratio=0.9):
    """
    Remove weights with smallest magnitudes

    Args:
        sparsity_ratio: Fraction of weights to remove (0.9 = remove 90%)
    """
    for name, param in model.named_parameters():
        if 'weight' in name:  # Only prune weights, not biases
            # Compute magnitude
            magnitude = torch.abs(param.data)

            # Find threshold
            k = int(param.numel() * sparsity_ratio)
            threshold = torch.kthvalue(magnitude.view(-1), k)[0]

            # Create mask
            mask = magnitude > threshold

            # Apply mask (set small weights to zero)
            param.data = param.data * mask.float()

# Usage:
model = load_pretrained_model()
magnitude_prune(model, sparsity_ratio=0.9)
# Model now has 90% weights set to zero!

Pros: ✅ Simple to implement ✅ Works with any model ✅ No retraining necessary

Cons: ❌ Unstructured (not all hardware optimized for sparse computation) ❌ Quality drops without fine-tuning ❌ Threshold selection is manual

2. Structured Pruning (More Efficient)

Instead of removing individual weights, remove entire channels/filters

Standard pruning (unstructured):
  - Remove scattered weights
  - Model still has same dimensions
  - Hardware must skip scattered computations
  - Not all hardware supports (complexity!)

Structured pruning:
  - Remove entire filters/channels
  - Actual model size and speed improve
  - Works with any hardware
  - But: Must remove full structures (less flexible)

Example: Prune filter from CNN

Before:
Input: (batch, 64, H, W)
   ↓
Filter layer (64 → 128 filters): each 3×3×64
   ↓
Output: (batch, 128, H, W)

After pruning (remove 32 filters):
Input: (batch, 64, H, W)
   ↓
Filter layer (64 → 96 filters): each 3×3×64
   ↓
Output: (batch, 96, H, W)

Benefit:
  - Actual computation reduced (matrix multiply on 96 not 128)
  - Memory reduced (96 filters < 128 filters)
  - All hardware supports (just smaller matrix)

3. Iterative Magnitude Pruning (Lottery Ticket)

Key finding: Pruning gradually better than one-shot

Approach:
1. Train model to convergence
2. Prune 20% of weights (magnitude-based)
3. Reset remaining weights to initial values (!)
4. Train again to convergence
5. Prune another 20% of remaining
6. Repeat until target sparsity

Result:
  - Iterative: 50% quality loss at 99% sparsity
  - One-shot: 90% quality loss at 99% sparsity
  - Iterative is MUCH better!

Why it works:
- Early iteration: Remove clearly unimportant weights
- Later iteration: Remove more nuanced patterns
- Gradual reduction better than aggressive one-shot

Code:
```python
def iterative_pruning(model, data, target_sparsity=0.9, steps=5):
    current_sparsity = 0.0
    sparsity_per_step = target_sparsity / steps

    for step in range(steps):
        # Train current model
        train_model(model, data, epochs=10)

        # Prune
        current_sparsity += sparsity_per_step
        magnitude_prune(model, current_sparsity)

        # Fine-tune
        train_model(model, data, epochs=5)

---

## Activation Sparsity

Different from weight sparsity!

Weight sparsity: Remove weights from model Activation sparsity: Skip computation when activations are zero

Example: Model with ReLU: - ReLU sets negative values to zero - Many activations are zero! - Can skip multiply-accumulate operations for zero activations - Potential: Skip 50-90% of compute!

Hardware support: - Traditional GPUs: No special support - Sparse tensor libraries: TensorRT, cuSPARSE - Some speedup but not proportional to sparsity - Challenge: Controlling sparsity patterns

---

## Knowledge Distillation vs Pruning

### Comparison

Technique Size Red. Speed Quality Loss Effort ────────────────────────────────────────────────────── Pruning 3-10x 2-5x 2-5% Medium Distillation 10-50x 10-100x 5-15% High Quantization 2-4x 1.5-2x 1-3% Low Combined 20-100x 20-50x 5-10% Very High

Pruning strategy: - Simpler than distillation - Works with existing model - Gradual quality loss - Good baseline optimization

Distillation strategy: - Requires teacher model - More aggressive compression - Better final quality - Use after pruning for maximum compression

---

## Sparsity-Aware Hardware

### Tensor Engines

Modern GPUs: - NVIDIA A100: Tensor cores with structured sparsity support - 2x speedup for 50% structured sparsity - NVIDIA H100: Better sparse support - More efficient sparse operations - AMD MI300: Sparse tensor support

CPUs: - Can utilize sparsity via special libraries - Smaller speedup than GPUs - Better for 5-10% sparsity

Specialized hardware: - SambaNova: Sparse tensor processor - Cerebras: Sparse network support - Groq: Optimized for inference sparsity - Custom chips: Designed for specific sparsity patterns

---

## When to Use Pruning

### Use Pruning When

✅ Model size is critical (mobile/edge) ✅ Hardware supports sparse operations ✅ Can accept 2-5% quality loss ✅ Inference speed important ✅ Model was over-trained / over-parameterized

### Avoid Pruning When

❌ Model is already optimized ❌ Hardware doesn't support sparsity ❌ Quality cannot degrade ❌ Simplicity is important

---

## Practical Pruning Pipeline

1. Train model to convergence - Get baseline quality

1. Evaluate structured vs unstructured
2. Structured: Check if hardware supports
3. Unstructured: Requires sparse tensor support

4. Usually: Start with structured

5. Apply iterative pruning

6. Step 1: Prune 20%, fine-tune
7. Step 2: Prune 20% more, fine-tune

8. Repeat until target sparsity

9. Measure quality and speed

10. Quality: MMLU, benchmark on your task
11. Speed: Profile inference on target hardware

12. If unsatisfactory, reduce sparsity or stop

13. Deploy with sparse kernels

14. Use sparse tensor library for inference
15. Verify speedup on actual hardware
16. Monitor quality in production ```

---

Key Takeaways

✂️ Most parameters redundant: 90% can be pruned with care
🎯 Iterative pruning much better than one-shot
⚙️ Structured pruning more hardware-friendly
📊 2-5x compression with 2-5% quality loss typical
🔄 Combine with quantization and distillation for max compression

---

Related Notes

• Quantization - Combine for maximum compression
• Model Distillation - More aggressive compression
• Llm Inference Optimization - Complete optimization stack