Advanced Training Techniques¶
Overview¶
The difference between "training works" and "training trains at scale, stably, reproducibly" is a toolbox of techniques: gradient accumulation, clipping, model averaging (EMA/SWA), activation checkpointing, and disciplined LR schedules. None are magic— each trades something (time, memory, complexity) for something else (stability, memory, final quality).
Topics¶
- Gradient Accumulation & Clipping - Fake big batches without memory; tame exploding grads.
- Model Averaging— EMA, SWA - Cheap ensemble effect and smoother loss landscapes.
- Activation Checkpointing & Memory Restructuring - Trade compute for activation memory in deep nets.
- Schedules, Warmup & Optimizer Tricks - LR shapes that actually help convergence.
Key Patterns¶
- Accumulate grads across micro-batches; divide by accumulation steps; sync only at real steps.
- Clip global norm (not per-param) for principled stability.
- EMA gives a better eval-time model for free— run it at inference.
- Checkpoint activations where memory-bound; keep the critical path uncheckpointed.
- Warmup + cosine/decay is the boring-but-robust schedule winner.