Distributed Training¶
Overview¶
Training on multiple GPUs/nodes is not just "more devices" — it's a coordination problem: gradient synchronization (DDP), state sharding (FSDP), and model decomposition (TP/PP). The right strategy depends on model size, memory budget, and interconnect speed.
Topics¶
- DDP in Depth - Data parallel + ring all-reduce of gradients; the default multi-GPU path.
- FSDP Deep Dive - Sharded optimizer/params/activations; the LLM-scale strategy.
- Tensor & Pipeline Parallelism - Split the model, not the data; when and how.
- Checkpointing & Fault Tolerance - Saving state correctly across processes, resuming, and disaster recovery.
Key Patterns¶
- Start with DDP; reach for FSDP when one GPU can't hold the model+optimizer slop.
- Synchronize gradients before optimizer.step(); never let ranks diverge.
- Always save/load distributed checkpoints rank-aware (or use a dedicated save rank).
- Measure comm time / compute time — that ratio decides which parallel strategy.