Mixed Precision Training

Authors: Micikevicius et al. Year: 2017 ArXiv/Link: https://arxiv.org/abs/1710.03740

Summary

Demonstrated that training with mixed FP16/FP32 precision achieves faster training and reduced memory without accuracy loss.

Key Concepts

• Mixed precision
• FP16 training
• Numerical stability
• Gradient scaling
• Training speedup

Impact

Practical technique enabling faster training of large models

Category

Optimization & Efficiency

---