Skip to content

Mixed Precision & Quantization

Overview

Do more with less: run matmuls in fp16/bf16 (2x faster, half memory) and ship inference in int8/int4 (4-8x smaller, faster). The two tracks are complementary: training precision (AMP) protects gradients; inference quantization (PTQ/QAT) shrinks the model.

Topics

  • Automatic Mixed Precision - The safe way to train in half precision with GradScaler.
  • BF16, FP8 & Low-Precision Numerics - The dtypes of frontier training, and their failure modes.
  • Post-Training Quantization - INT8/FP8 conversion without retraining.
  • Quantization-Aware Training - Bake quantization into training; the accuracy-preserving path.

Key Patterns

  • AMP: fp16 matmuls + fp32 master weights + GradScaler; or bf16 with no scaler.
  • Quantization: tensor-per-channel scaling; symmetric vs asymmetric; calibrate on real data.
  • Always validate accuracy after any precision change— never assume.