Skip to content

BF16, FP8 & Low-Precision Numerics

Overview

Beyond fp16 lies the frontier: bf16 (fp32-range, low precision) and FP8 (e4m3/e5m2, ~8-bit) power modern LLM training and inference. The win is memory and speed; the cost is careful numeric engineering: scaling, loss scaling, and overflow watch.

  • bf16: 8-bit exponent (same range as fp32), 7-bit mantissa. Tolerates tiny gradients — no scaler.
  • e4m3 (FP8): 4-bit exponent, 3-bit mantissa, max ~448 — great for activations/weights in inference, tiny range.
  • e5m2 (FP8): 5-bit exponent, 2-bit mantissa, max ~57344 — larger range, coarser, good for gradients.
  • FP8 needs scaling (per-tensor or per-channel) to land values in its narrow range.

💡 The #1 FP8 mistake: values outside [~6e-3, ~448] round to 0 or inf silently. Always scale before casting.

bf16 — Why LLMs Use It

import torch

x = torch.tensor([1e-6, 1e-3, 1e5], dtype=torch.float32)
print("fp32 :", x.tolist())
print("fp16 :", x.to(torch.float16).tolist())   # 1e-6 -> 0 (underflow!)
print("bf16 :", x.to(torch.bfloat16).tolist())  # range preserved, precision lost

# bf16 matmul — no scaler, same code as fp32:
with torch.amp.autocast("cuda", dtype=torch.bfloat16):
    y = model(x)   # works with fp32-loss immediately

✅ bf16 + fp32 master weights is the standard LLM recipe: 2x memory win, stable gradients, zero scaler drama.

FP8 — The 8-Bit Frontier

import torch

# two fp8 dtypes
print(torch.finfo(torch.float8_e4m3fn))   # max ~448, 3-bit mantissa
print(torch.finfo(torch.float8_e5m2))     # max ~57344, 2-bit mantissa

x = torch.tensor([0.5, 1.0, 100.0], dtype=torch.float32)
print(x.to(torch.float8_e4m3fn))          # 100 is fine; >448 would clamp to inf

The scaling dance (per-tensor)

def to_fp8(x, dtype=torch.float8_e4m3fn):
    scale = torch.finfo(dtype).max / x.abs().max()
    return (x * scale).to(dtype), scale   # scale to fit range

def from_fp8(qx, scale):
    return qx.to(torch.float32) / scale

💡 On H100+ with tensor cores, FP8 GEMMs run 2x bf16 speed. Frameworks (torchao, DeepSpeed, NeMo) handle scaling internally — but you must still verify with your own calibration data.

Choosing the Right Low-Precision Path

Component Typical dtype Why
Weights (train) fp32 master / bf16 copy accumulate in fp32
Activations (train) fp16/bf16 autocast range decides fp16 vs bf16
Gradients (train) fp32 (or e5m2 in FP8 training) range matters most
Weights (inference) int8/fp8/e4m3 memory-bound wins
KV cache (inference) fp8/int8 huge memory win

Numeric Failure Modes & Guards

def guard_fp8(x, name="tensor"):
    finfo = torch.finfo(torch.float8_e4m3fn)
    ratio = x.abs().max() / finfo.max
    if ratio > 0.8:
        print(f"WARNING: {name} near FP8 max, scale needed: {ratio:.2f}")

# NaN/Inf detector for low-precision training
def check_finite(t, name):
    if not torch.isfinite(t).all():
        print(f"NON-FINITE in {name}"); return False
    return True

Loss Scaling in Low Precision (beyond fp16)

fp8 training often scales gradients per-layer or per-tensor; bf16 usually skips it. General pattern:

class DynamicGradScale:
    """Very simplified per-step scale estimator."""
    def __init__(self, init=1.0):
        self.s = init
    def update(self, grad_norm):
        if not torch.isfinite(grad_norm):
            self.s *= 0.5
        elif grad_norm < 1e-4:
            self.s *= 2.0
        return self.s

⚠️ For real FP8 training use proven libraries (torchao, NVIDIA transformer-engine) — hand-rolled scaling is a research project.

Practical Decision Rules

  1. bf16 everywhere for training unless your hardware lacks fast bf16 (then fp16+scaler).
  2. FP8 for inference & frontier training only with a scaling-aware library.
  3. Never cast to fp8 without scaling, and clamp per-tensor.
  4. Keep fp32 for: master weights, loss, embeddings in huge-vocab models, and any reduction.
  5. Validate with a tiny subset before committing a whole run.
def safe_bf16_fp32(model):
    # common pattern: bf16 activations, fp32 master weights handled by optimizer
    return model.to(torch.bfloat16)   # + separate fp32 optimizer copy in frameworks

Key Takeaways

  • bf16 = fp32 range, lower precision — safe default for training.
  • FP8 = 8 bits with big range constraints — scaling is mandatory, use libraries.
  • e4m3 for weights/activations, e5m2 when range is the concern (gradients).
  • Master weights and reductions stay fp32; only compute-heavy ops drop precision.
  • Always monitor non-finite values and validate accuracy vs fp32 baseline.