Skip to content

Precision & Numerics — dtypes, fp16, bf16, fp8

Overview

Every dtype is a {range, precision, storage-cost} trade. Understand the bits and bias of each before choosing, because "float16 beats float32" is only true when your numbers actually fit.

  • float32 (fp32): 24-bit mantissa, ~1e-38..3e38 — the default, biggest range/precision.
  • float16 (fp16): 11-bit mantissa + limited exponent range (~±65504 max; subnormals below ~6e-5).
  • bfloat16 (bf16): 8-bit exponent (like fp32 range!) but only 8-bit mantissa — huge range, low precision, made for training.
  • float8 (e4m3/e5m2): extremely narrow; needs scaling by design.

💡 Gradients live near zero → fp16's tiny underflow threshold destroys them. This is exactly why bf16 exists, and why fp32 master weights + GradScaler are required for fp16.

Range vs Precision Cheat-Sheet

dtype Bits total Exponent Mantissa Max Smallest normal Typical use
float32 32 8 23 ~3.4e38 ~1.2e-38 default, master weights
float16 16 5 10 65504 ~6.1e-5 fp16 AMP/quantization
bfloat16 16 8 7 ~3.4e38 ~1.2e-38 training AMP
float8 e4m3 8 4 3 ~448 ~6e-3 frontier training/inference 
import torch
import struct

def info(dtype):
    t = torch.tensor(1.0, dtype=dtype)
    return "ok"
for dt in [torch.float32, torch.float16, torch.bfloat16]:
    t = torch.tensor([1.0, 2.0, 4.0], dtype=dt)
    print(dt, "-> next value after 1.0 represents", end=" ")
    print(f"", "see torch.finfo")
    print("   finfo:", torch.finfo(dt))

fp16 vs bf16 — The Exponent Problem

import torch
import numpy as np

x_f32 = torch.tensor([0.00001, 1000.0, 65504.0], dtype=torch.float32)

xf = x_f32.to(torch.float16)
xb = x_f32.to(torch.bfloat16)
print("fp32 :", x_f32.tolist())
print("fp16:", xf.tolist(), " <-- 0.00001 collapses to ~0 (underflow)")
print("bf16:", xb.tolist(), " <-- large values survive, low precision")

print("precision: fp16 nextof(1.0)=", torch.finfo(torch.float16).eps,
      " bf16 eps=", torch.finfo(torch.bfloat16).eps)

Rule of thumb: - fp16 can't represent tiny weights/grads unless scaled — use with a GradScaler. - bf16 keeps fp32 range, so training is stable without a scaler; you trade mantissa bits.

Casting Rules & DType Promotion

PyTorch does NOT silently mix dtypes in arithmetic; promotion follows specific rules.

a = torch.tensor([1.0], dtype=torch.float32)
b = torch.tensor([1], dtype=torch.int64)
try:
    print(a + b)      # RuntimeError in modern torch for some combos
except RuntimeError as e:
    print("promotion error:", str(e)[:70])

# Result dtype follows the higher-priority of the operands
print((a.float() + b).dtype)
print((a + a.to(torch.float16)).dtype, "<- fp32 + fp16 -> fp32")

Special Values & Silent Corruption

import math
x = torch.tensor([0.0, 0.0], dtype=torch.float32)
q = x[0] / x[1]              # 0/0
print("nan:", q.item(), q.isnan().item())

i = torch.tensor([1.0]) / torch.tensor([0.0])
print("inf:", i.item(), "inf path:", i.isinf().item())

# Detect NaN/Inf before they poison gradients
has_bad = torch.isnan(x).any() or torch.isinf(x).any()
print("has nan/inf:", has_bad.item())

Safe dtype guard

def check_finite(t: torch.Tensor, name="tensor"):
    assert torch.isfinite(t).all(), f"non-finite values in {name}"

Casting in Hot Loops — Watch float() & .numpy()

x = torch.randn(4, device='cuda')
s = float(x.sum())      # syncs + converts (slow in loops)
# prefer staying on-tensor until the end:
tot = x.sum().item()    # still one sync, but vectorized before it

Practical Numeric Defense

  1. Use bf16 for training AMP unless fp16 + GradScaler is clearly faster on your hardware.
  2. Keep fp32 master weights; accumulate gradient updates in fp32.
  3. Normalize inputs (0-mean, low-variance) to keep activations within representable range.
  4. torch.amax/clamp activations before quantizing to fp8/int to avoid overflow.
  5. Log NaN checks at least on the loss each N steps in production training.
# minimal robust forward under fp16
import torch as T
class SafeLayer(T.nn.Module):
    def forward(self, x):
        return T.nn.functional.gelu(x.float())  # upcast for stable activation

Key Takeaways

  • Range can break you more than precision: fp16 underflows near zero, bf16/fp32 don't.
  • Choose dtype per the distribution of your values, not just "half is smaller".
  • Conversion/promotion rules silently bite — cast explicitly.
  • Detect NaN/Inf early; they silently poison gradients and roll silently into checkpoints.
  • Keep master weights fp32; scale or widen exponent before low-precision pointwise ops.