Post-Training Quantization

Overview

Post-Training Quantization (PTQ) converts a trained fp32 model to int8 (or fp8) without retraining — you calibrate scale/zero-point on a small dataset and swap weights. The win: 4x smaller, 2-4x faster inference. The risk: accuracy loss, worst on outlier-heavy distributions.

• Dynamic quantization: quantize weights only, activations stay fp16/fp32 (great for LLM/LSTM text inference).
• Static quantization: quantize both weights and activations using calibrated ranges (needs a calibration set).
• Symmetric vs asymmetric: symmetric (zero=0) simpler/faster; asymmetric better for skewed data (e.g., ReLU outputs).
• torch.ao.quantization (modern) / quantize_fx (fx-based) / torch.quantization (legacy).

💡 Rule of thumb: dynamic quantization is a free 2-3x on CPU text models; static gets closer to 4x but needs calibration care.

---

Dynamic Quantization (weights-only)

import torch

class LM(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.emb = torch.nn.Embedding(1000, 64)
        self.lin = torch.nn.Linear(64, 1000)

    def forward(self, ids):
        return self.lin(self.emb(ids))

model = LM().eval()
dq = torch.ao.quantization.quantize_dynamic(model, {torch.nn.Linear}, dtype=torch.qint8)
print("orig size (bytes):", sum(p.numel() * 4 for p in model.parameters()))
print("dq   size (bytes):", sum(p.numel() for p in dq.parameters()))   # int8

✅ {torch.nn.Linear, torch.nn.LSTM} — pick modules to convert; others stay fp32.

---

Static Quantization — the Full Recipe

1. Fuse + prepare

import torch, torch.nn as nn

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv = nn.Conv2d(3, 8, 3, padding=1)
        self.relu = nn.ReLU()
        self.fc = nn.Linear(8 * 8 * 8, 2)
    def forward(self, x):
        return self.fc(self.relu(self.conv(x)).flatten(1))

net = Net().eval()
net.qconfig = torch.ao.quantization.get_default_qconfig("x86")      # per-tensor, symmetric
net_fused = torch.ao.quantization.fuse_modules(net, ["Conv", "Relu"](/"conv",-"relu"/))
net_prep = torch.ao.quantization.prepare(net_fused, inplace=False)

2. Calibrate on real data

def calibrate(model, loader, n=10):
    model.eval()
    with torch.no_grad():
        for i, (x, _) in enumerate(loader):
            model(x)
            if i >= n: break

calibrate(net_prep, calib_loader)          # observes activation ranges

⚠️ Calibration data should match deployment distribution; garbage-in → garbage scales → accuracy loss.

3. Convert

net_q = torch.ao.quantization.convert(net_prep, inplace=False)
print(net_q)   # now quantized ops
out = net_q(torch.randn(1, 3, 8, 8))       # int8 kernels on CPU

---

Symmetric vs Asymmetric, Per-Channel

# per-channel (better for weights; more scales to store)
qconfig = torch.ao.quantization.get_default_qconfig("fbgemm")
# custom:
from torch.ao.quantization import QConfig, MinMaxObserver, PerChannelMinMaxObserver
qconfig = QConfig(
    activation=MinMaxObserver.with_args(dtype=torch.quint8, qscheme=torch.per_tensor_affine),
    weight=PerChannelMinMaxObserver.with_args(dtype=torch.qint8, qscheme=torch.per_channel_symmetric),
)

Scheme	When
per-tensor symmetric	balanced weights (typical)
per-channel symmetric (weights)	big range differences across filters
per-tensor asymmetric	skewed activations (ReLU positive)

---

Accuracy-Checking Discipline

def eval_accuracy(model, loader, device="cpu"):
    model.eval()
    correct = total = 0
    with torch.no_grad():
        for x, y in loader:
            pred = model(x).argmax(1)
            correct += (pred == y).sum().item()
            total += y.numel()
    return correct / total

before = eval_accuracy(net, loader)
after  = eval_accuracy(net_q, loader)
print(f"fp32 acc {before:.4f} | int8 acc {after:.4f} | drop {before-after:.4f}")
# typical acceptable drop: <1% for int8; if more, try QAT (Ch 06-04)

---

PTQ vs QAT Decision

Signal	Choice
Drop < 0.5-1%	PTQ is fine
Drop 1-5%	try per-channel + better calibration first
Drop > 5%	QAT (Ch 06-04) or mixed-precision quantization

---

Deployment Notes

• Quantized models are CPU-first in core torch; GPU int8 needs TensorRT / torch_tensorrt or torchao fp8.
• Export: quantize then torch.jit/torch.export (Ch 07); ONNX has its own quantize pass.
• Always benchmark both latency and accuracy; small acc gains never justify 2x latency.

---

Key Takeaways

• Dynamic (weights-only) = free win on text/LLM; static = bigger win, needs calibration.
• Calibrate on representative data, fuse conv+relu, choose symmetric/per-channel by distribution.
• Validate accuracy vs fp32 before deploying; QAT if the drop is real.
• Core torch quant is CPU-oriented; GPU paths go through TRT / torchao.
• Keep the fp32 model around for fallback and comparison.

---

Related Topics

• Qat
• Fp8
• Export