GPTQ Quantization¶

Overview¶

GPTQ (GPT Quantization) is a post-training quantization technique that reduces Large Language Model weights to lower precision (typically int4 or int8) while maintaining near-original accuracy. It's one of the most practical and widely-used quantization methods for LLMs.

Paper: "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers"
Authors: Frantar et al., IST Austria (2023)
Key Innovation: Single-GPU quantization with minimal accuracy loss
Impact: 4x model compression, 2-4x inference speedup
Adoption: Widely used in production (vLLM, Ollama, HF Hub)

-

The Problem: Why Quantize LLMs?¶

Model Size Challenges¶

Model Sizes (float32 full precision):

Llama 2 7B:
 - Float32 (4 bytes/param): 7B × 4 = 28GB
 - Float16 (2 bytes/param): 7B × 2 = 14GB
 - Int8 (1 byte/param): 7B × 1 = 7GB
 - Int4 (0.5 bytes/param): 7B × 0.5 = 3.5GB

Llama 2 70B:
 - Float32: 70B × 4 = 280GB (no single GPU can fit!)
 - Float16: 70B × 2 = 140GB (only 8x A100s)
 - Int8: 70B × 1 = 70GB (2x A100 with effort)
 - Int4: 70B × 0.5 = 35GB (single A100 GPU!)

Hardware Constraints¶

Available GPU Memory:
 - Consumer GPU (RTX 4090): 24GB
 - Data Center GPU (A100): 40GB or 80GB
 - Multiple GPUs: Expensive and complex

Without Quantization:
 - Llama 2 7B: Needs 14-16GB GPU (close call!)
 - Llama 2 13B: Needs 26-30GB GPU (multiple GPUs)
 - Llama 2 70B: Needs 140GB+ GPU (8x GPUs!)

With GPTQ (int4):
 - Llama 2 7B: Fits in 8GB (consumer GPU!)
 - Llama 2 13B: Fits in 16GB (RTX 4090)
 - Llama 2 70B: Fits in 40GB (single A100)

Inference Speed Bottlenecks¶

Memory Bandwidth vs Compute Power:
GPU A100: 
 - FP32 throughput: 312 TFLOPS
 - Memory bandwidth: 2 TB/s
 - Byte ratio: 2 TB/s ÷ 312 TFLOPS = 6.4 bytes/FLOP

Loading weights:
 - FP32 model: Load 4 bytes/param
 - Time to load Llama 7B: (7B × 4 bytes) / 2TB/s = 14ms
 - Compute time for 1 token: ~3ms
 - Bottleneck: Memory, not compute!

With Int4:
 - Model size: 3.5GB
 - Load time: (7B × 0.5 bytes) / 2TB/s = 1.75ms
 - Net speedup: 4x reduction in memory bandwidth needed
 - Plus: Better cache locality, fewer memory stalls

-

What is Quantization?¶

Basic Concept¶

Quantization reduces precision of model weights:

Float32 (Full Precision):
 - 32 bits per value
 - Range: ±3.4e38
 - Precision: ~7 decimal digits
 - Example: 0.73825949 → 0.73825949

Float16 (Half Precision):
 - 16 bits per value (2x smaller)
 - Range: ±65504
 - Precision: ~3-4 decimal digits
 - Example: 0.73825949 → 0.7383 (rounded)

Int8 (1 byte per value):
 - 8 bits per value (4x smaller)
 - Range: -128 to 127
 - Integers only
 - Needs scaling: 0.73825949 → scale to [-128, 127]

Int4 (0.5 bytes per value):
 - 4 bits per value (8x smaller!)
 - Range: -8 to 7
 - Extreme precision loss
 - Heavy quantization needed

Quantization Equation¶

Quantized value = round(original_value / scale)

Example:
original_value = 0.75
scale = 0.1
quantized = round(0.75 / 0.1) = round(7.5) = 8

To recover (dequantize):
recovered_value = quantized × scale = 8 × 0.1 = 0.8
error =|0.75 - 0.8| = 0.05 (small!)

Why GPTQ Works: The Key Insight¶

The Observation¶

Not all weights are equally important!

In a 7B model:
- Some weights critically affect output
- Others have minimal impact
- We can quantize "unimportant" weights more aggressively
- We can quantize "important" weights less aggressively

GPTQ's Innovation:
Quantize one layer at a time, and track which weights
matter most using the Hessian (curvature information)

Hessian-Based Importance¶

The Hessian matrix H tells us:
"How much does the loss increase if we perturb this weight?"

High Hessian value = Important weight (sensitive)
Low Hessian value = Unimportant weight (robust)

GPTQ Strategy:
1. Compute Hessian for the layer
2. Quantize least important weights first
3. Adjust remaining weights to compensate
4. Move to next layer

-

The GPTQ Algorithm: Step-by-Step¶

Overview: Three Phases¶

Phase 1: Preparation
 - Load model
 - Compute Hessian for first layer
 - Sort weights by importance

Phase 2: Quantization (per layer)
 - For each weight:
 - Calculate optimal quantization
 - Quantize the weight
 - Adjust remaining weights to minimize error
 - Move to next layer

Phase 3: Finalization
 - Save quantized model
 - Verify accuracy on calibration set

Detailed Algorithm¶

GPTQ Algorithm Pseudocode:

for each layer L in model:
 # Step 1: Compute Hessian (Fisher Information)
 H = Hessian(layer_L) # (weights × weights) matrix
 H_inv = inverse(H) # Inverse (more efficient)

 # Step 2: Prepare quantization
 quantization_info = prepare_quantization(layer_L)

 for each weight w in layer_L (in order of importance):
 # Step 3: Find optimal quantization
 # Try all possible quantized values
 q = quantize(w) # Round to nearest int4/int8

 # Step 4: Calculate error
 error = w - (q × scale)

 # Step 5: Optimal compensation (key insight!)
 # Update all remaining weights to minimize total error
 # Using Hessian information: H_inv × error
 remaining_weights = remaining_weights - (H_inv × error)

 # Step 6: Update Hessian
 # Remove this weight from Hessian (matrix update)
 H_inv = update_hessian(H_inv, w, error)

 save_layer(layer_L, quantized=True)

Example: Quantizing a Single Weight¶

Layer: Linear weight matrix
Shape: (4096, 4096)

Scenario: Quantize first weight to int4

Before:
w[0,0] = 0.7382594

Step 1: Calculate Hessian for this layer
H[0,0] = 5.2 (this weight is fairly important)

Step 2: Quantize to int4
int4 range: [-8, 7]
Scale for this layer: 0.1
Quantized: round(0.7382594 / 0.1) = round(7.38) = 7
Dequantized: 7 × 0.1 = 0.7

Step 3: Calculate error
error = 0.7382594 - 0.7 = 0.0382594

Step 4: Compensate remaining weights
For each other weight w_j:
 compensation = H_inv[0, j] × error
 w_j = w_j - compensation

This spreads the quantization error across all weights
using the Hessian as a guide!

Result:
- w[0,0] is now quantized (saves 4 bits)
- Other weights adjusted to compensate (maintain accuracy)
- Net effect: Minimal accuracy loss!

-

Mathematical Foundation: The Hessian¶

Why Hessian Matters¶

Loss function: L = MSE(quantized_output - original_output)

First derivative (gradient):
∇L/∂w = tells us direction to minimize loss

Second derivative (Hessian):
H = ∂²L/∂w² = tells us "curvature" or sensitivity

High Hessian value:
 - Weight is in a sharp valley
 - Small change → big loss increase
 - Important weight! Quantize carefully.

Low Hessian value:
 - Weight is on a flat region
 - Small change → small loss change
 - Unimportant weight. Can quantize aggressively.

Computing the Hessian¶

For a neural network layer:
Input: x (activation)
Weight: W
Output: y = W^T × x

Hessian H = E[x × x^T]
This is the outer product of activations!

Efficient computation:
1. Run inference on calibration data
2. Collect activations for each layer
3. Compute H = sum(x_i × x_i^T) for each sample
4. Average across samples

Approximation (GPTQ trick):
- Only compute diagonal of H for efficiency
- Or use block-wise computation
- Makes algorithm tractable for 70B models!

-

Quantization Methods: INT4 vs INT8¶

INT4 Quantization (4-bit)¶

Range: -8 to 7 (16 values)
Bits per weight: 4
Model compression: 8x (float32) or 4x (float16)

Advantages:
 - Maximum compression (3.5GB for 7B model!)
 - 2-4x inference speedup
 - Fits in consumer GPU memory
 - Very practical for deployment

Disadvantages:
 - Highest quantization error
 - More challenging to maintain accuracy
 - Requires careful calibration
 - May need more calibration data

Use case: When model size is critical
 - Consumer GPU deployment
 - Mobile inference
 - Batch serving with limited memory

INT8 Quantization (8-bit)¶

Range: -128 to 127 (256 values)
Bits per weight: 8
Model compression: 4x (float32) or 2x (float16)

Advantages:
 - Easier to achieve high accuracy
 - Faster quantization process
 - More robust
 - Good balance of size/speed

Disadvantages:
 - Less aggressive compression (7GB for 7B model)
 - Smaller speedup (1.5-2x)
 - Still requires careful quantization

Use case: When accuracy is more critical
 - Critical production systems
 - Higher quality requirements
 - Fine-tuned models

Mixed-Precision Quantization¶

Strategy: Use different precision for different layers!

Observation:
- Last layers are more sensitive
- Early layers are more robust

Approach:
 - Layers 1-16: Int4 (robust)
 - Layers 17-24: Int4 (robust)
 - Layers 25-31: Int8 (sensitive)
 - Layer 32: Int8 (very sensitive)
 - Attention/Output: Int8 (critical)

Result:
- Average precision: ~5-6 bits
- Compression: ~6x
- Accuracy: Near original
- Speedup: Better than pure int8

Example sizes:
 - Llama 2 7B mixed: 4-5GB
 - Llama 2 70B mixed: 40-50GB

-

GPTQ in Practice: Complete Workflow¶

Step 1: Prepare Calibration Data¶

from datasets import load_dataset

# Load calibration data (small sample)
dataset = load_dataset('wikitext', 'wikitext-2-raw-v1', split='train')

# Take first 128 samples for calibration
calibration_data = dataset['text'][:128]

# Why calibration data?
 - # ├─ Compute accurate Hessian
 - # ├─ Estimate quantization error distribution
 - # ├─ Find good scale factors
 - # └─ Only need ~100-1000 samples (fast!)

Step 2: Quantize with AutoGPTQ¶

from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
from transformers import AutoTokenizer

# Configuration
quantize_config = BaseQuantizeConfig(
 bits=4, # INT4 quantization
 group_size=128, # Group size for scale factors
 desc_act=False, # Desc act (advanced, usually False)
 static_groups=False, # Static groups
 true_sequential=True, # Quantize sequentially (accurate but slower)
 damp_percent=0.1, # Damping for Hessian (numerical stability)
)

# Load model
model_name = "meta-llama/Llama-2-7b-hf"
model = AutoGPTQForCausalLM.from_pretrained(
 model_name,
 quantize_config=quantize_config,
 device="cuda:0"
)

tokenizer = AutoTokenizer.from_pretrained(model_name)

# Tokenize calibration data
examples = [
 tokenizer(example, return_tensors="pt") 
 for example in calibration_data
]

# Quantize! (This will take time, ~30 min for 7B on A100)
model.quantize(examples, cache_examples_on_gpu=False)

# Save quantized model
model.save_quantized(
 "./llama-2-7b-gptq",
 use_safetensors=True
)

Step 3: Use Quantized Model for Inference¶

from auto_gptq import AutoGPTQForCausalLM
from transformers import AutoTokenizer

# Load quantized model (much faster than quantizing!)
model = AutoGPTQForCausalLM.from_quantized(
 "./llama-2-7b-gptq",
 device="cuda:0",
 use_safetensors=True
)

tokenizer = AutoTokenizer.from_pretrained(
 "meta-llama/Llama-2-7b-hf"
)

# Generate with quantized model
prompt = "Explain quantum computing"
inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
 **inputs,
 max_new_tokens=256,
 do_sample=True,
 top_p=0.95,
 temperature=0.7
)

print(tokenizer.decode(outputs[0]))

Step 4: Integration with vLLM¶

from vllm import LLM, SamplingParams

# vLLM automatically detects GPTQ quantization!
llm = LLM(
 model="./llama-2-7b-gptq",
 quantization="gptq", # Explicitly specify
 dtype="half", # Use float16 for remaining compute
 gpu_memory_utilization=0.95
)

prompts = [
 "What is machine learning?",
 "Explain deep learning",
 "How do transformers work?"
]

sampling_params = SamplingParams(
 temperature=0.7,
 top_p=0.95,
 max_tokens=256
)

outputs = llm.generate(prompts, sampling_params)

for output in outputs:
 print(output.outputs[0].text)

-

Performance Metrics & Accuracy Loss¶

Benchmark Results: Llama 2 7B¶

Configuration:
 - Model: Llama 2 7B
 - Calibration: 128 samples from WikiText
 - Evaluation: Common benchmarks

 Original INT8 INT4 INT4+Damp
────────────────────────────────────────────────────
Model Size 14GB 7GB 3.5GB 3.5GB
Inference Speed 1x 1.8x 2.4x 2.3x
Memory Used 14GB 7GB 3.8GB 3.8GB

Accuracy (ARC-Challenge, higher is better):
 Original INT8 INT4 
────────────────────────────────────────
Score 53.8% 53.1% 52.2%
Loss 0% -0.7% -1.6%

Accuracy (MMLU, higher is better):
 Original INT8 INT4 
────────────────────────────────────────
Score 45.9% 45.3% 44.7%
Loss 0% -1.3% -2.6%

Perplexity (WikiText2, lower is better):
 Original INT8 INT4 
────────────────────────────────────────
Perplexity 6.02 6.11 6.28
Loss 0% +1.5% +4.3%

Key Insight:
- INT8: Almost no loss (~1%)
- INT4: Small loss (~2-4%)
- GPTQ is better than other quant methods!

Comparison: GPTQ vs Other Methods¶

Method Compression Speed Accuracy Difficulty
─────────────────────────────────────────────────────────────
Full Precision 1x 1x 100% N/A
Naive INT4 8x 4x 60-70% Easy (bad)
Quantization Aware 8x 4x 75-85% Hard (medium)
GPTQ INT4 8x 4x 95-98% Hard (good!)
GPTQ INT8 4x 2x 99% Easy (excellent)

Key: GPTQ achieves best accuracy for given compression level!

-

Configuration Deep Dive¶

Group Size¶

What is group_size?

Model weights are divided into groups.
Each group gets its own scale factor.

Smaller group_size:
 - More scale factors (more memory)
 - Better accuracy (fine-grained)
 - Slower inference (more overhead)
 - Typical: 128, 256

Larger group_size:
 - Fewer scale factors (less memory)
 - Worse accuracy (coarse-grained)
 - Faster inference (less overhead)
 - Typical: 1024, -1 (entire weight matrix)

Example: Weight matrix (4096 × 4096)

group_size=128:
 - Divide into (4096/128) × (4096/128) = 32 × 32 = 1024 groups
 - Each group has 128 × 128 = 16,384 weights
 - 1024 scale factors (2KB for float32)
 - Memory overhead: minimal (~0.5%)

group_size=1024:
 - Divide into (4096/1024) × (4096/1024) = 4 × 4 = 16 groups
 - Each group has 1024 × 1024 = 1M weights
 - 16 scale factors (64 bytes)
 - Memory overhead: minimal

Recommended: 128 for best accuracy

Damping Factor¶

What is damp_percent?

During quantization, numerical issues can occur.
The Hessian might be singular or ill-conditioned.

Damping adds regularization:
H_damped = H + damp_percent × trace(H) × I

damp_percent = 0.01:
 - Light damping (1%)
 - Better accuracy
 - May have numerical issues

damp_percent = 0.1:
 - Moderate damping (10%)
 - Good balance
 - Recommended default

damp_percent = 1.0:
 - Heavy damping (100%)
 - Safer numerically
 - Might hurt accuracy

Recommended: 0.01 for accuracy, 0.1 for stability

True Sequential¶

Quantization order matters!

true_sequential=True:
 - Quantize weights in order (first to last)
 - More accurate (uses latest compensation info)
 - Slower (O(n²) complexity)
 - Can take 1-2 hours for 70B

true_sequential=False:
 - Quantize in groups/blocks
 - Faster (O(n) complexity)
 - Slightly less accurate
 - Takes ~30 min for 70B

Recommended: True for best quality, False for speed

-

Real-World Examples¶

Example 1: Quantizing Llama 2 70B¶

Scenario:
- Hardware: Single A100 (40GB)
- Original model size: 140GB (won't fit!)
- Goal: Fit on single GPU

Solution: GPTQ INT4 quantization

Process:
1. Load model in 8-bit (uses CPU offloading): 15 min
2. Calibrate on 128 samples: 5 min
3. Quantize with GPTQ: 60-90 min (true_sequential=True)
4. Save: 5 min

Result:
 - Model size: 35GB (fits on A100!)
 - Load time: 2 min (vs would never load)
 - Inference speed: 2.5x faster
 - Accuracy loss: ~2-3% (acceptable)

Actual code:
```python
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig

# Only quantize (no loading full model in GPU)
model_name = "meta-llama/Llama-2-70b-hf"

# Config for 70B
quantize_config = BaseQuantizeConfig(
 bits=4,
 group_size=128,
 true_sequential=False, # Use False for 70B (still ~60min)
 damp_percent=0.1,
)

# Load in 8-bit first to reduce memory
model = AutoGPTQForCausalLM.from_pretrained(
 model_name,
 quantize_config=quantize_config,
 device_map="auto", # Auto distribute across GPUs if needed
 load_in_8bit=True, # Use 8-bit for loading
)

# Quantize (this is the slow step)
import time
start = time.time()
model.quantize(examples, cache_examples_on_gpu=False)
print(f"Quantization took {(time.time() - start) / 60:.1f} minutes")

# Save
model.save_quantized("./llama-2-70b-gptq")

Example 2: Fine-Tuned Model Quantization¶

Scenario:
- Start with quantized Llama 2 7B
- Fine-tune on custom data
- Re-quantize to reduce size further

Process:
1. Load quantized model
2. Fine-tune on custom data (low-rank adapters)
3. Merge LoRA weights back
4. Re-quantize (different scales, better for data)

Code:
```python
from auto_gptq import AutoGPTQForCausalLM
from peft import get_peft_model, LoraConfig, TaskType

# Load pre-quantized model
model = AutoGPTQForCausalLM.from_quantized(
 "./llama-2-7b-gptq"
)

# Add LoRA for fine-tuning
peft_config = LoraConfig(
 task_type=TaskType.CAUSAL_LM,
 r=8,
 lora_alpha=16,
 lora_dropout=0.05,
 bias="none",
)
model = get_peft_model(model, peft_config)

# Fine-tune on your data
#... training code...

# Merge LoRA weights
model = model.merge_and_unload()

# Re-quantize!
# Note
model.quantize(new_calibration_data)
model.save_quantized("./llama-2-7b-finetuned-gptq")

Example 3: Quantization for Mobile¶

Scenario:
- Deploy LLM on mobile (iPhone, Android)
- Extreme size constraints (< 4GB)
- Need very fast inference

Solution: Ultra-aggressive GPTQ + group size optimization

Config:
- INT3 or INT2 (requires custom implementation)
- Or INT4 with large group_size (1024)
- Add layer pruning for additional compression
- Quantize activations too (advanced)

Alternative: Use smaller model + aggressive quant
- Llama 2 7B INT4: 3.5GB (too big)
- Phi 3B INT4: 1.5GB (fits!)
- Mistral 7B INT4: 3.5GB (alternative)

Code concept:
```python
# Ultra-aggressive quantization
config = BaseQuantizeConfig(
 bits=4,
 group_size=1024, # Larger groups (faster)
 true_sequential=False,
 desc_act=True, # Use activation order
)

# Or use smaller model
model = AutoGPTQForCausalLM.from_pretrained(
 "microsoft/phi-3-mini", # Only 3.8B parameters
 quantize_config=config
)

-

Tools and Ecosystems¶

AutoGPTQ Library¶

# Main library for GPTQ quantization
from auto_gptq import (
 AutoGPTQForCausalLM,
 BaseQuantizeConfig,
)

# Features:
 - # ├─ Easy quantization API
 - # ├─ Support for many models
 - # ├─ Multi-GPU quantization
 - # └─ Efficient inference kernels

Hugging Face Hub Integration¶

Many pre-quantized models available!

Examples:
 - TheBloke/Llama-2-7B-Chat-GPTQ
 - TheBloke/Mistral-7B-v0.1-GPTQ
 - TheBloke/Llama-2-70B-Chat-GPTQ
 -... 1000+ quantized models

Search at: huggingface.co/search/full-text?q=GPTQ

Usage:
```python
from auto_gptq import AutoGPTQForCausalLM

# Download pre-quantized model!
model = AutoGPTQForCausalLM.from_quantized(
 "TheBloke/Llama-2-7B-Chat-GPTQ",
 device="cuda:0"
)

Integration with Inference Engines¶

vLLM:
 - Auto-detects GPTQ quantization
 - Uses optimized kernels
 - 2-3x faster than naive inference

```python
from vllm import LLM

llm = LLM("TheBloke/Llama-2-7B-Chat-GPTQ")
# Automatic!

Ollama:

Can load GPTQ models
Simple command-line interface

Text Generation WebUI (oobabooga):

Supports GPTQ loading
GUI for easy interaction

LM Studio:

Desktop app supporting GPTQ
One-click quantized model download

-

## Challenges and Limitations

### Challenge 1: Accuracy Loss with INT4

Problem: Compressing 32 bits → 4 bits is aggressive!

Solution progression:

Naive INT4: ~70% accuracy (bad)
Quantization Aware Training: ~90% (better, slow)
GPTQ INT4: ~95-98% (good!)
GPTQ INT4 + calibration: ~98%+ (excellent)

Trade-off curve: Accuracy % │

100│ ●─── Full precision (baseline)
95 │ ●─── GPTQ INT4
╱ (calibration matters)
90 │ ●
╱ Other methods (slower to train)
85 │ ●
╱
80 │● │ └───────────────────── Time to quantize (GPTQ: fast!)

### Challenge 2: Model Architecture Support

GPTQ works best on: Transformer-based LLMs Models with linear layers Standard attention

May have issues with: Mixture of Experts (needs research) Custom attention patterns Very unusual architectures

Solution: Use widely-used model families

Llama
Mistral
Falcon
Qwen
And many others

### Challenge 3: Quantization Time

Time to quantize:

7B model on A100: 20-30 min
13B model on A100: 40-60 min
70B model on A100: 60-120 min
Multiple A100s: Can parallelize somewhat

Trade: One-time cost for permanent speedup

Quantize once: 2 hours
Use forever: 2-4x speedup
ROI: Positive within days/weeks

Optimization:

Use true_sequential=False for speed
Parallelize across GPUs
Use smaller calibration set

-

## GPTQ vs Other Quantization Methods

Comparison:

Method Bits Speed Acc% Training Size Practical ───────────────────────────────────────────────────────────────── Full Precision 32 1x 100% N/A 14GB Float16 16 1.5x 99% N/A 7GB INT8 (Post-train) 8 2x 98% No 7GB INT8 (QAT) 8 2x 99% Yes 7GB INT4 (Naive) 4 4x 70% No 3.5GB INT4 (QAT) 4 4x 90% Yes 3.5GB (slow) GPTQ INT4 4 3x 96% No 3.5GB Iterative Quant 4 3x 94% No 3.5GB AWQ 4 3x 97% No 3.5GB

Best practical option: GPTQ INT4

High accuracy (96%+)
Fast quantization (no training needed)
8x compression
3-4x speedup
Widely adopted (lots of pre-quantized models)

---

## Best Practices

### Do's

1. **Use GPTQ for production LLM deployment**
2. **Start with INT4** (most practical)
3. **Use group_size=128** for best accuracy
4. **Calibrate on representative data** (different from training!)
5. **Verify accuracy on benchmarks** before deployment
6. **Use pre-quantized models** if available (save time)
7. **Combine with KV cache quantization** for max speedup
8. **Profile performance** on target hardware

### Don'ts

1. Use INT4 without proper calibration
2. Quantize without verifying accuracy
3. Use group_size=1 (will fail)
4. Expect zero accuracy loss
5. Quantize for very small models (<3B may not help)
6. Assume quantization will work for custom architectures
7. Ignore numerical stability (damping helps!)
8. Quantize training data (use different calibration data!)

-

## Performance Estimation

### Will GPTQ Help Your Use Case?

Decision tree:

Is model > 7B?

Yes: GPTQ helps (almost always)
No: Maybe (depends on constraints)

Do you have GPU memory constraints?

Yes: GPTQ definitely helps
No: Still useful for speed

Are you doing inference (not training)?

Yes: Perfect for GPTQ
No: Use LoRA for training

Expected speedup:

2-3x for INT4 (compute still bottleneck)
1.5-2x for INT8 (memory bandwidth matters)
Plus: Fit in smaller GPU/CPU

Expected accuracy loss:

INT8: <1% (usually acceptable)
INT4: 2-4% (usually acceptable)
Varies by task (LLM output quality, MMLU, etc.)

```

Key Takeaways¶

GPTQ reduces model size by 4-8x while maintaining accuracy 2-4x inference speedup (compute+memory bound together) Makes large models fit on single consumer GPUs Best practical quantization method for LLMs INT4: 96%+ accuracy (2-3% loss acceptable) One-time quantization cost, permanent speedup benefit

-