Inference & Deployment (2017-2024)

Techniques for efficient inference, quantization, and serving LLMs.

Quantization

• Quantization and Training of Neural Networks - QAT foundations
• GPTQ: Accurate Post-Training Quantization - Weight-only quantization
• AWQ: Activation-Aware Weight Quantization - Improved quantization

Decoding Strategies

• The Curious Case of Neural Text Degeneration - Sampling vs greedy analysis
• Speculative Decoding for Faster Large Language Model Inference - Efficient generation
• Medusa: Parallel Decoding with Multiple Heads - Multi-token prediction

Serving & Optimization

• vLLM: Easy, Fast, and Cheap LLM Serving - PagedAttention memory optimization

---

Key Insight: Inference optimizations are critical for deploying LLMs in production with acceptable latency and cost.