Skip to content

LLM Inference Optimization: Complete Guide to Production Serving

Overview

LLM Inference Optimization is the art and science of making Large Language Models run fast, efficiently, and cost-effectively in production. This guide synthesizes all major optimization techniques into an integrated framework.

  • Goal: Maximize throughput and minimize latency/cost
  • Key Insight: Combine multiple techniques for exponential gains
  • Target: Enable practical, profitable LLM deployment
  • Expected Outcome: 10-40x improvement over baseline

The Complete Inference Pipeline

Standard Inference Flow (Unoptimized)

Input Tokens
    ↓
[1] Tokenization (CPU)
    ↓
[2] Load Model Weights (GPU memory)
    ↓
[3] Prefill Phase
  - Process entire prompt at once
  - Compute attention for all tokens
  - Return first output token
  - Time: 100-500ms
    ↓
[4] Decoding Phase (per token)
  - Compute attention for new token
  - Process through feedforward
  - Generate next token
  - Time: 50-200ms per token
    ↓
[5] Repeat [4] until done
    ↓
Output Tokens

Bottlenecks identified:
  - [2] Memory bandwidth (loading 14GB model)
  - [3] Attention is O(N²) on sequence length
  - [4] Decoding repeats computation
  - Overall throughput: 1-5 requests/second

Optimized Inference Flow

Input Tokens
    ↓
[1] Tokenization (CPU)
    ↓
[2] Load Model Weights (GPU) + Quantization (INT4)
  - Size: 14GB → 3.5GB (4x smaller)
  - Load time: 20s → 5s
  - Memory: 14GB → 8GB
    ↓
[3] Prefill Phase (Flash Attention + Paged KV Cache)
  - Process prompt with Flash Attention (2.8x faster)
  - Store KV cache in pages (50% memory)
  - Compute efficiently with I/O awareness
  - Time: 100ms (vs 300ms standard)
    ↓
[4] Continuous Batch Management
  - Remove completed requests
  - Add waiting requests
  - Maintain GPU at 90% utilization
  - Decoding time: 50ms (same, but higher throughput)
    ↓
[5] Repeat [4] with continuous batching
  - Process 32 concurrent requests efficiently
    ↓
Output Tokens

Improvements:
  - [2] 4x memory reduction (fit on single GPU)
  - [3] 2.8x faster attention computation
  - [4] Incremental output (reuse cached KV)
  - [5] Continuous batching (100x throughput)
  - Overall throughput: 100-300 requests/second!

Optimization Stack: Layer by Layer

Layer 1: Model Compression

Quantization (INT4 with GPTQ/AWQ)

Problem: Model too large to load efficiently
Solution: Reduce precision of weights

Impact:
  - Model size: 14GB → 3.5GB (4x reduction)
  - Loading time: 20s → 5s
  - GPU memory: 14GB → 8GB
  - Inference speed: 2-3x faster (GPU compute, memory bandwidth)
  - Accuracy: 98-99% (1-2% loss acceptable)

When to apply:
✓ Always (it's almost free performance!)
✓ INT4 for size critical
✓ INT8 for accuracy critical

Cost: Quantization one-time (5-30 min), permanent speedup

Layer 2: KV Cache Management

Caching + Paging (KV Cache + PagedAttention)

Problem: Recompute K, V for every token (expensive)
         KV cache memory fragments (wasteful)

Solution 1 - KV Cache:
  - Store computed K, V matrices
  - Reuse for next token generation
  - Speedup: 10x (avoid recomputation)
  - Memory: O(N) where N is sequence length

Solution 2 - PagedAttention:
  - Divide KV cache into pages (512 tokens each)
  - Allocate pages on-demand
  - Support non-contiguous allocation
  - Memory efficiency: 50% (vs contiguous)
  - Enables 2-4x more concurrent requests

Combined impact:
  - Speedup: 10x (cache) × 1.5x (paging) = 15x total
  - Memory: 50% reduction in peak usage
  - Concurrency: Serve 8-10 concurrent requests

Layer 3: Attention Computation

Flash Attention

Problem: Standard attention needs O(N²) intermediate storage
         Attention is I/O-bound, not compute-bound

Solution: Block-wise computation with SRAM caching

Impact:
  - I/O complexity: O(N²) → O(N)
  - Speedup: 2.8x (Flash Attention v2)
  - Memory: 50-60% reduction
  - Works best on: Longer sequences (>512 tokens)

Why it helps:
  - Moves intermediate results to fast cache (SRAM)
  - Reduces memory bandwidth bottleneck
  - Better GPU utilization (80-90% vs 40%)

Layer 4: Request Scheduling

Continuous Batching

Problem: Fixed batch size wastes GPU time on slow requests
         Can't add new requests until batch complete

Solution: Dynamic batching with request-level scheduling

Impact:
  - Throughput: 2-4x improvement
  - Latency: More predictable
  - GPU utilization: 80-90% (vs 50-60%)
  - Concurrency: Adaptive based on request length

How it works:
Time 0: [Req1, Req2, Req3, Req4] (all running)
Time 1: [Req1, Req3, Req4, Req5] (Req2 done, Req5 added)
Time 2: [Req1, Req4, Req5, Req6] (Req3 done, Req6 added)
...

Result: Always full batch, no idle GPU slots!

Combined Impact: Real Numbers

Baseline vs Fully Optimized

Scenario: Serve 100 concurrent inference requests
Model: Llama 2 7B
Hardware: Single A100 GPU
Average request: 512-token input, 256-token output

BASELINE (No Optimization):
  - Model size: 14GB (barely fits)
  - KV cache per request: 1GB
  - Max concurrent: 1-2 requests (28GB needed)
  - Throughput: 2-3 requests/second
  - Latency (p99): 30-40 seconds
  - Cost: $75/hour GPU × 33 hours = $2,475
  - Total time: 33 hours for 100 requests

Fully Optimized:
  - Model size: 3.5GB (INT4 quantization)
  - KV cache per request: 250MB (PagedAttention)
  - Max concurrent: 32+ requests
  - Throughput: 100-150 requests/second (50-75x!)
  - Latency (p99): 1-2 seconds (20x better!)
  - Cost: $75/hour × 0.7 hours = $52.50
  - Total time: 0.7 hours for 100 requests (47x faster!)

Breakdown of gains:
  - INT4 quantization: 2-3x speedup
  - KV Cache: 10x speedup (avoid recomputation)
  - PagedAttention: 1.5x speedup (memory efficiency)
  - Flash Attention: 2.8x speedup
  - Continuous Batching: 4x throughput
  - Combined (multiplicative): 2 × 10 × 1.5 × 2.8 × 4 ≈ 336x theoretical
  - Practical: 50-75x observed (due to overheads)

Cost savings:
  - Baseline: $2,475 for 100 requests ($24.75 per request)
  - Optimized: $52.50 for 100 requests ($0.52 per request)
  - Savings: 97.8% cost reduction!

Optimization Strategy: Which Techniques When?

Progressive Optimization Levels

Level 0: Baseline (No Optimization)
  - Standard attention
  - No caching
  - Single request at a time
  - Result: 2-3 req/sec, $10 per 1M tokens

Level 1: Basic Optimization (Quick Wins)
  - Enable KV cache
  - Use vLLM (automatic)
  - Result: 5-10 req/sec, $5 per 1M tokens

Level 2: Hardware Optimization
  - Add Flash Attention
  - Add continuous batching
  - Use better GPU (A100 vs V100)
  - Result: 20-40 req/sec, $2 per 1M tokens

Level 3: Full Optimization (Best Performance)
  - INT4 Quantization
  - PagedAttention + KV Cache
  - Flash Attention v2
  - Continuous batching
  - Kernel fusion
  - Result: 100-300 req/sec, $0.10-0.50 per 1M tokens

Level 4: Advanced (Research)
  - Speculative decoding
  - Pruning
  - Distillation
  - Custom kernels
  - Result: 300-500 req/sec, $0.05-0.10 per 1M tokens

Decision Matrix: Which Optimization?

Constraint                      → Recommended Optimizations
─────────────────────────────────────────────────────────────
Memory limited (consumer GPU)   → INT4 + KV Cache + Flash Attn
Latency critical (streaming)    → Flash Attn + Continuous Batch
Throughput critical (batch)     → Continuous Batch + KV Cache
Cost sensitive                  → INT4 + PagedAttention
Already using vLLM              → Just works! (auto-optimized)
Research/accuracy important     → INT8 + LoRA fine-tuning
Small model (<7B)               → Flash Attn + KV Cache enough
Large model (>70B)              → INT4 + Continuous Batch + Quantization

Implementation Strategies

from vllm import LLM, SamplingParams

# Everything is automatic!
llm = LLM(
    model="meta-llama/Llama-2-7b-hf",
    quantization="gptq",  # Optional: add quantization
    gpu_memory_utilization=0.95,
    max_model_len=4096
)

# vLLM automatically handles:
# ✓ KV Cache management
# ✓ Continuous batching
# ✓ Flash Attention (if available)
# ✓ Kernel optimizations
# ✓ Request scheduling

sampling_params = SamplingParams(
    temperature=0.7,
    max_tokens=256,
    top_p=0.95
)

# Serve massive throughput
prompts = [f"Prompt {i}" for i in range(1000)]
outputs = llm.generate(prompts, sampling_params)

# Result: 100-300 req/sec on single A100!

Strategy 2: Manual Optimization (Fine Control)

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from flash_attn import flash_attn_func

# Load quantized model
model = AutoModelForCausalLM.from_pretrained(
    "TheBloke/Llama-2-7B-Chat-GPTQ",
    device_map="auto"
)

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")

# Custom inference with optimizations
class OptimizedInference:
    def __init__(self, model, tokenizer):
        self.model = model
        self.tokenizer = tokenizer
        self.kv_cache = {}  # Manual KV cache

    def generate(self, prompt, max_tokens=256):
        # Prefill phase
        inputs = self.tokenizer(prompt, return_tensors="pt")

        # Forward pass with KV cache
        with torch.no_grad():
            outputs = self.model(
                **inputs,
                use_cache=True,  # Enable KV caching
                output_attentions=False
            )

        # Decoding phase (per token)
        generated = []
        for _ in range(max_tokens):
            next_token = outputs.logits[:, -1, :].argmax(dim=-1)
            generated.append(next_token.item())

            # Reuse KV cache
            outputs = self.model(
                input_ids=next_token.unsqueeze(-1),
                use_cache=True,
                past_key_values=outputs.past_key_values
            )

        return self.tokenizer.decode(generated)

inf = OptimizedInference(model, tokenizer)
result = inf.generate("Explain quantum computing")
print(result)

Strategy 3: Hybrid (LangChain + vLLM)

from langchain.llms.vllm import VLLMOpenAI
from langchain.chains import LLMChain
from langchain.prompts import PromptTemplate

# Use vLLM backend through LangChain
llm = VLLMOpenAI(
    openai_api_key="empty",
    openai_api_base="http://localhost:8000/v1",
    model_name="meta-llama/Llama-2-7b-hf",
    temperature=0.7,
    max_tokens=256
)

# Use LangChain for application logic
prompt = PromptTemplate(
    input_variables=["topic"],
    template="Explain {topic} in detail"
)

chain = LLMChain(llm=llm, prompt=prompt)

# Benefits:
# ✓ vLLM optimizations
# ✓ LangChain abstraction
# ✓ Easy to swap models
# ✓ Production-ready patterns

result = chain.run(topic="Machine Learning")

Real-World Deployments

Deployment 1: High-Throughput Chat Server

Requirements:
  - Model: Llama 2 7B
  - Users: 1000 concurrent
  - Queries/day: 100K
  - Budget: $5K/month
  - Latency target: p99 < 5 seconds

Architecture:
- ┌─────────────────────────────────────┐
    - Load Balancer (nginx)               │
  - ┤
    - vLLM Inference Servers              │
      - Server 1: A100 (100 req/sec)    │
      - Server 2: A100 (100 req/sec)    │
      - Server 3: A100 (100 req/sec)    │
  - ┤
    - Cache (Redis)                       │
  - ┤
    - Database (PostgreSQL)               │
  - ┘

Configuration:
  - Model: Llama 2 7B INT4 (3.5GB)
  - Batch size: 32 (continuous batching)
  - Flash Attention: Enabled
  - KV Cache: PagedAttention enabled
  - Quantization: GPTQ INT4

Performance:
  - Single server: 100 req/sec
  - 3 servers: 300 req/sec
  - Queue depth: 10K requests
  - Average latency: 1-2 seconds
  - P99 latency: 4-5 seconds ✓

Cost:
  - 3x A100 @ $2.50/hour: $1.80/hour
  - Networking/storage: $0.20/hour
  - Total: $2/hour
  - Per request: $2 / (300 × 3600) = $0.00019
  - Per 1M tokens: $0.50-1.00

ROI:
  - Setup cost: $5K (one-time)
  - Monthly ops: $1,500
  - Revenue per user: $5-10
  - Break-even: 150-300 users

Deployment 2: Batch Processing (Document Analysis)

Requirements:
  - Task: Analyze 1M documents
  - Model: Llama 2 13B
  - Time budget: 24 hours
  - Cost budget: $500
  - Quality: Maximum accuracy

Optimizations:
  - Model: Llama 2 13B INT8 (7GB, vs 26GB)
  - Quantization: AWQ (better accuracy than GPTQ)
  - Batch size: 128 (throughput-optimized)
  - Flash Attention: Enabled
  - PagedAttention: Enabled
  - Hardware: 8x A100 40GB

Performance:
  - Single A100 INT8: 30 req/sec
  - 8x A100: 240 req/sec
  - 1M documents / 240 req/sec = 4167 seconds
  - = 1.16 hours (easily within 24-hour budget)
  - P50 latency: 10 seconds per 2K-token document
  - P99 latency: 15 seconds

Cost:
  - 8x A100 @ $3.06/hour (spot pricing): $24.48/hour
  - 1.16 hours: $28.40
  - Plus data transfer (1GB): ~$10
  - Total: ~$40 (within $500 budget!)

Vs Unoptimized:
  - Llama 13B FP16: 7 GPU-hours = $21 per hour × 8 = $168
  - Would need 40+ GPU-hours = $600+
  - Optimized saves: $560 (93% reduction!)

Deployment 3: Mobile/Edge (On-Device)

Requirements:
  - Device: iPhone 15 Pro (8GB memory)
  - Model: Llama 2 7B
  - Latency: < 100ms per token
  - Memory: < 4GB used

Optimizations:
  - Model size: INT4 quantization (3.5GB)
  - Further compression: INT3 (2.3GB, experimental)
  - No KV cache (one-shot generation)
  - Metal Performance Shaders (MLX)
  - Batch size: 1 (single inference)

Architecture:
  - Layer 1: Compress model (3.5GB → 2.3GB)
  - Layer 2: Use optimized kernels (MLX, CoreML)
  - Layer 3: Stream output (start generating ASAP)
  - Layer 4: Local processing (no cloud calls)

Performance:
  - Loading: 5-10 seconds (first run)
  - Generation: 50-100ms per token
  - Memory peak: 3.5GB
  - Fits on iPhone!

Use cases:
  - Offline chat (no internet needed)
  - Private queries (data stays on device)
  - Fast response (no network latency)

Performance Metrics and Monitoring

Key Metrics to Track

Throughput Metrics:
  - Requests per second (req/s)
  - Tokens per second (tok/s)
  - GPU utilization (%)
  - Memory bandwidth utilization

Latency Metrics:
  - Time to First Token (TTFT)
  - Inter-token Latency (ITL)
  - End-to-End Latency (E2E)
  - P50, P95, P99 latencies

Cost Metrics:
  - Cost per 1M tokens
  - Cost per request
  - Cost per hour (infra)
  - ROI on optimization investment

Quality Metrics:
  - Token accuracy (vs original model)
  - Semantic similarity (embedding-based)
  - Task-specific metrics (BLEU, ROUGE, etc.)

Example Dashboard:
```python
from prometheus_client import Counter, Histogram, Gauge

# Throughput
requests_total = Counter('requests_total', 'Total requests')
tokens_generated = Counter('tokens_generated', 'Total tokens')

# Latency
request_latency = Histogram('request_latency_seconds', 'Request latency')
ttft_latency = Histogram('ttft_seconds', 'Time to first token')

# Resources
gpu_memory = Gauge('gpu_memory_bytes', 'GPU memory usage')
gpu_utilization = Gauge('gpu_utilization_percent', 'GPU utilization')

# Track everything
def process_request(request):
    start = time.time()
    result = llm.generate(request)
    latency = time.time() - start

    requests_total.inc()
    tokens_generated.add(len(result.tokens))
    request_latency.observe(latency)
    gpu_utilization.set(get_gpu_util())

    return result

---

## Optimization Trade-Offs

### Accuracy vs Speed
Dimension: Quality of output

Full Precision (FP32): - Accuracy: 100% (baseline) - Speed: 1x - Memory: 100% - Use: Research, benchmarking

Mixed Precision (FP16): - Accuracy: 99.9% (nearly identical) - Speed: 1.5x - Memory: 50% - Use: Standard production

INT8 Quantization: - Accuracy: 99% (very close) - Speed: 2x - Memory: 25% - Use: Most production cases

INT4 Quantization: - Accuracy: 98% (slight loss) - Speed: 3x - Memory: 12.5% - Use: When size critical

INT3/INT2 (experimental): - Accuracy: 95% (noticeable loss) - Speed: 4-5x - Memory: <10% - Use: Research only

Recommendation: - Default: INT8 or FP16 (best balance) - Size constraint: INT4 - Accuracy critical: INT8 

### Throughput vs Latency
Scenario: Process 1000 requests

High Throughput Focus: - Batch size: 128 - Continuous batching: Enabled - Per-request latency: 10-20 seconds (queue waiting) - Total time: 8-10 seconds (parallel) - GPU: Constantly busy - Cost: Efficient - Use: Batch processing, not interactive

High Latency Focus: - Batch size: 1 - Continuous batching: Disabled - Per-request latency: 1-2 seconds - Total time: 1000-2000 seconds (serial) - GPU: Idle between requests - Cost: Wasteful - Use: Real-time chat, interactive

Balanced: - Batch size: 8-16 - Continuous batching: Enabled - Per-request latency: 2-5 seconds - Total time: 50-100 seconds (parallel) - GPU: Well-utilized - Cost: Good balance - Use: Most production systems 

---

## Benchmarking and Testing

### Benchmarking Code

```python
import time
import numpy as np
from vllm import LLM, SamplingParams

# Setup
llm = LLM(model="meta-llama/Llama-2-7b-hf")
sampling_params = SamplingParams(max_tokens=256, temperature=0.7)

# Prepare test data
prompts = [f"Prompt {i}" for i in range(100)]

# Benchmark 1: Single request
print("=== Benchmark 1: Single Request ===")
start = time.time()
output = llm.generate(prompts[0], sampling_params)
latency = time.time() - start
print(f"Latency: {latency*1000:.2f}ms")

# Benchmark 2: Batch requests
print("\n=== Benchmark 2: Batch (100 requests) ===")
start = time.time()
outputs = llm.generate(prompts, sampling_params)
total_time = time.time() - start
throughput = len(prompts) / total_time
print(f"Total time: {total_time:.2f}s")
print(f"Throughput: {throughput:.2f} req/s")

# Benchmark 3: Concurrent simulation
print("\n=== Benchmark 3: Throughput Benchmark ===")
latencies = []
for i in range(10):
    start = time.time()
    output = llm.generate(prompts[i], sampling_params)
    latencies.append(time.time() - start)

print(f"Mean latency: {np.mean(latencies)*1000:.2f}ms")
print(f"P50 latency: {np.percentile(latencies, 50)*1000:.2f}ms")
print(f"P99 latency: {np.percentile(latencies, 99)*1000:.2f}ms")

# Benchmark 4: Memory tracking
print("\n=== Benchmark 4: Memory Usage ===")
import torch
torch.cuda.reset_peak_memory_stats()

outputs = llm.generate(prompts, sampling_params)
peak_memory = torch.cuda.max_memory_allocated() / 1e9  # GB

print(f"Peak memory: {peak_memory:.2f} GB")
print(f"Efficiency: {len(prompts) / peak_memory:.0f} requests per GB")

Deployment Checklist

Pre-Deployment

□ Model Selection
  □ Choose model size (7B, 13B, 70B, etc.)
  □ Choose architecture (Llama, Mistral, Qwen, etc.)
  □ Verify accuracy on task

□ Quantization Decision
  □ Test INT4 vs INT8 vs FP16
  □ Measure accuracy loss
  □ Benchmark on target hardware
  □ Choose based on latency/accuracy trade-off

□ Hardware Planning
  □ Calculate memory requirements
  □ Choose GPU (A100, H100, RTX 4090, etc.)
  □ Plan for 2-3x headroom
  □ Test on representative workload

□ Optimization Selection
  □ Enable KV cache (always)
  □ Enable Flash Attention (if available)
  □ Enable continuous batching (for throughput)
  □ Test different batch sizes

□ Benchmarking
  □ Measure latency (TTFT, ITL)
  □ Measure throughput
  □ Measure memory usage
  □ Measure accuracy
  □ Document baseline

Deployment

□ Infrastructure
  □ Set up GPU cluster
  □ Configure networking
  □ Set up monitoring
  □ Set up logging

□ Serving Framework
  □ Install vLLM / TGI / etc.
  □ Configure model loading
  □ Set up API endpoints
  □ Configure autoscaling

□ Testing
  □ Unit tests (single request)
  □ Load tests (concurrent requests)
  □ Stress tests (beyond capacity)
  □ Accuracy tests (random samples)

□ Monitoring
  □ Set up metrics collection
  □ Set up dashboards
  □ Set up alerts
  □ Document runbooks

Post-Deployment

□ Ongoing Monitoring
  □ Track latency trends
  □ Track throughput trends
  □ Track error rates
  □ Track cost per request

□ Optimization
  □ A/B test new models
  □ A/B test different batch sizes
  □ A/B test different quantization
  □ Monitor for regressions

□ Maintenance
  □ Regular backups
  □ Update models
  □ Security updates
  □ Performance reviews (weekly/monthly)

Key Takeaways

🔑 Combine techniques for exponential gains (10-40x)
💾 Quantization + KV Cache: Most impactful
⚡ Flash Attention: Essential for long sequences
📊 Continuous Batching: Maximize throughput
🎯 vLLM: Automatic optimization (use it!)
💰 97% cost reduction with full optimization

Optimization Decision Tree

Start: Want to serve LLM in production?
    ↓
  - Memory limited? → YES
    - Use INT4 quantization
    - Use PagedAttention
    - Use Flash Attention
    - Use continuous batching
    - Deploy with vLLM
│
  - Latency critical? → YES
    - Use Flash Attention v2
    - Reduce batch size (1-8)
    - Use KV cache
    - Deploy with FastAPI + vLLM
│
  - Throughput critical? → YES
    - Use continuous batching
    - Use large batch size (32-128)
    - Use KV cache + PagedAttention
    - Deploy with vLLM
│
  - Cost sensitive? → YES
  - Use INT4 quantization
  - Use all optimizations
  - Deploy on spot GPUs
  - Use vLLM + autoscaling

Result: Full optimization stack with vLLM!

Further Reading

Conclusion

LLM inference optimization is a multi-dimensional problem requiring careful consideration of:

  1. Model: Size, architecture, precision
  2. Hardware: GPU type, memory, bandwidth
  3. Algorithms: Attention, caching, quantization
  4. Systems: Batching, scheduling, serving
  5. Metrics: Latency, throughput, cost, accuracy

By combining all techniques presented in this guide, you can achieve: - 50-100x throughput improvement - 95% cost reduction - 2-3x latency reduction - 97% accuracy retention

The key is understanding trade-offs and optimizing for your specific use case. Use vLLM as the baseline (it does most optimizations automatically), then add specialized techniques as needed.

Happy optimizing! 🚀