LLM Inference Optimization¶

Overview¶

LLM Inference Optimization is the art and science of making Large Language Models run fast, efficiently, and cost-effectively in production. This guide synthesizes all major optimization techniques into an integrated framework.

Goal: Maximize throughput and minimize latency/cost
Key Insight: Combine multiple techniques for exponential gains
Target: Enable practical, profitable LLM deployment
Expected Outcome: 10-40x improvement over baseline

-

The Complete Inference Pipeline¶

Standard Inference Flow (Unoptimized)¶

Input Tokens
 ↓
[1] Tokenization (CPU)
 ↓
[2] Load Model Weights (GPU memory)
 ↓
[3] Prefill Phase
 - Process entire prompt at once
 - Compute attention for all tokens
 - Return first output token
 - Time: 100-500ms
 ↓
[4] Decoding Phase (per token)
 - Compute attention for new token
 - Process through feedforward
 - Generate next token
 - Time: 50-200ms per token
 ↓
[5] Repeat [4] until done
 ↓
Output Tokens

Bottlenecks identified:
 - [2] Memory bandwidth (loading 14GB model)
 - [3] Attention is O(N²) on sequence length
 - [4] Decoding repeats computation
 - Overall throughput: 1-5 requests/second

Optimized Inference Flow¶

Input Tokens
 ↓
[1] Tokenization (CPU)
 ↓
[2] Load Model Weights (GPU) + Quantization (INT4)
 - Size: 14GB → 3.5GB (4x smaller)
 - Load time: 20s → 5s
 - Memory: 14GB → 8GB
 ↓
[3] Prefill Phase (Flash Attention + Paged KV Cache)
 - Process prompt with Flash Attention (2.8x faster)
 - Store KV cache in pages (50% memory)
 - Compute efficiently with I/O awareness
 - Time: 100ms (vs 300ms standard)
 ↓
[4] Continuous Batch Management
 - Remove completed requests
 - Add waiting requests
 - Maintain GPU at 90% utilization
 - Decoding time: 50ms (same, but higher throughput)
 ↓
[5] Repeat [4] with continuous batching
 - Process 32 concurrent requests efficiently
 ↓
Output Tokens

Improvements:
 - [2] 4x memory reduction (fit on single GPU)
 - [3] 2.8x faster attention computation
 - [4] Incremental output (reuse cached KV)
 - [5] Continuous batching (100x throughput)
 - Overall throughput: 100-300 requests/second!

-

Optimization Stack: Layer by Layer¶

Layer 1: Model Compression¶

Quantization (INT4 with GPTQ/AWQ)

Problem: Model too large to load efficiently
Solution: Reduce precision of weights

Impact:
 - Model size: 14GB → 3.5GB (4x reduction)
 - Loading time: 20s → 5s
 - GPU memory: 14GB → 8GB
 - Inference speed: 2-3x faster (GPU compute, memory bandwidth)
 - Accuracy: 98-99% (1-2% loss acceptable)

When to apply:
Always (it's almost free performance!)
INT4 for size critical
INT8 for accuracy critical

Cost: Quantization one-time (5-30 min), permanent speedup

Layer 2: KV Cache Management¶

Caching + Paging (KV Cache + PagedAttention)

Problem: Recompute K, V for every token (expensive)
 KV cache memory fragments (wasteful)

Solution 1 - KV Cache:
 - Store computed K, V matrices
 - Reuse for next token generation
 - Speedup: 10x (avoid recomputation)
 - Memory: O(N) where N is sequence length

Solution 2 - PagedAttention:
 - Divide KV cache into pages (512 tokens each)
 - Allocate pages on-demand
 - Support non-contiguous allocation
 - Memory efficiency: 50% (vs contiguous)
 - Enables 2-4x more concurrent requests

Combined impact:
 - Speedup: 10x (cache) × 1.5x (paging) = 15x total
 - Memory: 50% reduction in peak usage
 - Concurrency: Serve 8-10 concurrent requests

Layer 3: Attention Computation¶

Flash Attention

Problem: Standard attention needs O(N²) intermediate storage
 Attention is I/O-bound, not compute-bound

Solution: Block-wise computation with SRAM caching

Impact:
 - I/O complexity: O(N²) → O(N)
 - Speedup: 2.8x (Flash Attention v2)
 - Memory: 50-60% reduction
 - Works best on: Longer sequences (>512 tokens)

Why it helps:
 - Moves intermediate results to fast cache (SRAM)
 - Reduces memory bandwidth bottleneck
 - Better GPU utilization (80-90% vs 40%)

Layer 4: Request Scheduling¶

Continuous Batching

Problem: Fixed batch size wastes GPU time on slow requests
 Can't add new requests until batch complete

Solution: Dynamic batching with request-level scheduling

Impact:
 - Throughput: 2-4x improvement
 - Latency: More predictable
 - GPU utilization: 80-90% (vs 50-60%)
 - Concurrency: Adaptive based on request length

How it works:
Time 0: [Req1, Req2, Req3, Req4] (all running)
Time 1: [Req1, Req3, Req4, Req5] (Req2 done, Req5 added)
Time 2: [Req1, Req4, Req5, Req6] (Req3 done, Req6 added)
...

Result: Always full batch, no idle GPU slots!

-

Combined Impact: Real Numbers¶

Baseline vs Fully Optimized¶

Scenario: Serve 100 concurrent inference requests
Model: Llama 2 7B
Hardware: Single A100 GPU
Average request: 512-token input, 256-token output

BASELINE (No Optimization):
 - Model size: 14GB (barely fits)
 - KV cache per request: 1GB
 - Max concurrent: 1-2 requests (28GB needed)
 - Throughput: 2-3 requests/second
 - Latency (p99): 30-40 seconds
 - Cost: $75/hour GPU × 33 hours = $2,475
 - Total time: 33 hours for 100 requests

Fully Optimized:
 - Model size: 3.5GB (INT4 quantization)
 - KV cache per request: 250MB (PagedAttention)
 - Max concurrent: 32+ requests
 - Throughput: 100-150 requests/second (50-75x!)
 - Latency (p99): 1-2 seconds (20x better!)
 - Cost: $75/hour × 0.7 hours = $52.50
 - Total time: 0.7 hours for 100 requests (47x faster!)

Breakdown of gains:
 - INT4 quantization: 2-3x speedup
 - KV Cache: 10x speedup (avoid recomputation)
 - PagedAttention: 1.5x speedup (memory efficiency)
 - Flash Attention: 2.8x speedup
 - Continuous Batching: 4x throughput
 - Combined (multiplicative): 2 × 10 × 1.5 × 2.8 × 4 ≈ 336x theoretical
 - Practical: 50-75x observed (due to overheads)

Cost savings:
 - Baseline: $2,475 for 100 requests ($24.75 per request)
 - Optimized: $52.50 for 100 requests ($0.52 per request)
 - Savings: 97.8% cost reduction!

-

Optimization Strategy: Which Techniques When?¶

Progressive Optimization Levels¶

Level 0: Baseline (No Optimization)
 - Standard attention
 - No caching
 - Single request at a time
 - Result: 2-3 req/sec, $10 per 1M tokens

Level 1: Basic Optimization (Quick Wins)
 - Enable KV cache
 - Use vLLM (automatic)
 - Result: 5-10 req/sec, $5 per 1M tokens

Level 2: Hardware Optimization
 - Add Flash Attention
 - Add continuous batching
 - Use better GPU (A100 vs V100)
 - Result: 20-40 req/sec, $2 per 1M tokens

Level 3: Full Optimization (Best Performance)
 - INT4 Quantization
 - PagedAttention + KV Cache
 - Flash Attention v2
 - Continuous batching
 - Kernel fusion
 - Result: 100-300 req/sec, $0.10-0.50 per 1M tokens

Level 4: Advanced (Research)
 - Speculative decoding
 - Pruning
 - Distillation
 - Custom kernels
 - Result: 300-500 req/sec, $0.05-0.10 per 1M tokens

Decision Matrix: Which Optimization?¶

Constraint → Recommended Optimizations
─────────────────────────────────────────────────────────────
Memory limited (consumer GPU) → INT4 + KV Cache + Flash Attn
Latency critical (streaming) → Flash Attn + Continuous Batch
Throughput critical (batch) → Continuous Batch + KV Cache
Cost sensitive → INT4 + PagedAttention
Already using vLLM → Just works! (auto-optimized)
Research/accuracy important → INT8 + LoRA fine-tuning
Small model (<7B) → Flash Attn + KV Cache enough
Large model (>70B) → INT4 + Continuous Batch + Quantization

-

Implementation Strategies¶

Strategy 1: vLLM (Recommended for Most)¶

from vllm import LLM, SamplingParams

# Everything is automatic!
llm = LLM(
 model="meta-llama/Llama-2-7b-hf",
 quantization="gptq", # Optional: add quantization
 gpu_memory_utilization=0.95,
 max_model_len=4096
)

# vLLM automatically handles:
# KV Cache management
# Continuous batching
# Flash Attention (if available)
# Kernel optimizations
# Request scheduling

sampling_params = SamplingParams(
 temperature=0.7,
 max_tokens=256,
 top_p=0.95
)

# Serve massive throughput
prompts = [f"Prompt {i}" for i in range(1000)]
outputs = llm.generate(prompts, sampling_params)

# Result

Strategy 2: Manual Optimization (Fine Control)¶

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from flash_attn import flash_attn_func

# Load quantized model
model = AutoModelForCausalLM.from_pretrained(
 "TheBloke/Llama-2-7B-Chat-GPTQ",
 device_map="auto"
)

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")

# Custom inference with optimizations
class OptimizedInference:
 def __init__(self, model, tokenizer):
 self.model = model
 self.tokenizer = tokenizer
 self.kv_cache = {} # Manual KV cache

 def generate(self, prompt, max_tokens=256):
 # Prefill phase
 inputs = self.tokenizer(prompt, return_tensors="pt")

 # Forward pass with KV cache
 with torch.no_grad():
 outputs = self.model(
 **inputs,
 use_cache=True, # Enable KV caching
 output_attentions=False
)

 # Decoding phase (per token)
 generated = []
 for _ in range(max_tokens):
 next_token = outputs.logits[:, -1,:].argmax(dim=-1)
 generated.append(next_token.item())

 # Reuse KV cache
 outputs = self.model(
 input_ids=next_token.unsqueeze(-1),
 use_cache=True,
 past_key_values=outputs.past_key_values
)

 return self.tokenizer.decode(generated)

inf = OptimizedInference(model, tokenizer)
result = inf.generate("Explain quantum computing")
print(result)

Strategy 3: Hybrid (LangChain + vLLM)¶

from langchain.llms.vllm import VLLMOpenAI
from langchain.chains import LLMChain
from langchain.prompts import PromptTemplate

# Use vLLM backend through LangChain
llm = VLLMOpenAI(
 openai_api_key="empty",
 openai_api_base="http://localhost:8000/v1",
 model_name="meta-llama/Llama-2-7b-hf",
 temperature=0.7,
 max_tokens=256
)

# Use LangChain for application logic
prompt = PromptTemplate(
 input_variables=["topic"],
 template="Explain {topic} in detail"
)

chain = LLMChain(llm=llm, prompt=prompt)

# Benefits:
# vLLM optimizations
# LangChain abstraction
# Easy to swap models
# Production-ready patterns

result = chain.run(topic="Machine Learning")

-

Real-World Deployments¶

Deployment 1: High-Throughput Chat Server¶

Requirements:
 - Model: Llama 2 7B
 - Users: 1000 concurrent
 - Queries/day: 100K
 - Budget: $5K/month
 - Latency target: p99 < 5 seconds

Architecture:
- ┌─────────────────────────────────────┐
 - Load Balancer (nginx) │
 - ┤
 - vLLM Inference Servers │
 - Server 1: A100 (100 req/sec) │
 - Server 2: A100 (100 req/sec) │
 - Server 3: A100 (100 req/sec) │
 - ┤
 - Cache (Redis) │
 - ┤
 - Database (PostgreSQL) │
 - ┘

Configuration:
 - Model: Llama 2 7B INT4 (3.5GB)
 - Batch size: 32 (continuous batching)
 - Flash Attention: Enabled
 - KV Cache: PagedAttention enabled
 - Quantization: GPTQ INT4

Performance:
 - Single server: 100 req/sec
 - 3 servers: 300 req/sec
 - Queue depth: 10K requests
 - Average latency: 1-2 seconds
 - P99 latency: 4-5 seconds 

Cost:
 - 3x A100 @ $2.50/hour: $1.80/hour
 - Networking/storage: $0.20/hour
 - Total: $2/hour
 - Per request: $2 / (300 × 3600) = $0.00019
 - Per 1M tokens: $0.50-1.00

ROI:
 - Setup cost: $5K (one-time)
 - Monthly ops: $1,500
 - Revenue per user: $5-10
 - Break-even: 150-300 users

Deployment 2: Batch Processing (Document Analysis)¶

Requirements:
 - Task: Analyze 1M documents
 - Model: Llama 2 13B
 - Time budget: 24 hours
 - Cost budget: $500
 - Quality: Maximum accuracy

Optimizations:
 - Model: Llama 2 13B INT8 (7GB, vs 26GB)
 - Quantization: AWQ (better accuracy than GPTQ)
 - Batch size: 128 (throughput-optimized)
 - Flash Attention: Enabled
 - PagedAttention: Enabled
 - Hardware: 8x A100 40GB

Performance:
 - Single A100 INT8: 30 req/sec
 - 8x A100: 240 req/sec
 - 1M documents / 240 req/sec = 4167 seconds
 - = 1.16 hours (easily within 24-hour budget)
 - P50 latency: 10 seconds per 2K-token document
 - P99 latency: 15 seconds

Cost:
 - 8x A100 @ $3.06/hour (spot pricing): $24.48/hour
 - 1.16 hours: $28.40
 - Plus data transfer (1GB): ~$10
 - Total: ~$40 (within $500 budget!)

Vs Unoptimized:
 - Llama 13B FP16: 7 GPU-hours = $21 per hour × 8 = $168
 - Would need 40+ GPU-hours = $600+
 - Optimized saves: $560 (93% reduction!)

Deployment 3: Mobile/Edge (On-Device)¶

Requirements:
 - Device: iPhone 15 Pro (8GB memory)
 - Model: Llama 2 7B
 - Latency: < 100ms per token
 - Memory: < 4GB used

Optimizations:
 - Model size: INT4 quantization (3.5GB)
 - Further compression: INT3 (2.3GB, experimental)
 - No KV cache (one-shot generation)
 - Metal Performance Shaders (MLX)
 - Batch size: 1 (single inference)

Architecture:
 - Layer 1: Compress model (3.5GB → 2.3GB)
 - Layer 2: Use optimized kernels (MLX, CoreML)
 - Layer 3: Stream output (start generating ASAP)
 - Layer 4: Local processing (no cloud calls)

Performance:
 - Loading: 5-10 seconds (first run)
 - Generation: 50-100ms per token
 - Memory peak: 3.5GB
 - Fits on iPhone!

Use cases:
 - Offline chat (no internet needed)
 - Private queries (data stays on device)
 - Fast response (no network latency)

-

Performance Metrics and Monitoring¶

Key Metrics to Track¶

Throughput Metrics:
 - Requests per second (req/s)
 - Tokens per second (tok/s)
 - GPU utilization (%)
 - Memory bandwidth utilization

Latency Metrics:
 - Time to First Token (TTFT)
 - Inter-token Latency (ITL)
 - End-to-End Latency (E2E)
 - P50, P95, P99 latencies

Cost Metrics:
 - Cost per 1M tokens
 - Cost per request
 - Cost per hour (infra)
 - ROI on optimization investment

Quality Metrics:
 - Token accuracy (vs original model)
 - Semantic similarity (embedding-based)
 - Task-specific metrics (BLEU, ROUGE, etc.)

Example Dashboard:
```python
from prometheus_client import Counter, Histogram, Gauge

# Throughput
requests_total = Counter('requests_total', 'Total requests')
tokens_generated = Counter('tokens_generated', 'Total tokens')

# Latency
request_latency = Histogram('request_latency_seconds', 'Request latency')
ttft_latency = Histogram('ttft_seconds', 'Time to first token')

# Resources
gpu_memory = Gauge('gpu_memory_bytes', 'GPU memory usage')
gpu_utilization = Gauge('gpu_utilization_percent', 'GPU utilization')

# Track everything
def process_request(request):
 start = time.time()
 result = llm.generate(request)
 latency = time.time() - start

 requests_total.inc()
 tokens_generated.add(len(result.tokens))
 request_latency.observe(latency)
 gpu_utilization.set(get_gpu_util())

 return result

-

## Optimization Trade-Offs

### Accuracy vs Speed

Dimension: Quality of output

Full Precision (FP32):

Accuracy: 100% (baseline)
Speed: 1x
Memory: 100%
Use: Research, benchmarking

Mixed Precision (FP16):

Accuracy: 99.9% (nearly identical)
Speed: 1.5x
Memory: 50%
Use: Standard production

INT8 Quantization:

Accuracy: 99% (very close)
Speed: 2x
Memory: 25%
Use: Most production cases

INT4 Quantization:

Accuracy: 98% (slight loss)
Speed: 3x
Memory: 12.5%
Use: When size critical

INT3/INT2 (experimental):

Accuracy: 95% (noticeable loss)
Speed: 4-5x
Memory: <10%
Use: Research only

Recommendation:

Default: INT8 or FP16 (best balance)
Size constraint: INT4
Accuracy critical: INT8

### Throughput vs Latency

Scenario: Process 1000 requests

High Throughput Focus:

Batch size: 128
Continuous batching: Enabled
Per-request latency: 10-20 seconds (queue waiting)
Total time: 8-10 seconds (parallel)
GPU: Constantly busy
Cost: Efficient
Use: Batch processing, not interactive

High Latency Focus:

Batch size: 1
Continuous batching: Disabled
Per-request latency: 1-2 seconds
Total time: 1000-2000 seconds (serial)
GPU: Idle between requests
Cost: Wasteful
Use: Real-time chat, interactive

Balanced:

Batch size: 8-16
Continuous batching: Enabled
Per-request latency: 2-5 seconds
Total time: 50-100 seconds (parallel)
GPU: Well-utilized
Cost: Good balance
Use: Most production systems

-

## Benchmarking and Testing

### Benchmarking Code

```python
import time
import numpy as np
from vllm import LLM, SamplingParams

# Setup
llm = LLM(model="meta-llama/Llama-2-7b-hf")
sampling_params = SamplingParams(max_tokens=256, temperature=0.7)

# Prepare test data
prompts = [f"Prompt {i}" for i in range(100)]

# Benchmark 1
print("=== Benchmark 1: Single Request ===")
start = time.time()
output = llm.generate(prompts[0], sampling_params)
latency = time.time() - start
print(f"Latency: {latency*1000:.2f}ms")

# Benchmark 2
print("\n=== Benchmark 2: Batch (100 requests) ===")
start = time.time()
outputs = llm.generate(prompts, sampling_params)
total_time = time.time() - start
throughput = len(prompts) / total_time
print(f"Total time: {total_time:.2f}s")
print(f"Throughput: {throughput:.2f} req/s")

# Benchmark 3
print("\n=== Benchmark 3: Throughput Benchmark ===")
latencies = []
for i in range(10):
 start = time.time()
 output = llm.generate(prompts[i], sampling_params)
 latencies.append(time.time() - start)

print(f"Mean latency: {np.mean(latencies)*1000:.2f}ms")
print(f"P50 latency: {np.percentile(latencies, 50)*1000:.2f}ms")
print(f"P99 latency: {np.percentile(latencies, 99)*1000:.2f}ms")

# Benchmark 4
print("\n=== Benchmark 4: Memory Usage ===")
import torch
torch.cuda.reset_peak_memory_stats()

outputs = llm.generate(prompts, sampling_params)
peak_memory = torch.cuda.max_memory_allocated() / 1e9 # GB

print(f"Peak memory: {peak_memory:.2f} GB")
print(f"Efficiency: {len(prompts) / peak_memory:.0f} requests per GB")

-

Deployment Checklist¶

Pre-Deployment¶

□ Model Selection
 □ Choose model size (7B, 13B, 70B, etc.)
 □ Choose architecture (Llama, Mistral, Qwen, etc.)
 □ Verify accuracy on task

□ Quantization Decision
 □ Test INT4 vs INT8 vs FP16
 □ Measure accuracy loss
 □ Benchmark on target hardware
 □ Choose based on latency/accuracy trade-off

□ Hardware Planning
 □ Calculate memory requirements
 □ Choose GPU (A100, H100, RTX 4090, etc.)
 □ Plan for 2-3x headroom
 □ Test on representative workload

□ Optimization Selection
 □ Enable KV cache (always)
 □ Enable Flash Attention (if available)
 □ Enable continuous batching (for throughput)
 □ Test different batch sizes

□ Benchmarking
 □ Measure latency (TTFT, ITL)
 □ Measure throughput
 □ Measure memory usage
 □ Measure accuracy
 □ Document baseline

Deployment¶

□ Infrastructure
 □ Set up GPU cluster
 □ Configure networking
 □ Set up monitoring
 □ Set up logging

□ Serving Framework
 □ Install vLLM / TGI / etc.
 □ Configure model loading
 □ Set up API endpoints
 □ Configure autoscaling

□ Testing
 □ Unit tests (single request)
 □ Load tests (concurrent requests)
 □ Stress tests (beyond capacity)
 □ Accuracy tests (random samples)

□ Monitoring
 □ Set up metrics collection
 □ Set up dashboards
 □ Set up alerts
 □ Document runbooks

Post-Deployment¶

□ Ongoing Monitoring
 □ Track latency trends
 □ Track throughput trends
 □ Track error rates
 □ Track cost per request

□ Optimization
 □ A/B test new models
 □ A/B test different batch sizes
 □ A/B test different quantization
 □ Monitor for regressions

□ Maintenance
 □ Regular backups
 □ Update models
 □ Security updates
 □ Performance reviews (weekly/monthly)

Key Takeaways¶

Combine techniques for exponential gains (10-40x) Quantization + KV Cache: Most impactful Flash Attention: Essential for long sequences Continuous Batching: Maximize throughput vLLM: Automatic optimization (use it!) 97% cost reduction with full optimization

Optimization Decision Tree¶

Start: Want to serve LLM in production?
 ↓
 - Memory limited? → YES
 - Use INT4 quantization
 - Use PagedAttention
 - Use Flash Attention
 - Use continuous batching
 - Deploy with vLLM
│
 - Latency critical? → YES
 - Use Flash Attention v2
 - Reduce batch size (1-8)
 - Use KV cache
 - Deploy with FastAPI + vLLM
│
 - Throughput critical? → YES
 - Use continuous batching
 - Use large batch size (32-128)
 - Use KV cache + PagedAttention
 - Deploy with vLLM
│
 - Cost sensitive? → YES
 - Use INT4 quantization
 - Use all optimizations
 - Deploy on spot GPUs
 - Use vLLM + autoscaling

Result: Full optimization stack with vLLM!

-

Conclusion¶

LLM inference optimization is a multi-dimensional problem requiring careful consideration of:

Model: Size, architecture, precision
Hardware: GPU type, memory, bandwidth
Algorithms: Attention, caching, quantization
Systems: Batching, scheduling, serving
Metrics: Latency, throughput, cost, accuracy

By combining all techniques presented in this guide, you can achieve:

50-100x throughput improvement
95% cost reduction
2-3x latency reduction
97% accuracy retention

The key is understanding trade-offs and optimizing for your specific use case. Use vLLM as the baseline (it does most optimizations automatically), then add specialized techniques as needed.

Happy optimizing!