Skip to content

Inference Frameworks Fundamentals: Choosing the Right Tool

Overview

Inference Frameworks are software systems optimized for running LLM inference in production. Different frameworks offer different trade-offs in throughput, latency, ease-of-use, and features.

  • Purpose: Serve models with high throughput, low latency, and fault tolerance
  • Popular options: vLLM, TensorRT-LLM, DeepSpeed-MII, Ollama, MLflow
  • Trade-off: Performance vs. Simplicity vs. Features
  • Decision: Depends on use case, scale, and requirements

Why Inference Frameworks Matter

Problem: Native Inference is Slow

Baseline: Run LLM with PyTorch/Transformers

Code:
```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b")

input_ids = tokenizer("Hello world", return_tensors="pt").input_ids
output = model.generate(input_ids, max_length=50)
print(tokenizer.decode(output[0]))

Performance: - Throughput: 50-100 tokens/second - Latency (first token): 500-1000ms - Batch size: 1 (inefficient!) - Memory: High (not optimized) - Concurrent users: ~1 (very limited)

Problems: - No continuous batching - Inefficient KV cache management - No request scheduling - Memory thrashing - Not production-ready!

Result: Can't serve 100+ users on single GPU 

### Solution: Inference Framework
Using vLLM (inference framework)

Code: 

from vllm import LLM, SamplingParams

llm = LLM(model="meta-llama/Llama-2-7b", tensor_parallel_size=1)
sampling_params = SamplingParams(temperature=0.7, top_p=0.95)

prompts = ["Hello world", "What is AI?", "Python tutorial"]
outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    print(output.outputs[0].text)

Performance: - Throughput: 1000+ tokens/second (10x!) - Latency (first token): 50-100ms (10x better!) - Batch size: 32-128 (fully utilized) - Memory: Optimized (paged attention) - Concurrent users: 50-100 (production-ready!)

Why the difference: - Continuous batching - Optimized KV cache (PagedAttention) - Smart scheduling - Kernel optimization - Everything optimized for inference 

---

## Key Metrics for Inference

### Throughput (Tokens/Second)
Definition: How many tokens can be generated per second

Measurement: - Single request: 1 prompt, measure time to generate N tokens - Tokens per second = N / time_in_seconds - Example: Generate 100 tokens in 1 second = 100 tokens/sec

Factors: - Model size: Larger model = lower throughput - Batch size: Larger batch = higher throughput - Hardware: Better GPU = higher throughput - Optimization: Clever framework = higher throughput - Technique: KV cache, quantization = higher throughput

Benchmarks:

Model Hardware Batch 1 Batch 8 Batch 32 Framework ──────────────────────────────────────────────────────── LLaMA 7B A100 150 300 400 PyTorch LLaMA 7B A100 200 1000 3000 vLLM LLaMA 70B A100 30 60 100 PyTorch LLaMA 70B A100 50 400 800 vLLM

Key insight: vLLM 10x better with batching! 

### Latency (Milliseconds to First Token)
Definition: Time from request to first token generation

Measurement: - Send request at T=0 - Receive first token at T=X - Latency = X milliseconds

Factors: - Model size: Larger = slower - Optimization: Can reduce overhead - Queue depth: Many requests = longer wait - Hardware: Better GPU = faster

Targets by use case:

Use case Target Latency Requirement ────────────────────────────────────────── Real-time chat <100ms Interactive Batch API 1-5s Not urgent Stream <500ms Smooth UX

Framework comparison:

Framework Latency (empty queue) With requests queued ────────────────────────────────────────────────────────── PyTorch 200-500ms 2-10s vLLM 50-100ms 100-500ms TensorRT 30-50ms 50-200ms DeepSpeed 50-100ms 100-500ms 

### Memory Usage
Metric: How much GPU memory needed

Baseline (PyTorch): LLaMA 7B - Model weights (FP16): 14GB - Activations (batch 1): 2GB - Total: ~16GB

With framework optimization:

vLLM (continuous batching): - Model weights: 14GB - KV cache (optimized): 2GB (vs 8GB naive!) - Active batches: 1GB - Total: ~17GB (same!)

Benefit: Same memory, more throughput! 

---

## Framework Categories

### 1. Lightweight (Simple, Easy to Use)
Examples: Ollama, llama.cpp, HuggingFace Transformers

Characteristics: - Easy setup (one command) - Single-GPU focus - Limited production features - Good for prototyping

Use case: - Local development, single user, no scaling

Performance: - Throughput: Moderate (100-300 tok/s) - Latency: Moderate (100-500ms) - Concurrency: Limited (1-4 users)

Ollama example: 

# Download and run in one command
ollama run llama2

# Can run locally on any GPU
# But: Limited to single machine

### 2. Production (High Performance, Feature-Rich)
Examples: vLLM, TensorRT-LLM, DeepSpeed

Characteristics: - Complex setup - Distributed support - Full optimization - Production features (monitoring, etc.)

Use case: - Production systems, scale, performance critical

Performance: - Throughput: Excellent (1000+ tok/s with batching) - Latency: Low (50-100ms first token) - Concurrency: High (50-100+ users)

vLLM example: 

from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-2-70b",
    tensor_parallel_size=4,  # Distributed!
    dtype="float16",
    gpu_memory_utilization=0.9,
)

# Can handle 100+ concurrent requests!

### 3. Cloud (Managed Service)
Examples: OpenAI API, Anthropic Claude API, Hugging Face Inference API

Characteristics: - Zero infrastructure - Managed scaling - Pay-per-token - Minimal setup

Use case: - Prototyping, low volume, don't want to manage

Performance: - Throughput: Variable (depends on provider) - Latency: Varies (100ms-1s) - Concurrency: Unlimited (they handle it)

Cost: $0.001-0.01 per 1K tokens (expensive for high volume) 

---

## Framework Selection Decision Tree
Start: Which inference framework should I use? │ - Are you deploying to production at scale? │ │ - No → Use lightweight framework or cloud API - (Ollama for local, OpenAI API for cloud) │ │ - Yes → Continue below │ - Do you need multi-GPU/multi-node? │ │ - No → vLLM (easiest, great performance) │ │ - Yes → Continue below │ - Is latency critical (<100ms)? │ │ - Yes → TensorRT-LLM (best latency) │ │ - No → vLLM or DeepSpeed (both great) │ - Do you need specific hardware support? │ │ - NVIDIA GPU → vLLM, TensorRT, DeepSpeed (all good) │ │ - AMD GPU → vLLM (best AMD support) │ │ - CPU → DeepSpeed or llama.cpp (CPU optimized) │ - What's your experience level? │ - Beginner → vLLM (simplest production framework) │ - Experienced → TensorRT for latency, DeepSpeed for scale │ - Expert → Custom setup with best components 
---

## Performance Comparison Summary
Framework Ease Latency Throughput Multi-GPU Community ──────────────────────────────────────────────────────────────────── PyTorch Easy Slow Low Built-in Excellent Ollama Easy Medium Medium No Growing llama.cpp Easy Medium Medium No Growing

vLLM Medium Good Excellent Yes Excellent TensorRT-LLM Hard Best Very Good Yes Growing DeepSpeed-MII Hard Good Excellent Yes Growing

OpenAI API Easy Depends Depends Cloud N/A Anthropic API Easy Depends Depends Cloud N/A

Recommendation by scenario:

Single GPU, local development: - Ollama (easiest)

Production, <1000 req/day: - vLLM (great balance)

Production, high throughput: - vLLM (default) or TensorRT-LLM (if latency critical)

Cost-critical, low volume: - Cloud API (pay only for what you use)

Research/experimentation: - PyTorch (familiar, no overhead) ```

Key Takeaways

🎯 Inference frameworks: 10-100x faster than naive PyTorch
📊 Continuous batching: The key innovation enabling high throughput
âš¡ vLLM: Best all-around choice (performance + ease)
🚀 TensorRT: Best latency, harder to use
💡 Choose based on: Scale, latency requirements, team expertise