Inference Frameworks Fundamentals¶

Overview¶

Inference Frameworks are software systems optimized for running LLM inference in production. Different frameworks offer different trade-offs in throughput, latency, ease-of-use, and features.

Purpose: Serve models with high throughput, low latency, and fault tolerance
Popular options: vLLM, TensorRT-LLM, DeepSpeed-MII, Ollama, MLflow
Trade-off: Performance vs. Simplicity vs. Features
Decision: Depends on use case, scale, and requirements

-

Why Inference Frameworks Matter¶

Problem: Native Inference is Slow¶

Baseline: Run LLM with PyTorch/Transformers

Code:
```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b")

input_ids = tokenizer("Hello world", return_tensors="pt").input_ids
output = model.generate(input_ids, max_length=50)
print(tokenizer.decode(output[0]))

Performance:

Throughput: 50-100 tokens/second
Latency (first token): 500-1000ms
Batch size: 1 (inefficient!)
Memory: High (not optimized)
Concurrent users: ~1 (very limited)

Problems:

No continuous batching
Inefficient KV cache management
No request scheduling
Memory thrashing
Not production-ready!

Result: Can't serve 100+ users on single GPU

### Solution: Inference Framework

Using vLLM (inference framework)

Code:

from vllm import LLM, SamplingParams

llm = LLM(model="meta-llama/Llama-2-7b", tensor_parallel_size=1)
sampling_params = SamplingParams(temperature=0.7, top_p=0.95)

prompts = ["Hello world", "What is AI?", "Python tutorial"]
outputs = llm.generate(prompts, sampling_params)

for output in outputs:
 print(output.outputs[0].text)

Performance:

Throughput: 1000+ tokens/second (10x!)
Latency (first token): 50-100ms (10x better!)
Batch size: 32-128 (fully utilized)
Memory: Optimized (paged attention)
Concurrent users: 50-100 (production-ready!)

Why the difference:

Continuous batching
Optimized KV cache (PagedAttention)
Smart scheduling
Kernel optimization
Everything optimized for inference

-

## Key Metrics for Inference

### Throughput (Tokens/Second)

Definition: How many tokens can be generated per second

Measurement:

Single request: 1 prompt, measure time to generate N tokens
Tokens per second = N / time_in_seconds
Example: Generate 100 tokens in 1 second = 100 tokens/sec

Factors:

Model size: Larger model = lower throughput
Batch size: Larger batch = higher throughput
Hardware: Better GPU = higher throughput
Optimization: Clever framework = higher throughput
Technique: KV cache, quantization = higher throughput

Benchmarks:

Model Hardware Batch 1 Batch 8 Batch 32 Framework ──────────────────────────────────────────────────────── LLaMA 7B A100 150 300 400 PyTorch LLaMA 7B A100 200 1000 3000 vLLM LLaMA 70B A100 30 60 100 PyTorch LLaMA 70B A100 50 400 800 vLLM

Key insight: vLLM 10x better with batching!

### Latency (Milliseconds to First Token)

Definition: Time from request to first token generation

Measurement:

Send request at T=0
Receive first token at T=X
Latency = X milliseconds

Factors:

Model size: Larger = slower
Optimization: Can reduce overhead
Queue depth: Many requests = longer wait
Hardware: Better GPU = faster

Targets by use case:

Use case Target Latency Requirement ────────────────────────────────────────── Real-time chat <100ms Interactive Batch API 1-5s Not urgent Stream <500ms Smooth UX

Framework comparison:

Framework Latency (empty queue) With requests queued ────────────────────────────────────────────────────────── PyTorch 200-500ms 2-10s vLLM 50-100ms 100-500ms TensorRT 30-50ms 50-200ms DeepSpeed 50-100ms 100-500ms

### Memory Usage

Metric: How much GPU memory needed

Baseline (PyTorch): LLaMA 7B

Model weights (FP16): 14GB
Activations (batch 1): 2GB
Total: ~16GB

With framework optimization:

vLLM (continuous batching):

Model weights: 14GB
KV cache (optimized): 2GB (vs 8GB naive!)
Active batches: 1GB
Total: ~17GB (same!)

Benefit: Same memory, more throughput!

-

## Framework Categories

### 1. Lightweight (Simple, Easy to Use)

Examples: Ollama, llama.cpp, HuggingFace Transformers

Characteristics:

Easy setup (one command)
Single-GPU focus
Limited production features
Good for prototyping

Use case:

Local development, single user, no scaling

Performance:

Throughput: Moderate (100-300 tok/s)
Latency: Moderate (100-500ms)
Concurrency: Limited (1-4 users)

Ollama example:

# Download and run in one command
ollama run llama2

# Can run locally on any GPU
# But

### 2. Production (High Performance, Feature-Rich)

Examples: vLLM, TensorRT-LLM, DeepSpeed

Characteristics:

Complex setup
Distributed support
Full optimization
Production features (monitoring, etc.)

Use case:

Production systems, scale, performance critical

Performance:

Throughput: Excellent (1000+ tok/s with batching)
Latency: Low (50-100ms first token)
Concurrency: High (50-100+ users)

vLLM example:

from vllm import LLM, SamplingParams

llm = LLM(
 model="meta-llama/Llama-2-70b",
 tensor_parallel_size=4, # Distributed!
 dtype="float16",
 gpu_memory_utilization=0.9,
)

# Can handle 100+ concurrent requests!

### 3. Cloud (Managed Service)

Examples: OpenAI API, Anthropic Claude API, Hugging Face Inference API

Characteristics:

Zero infrastructure
Managed scaling
Pay-per-token
Minimal setup

Use case:

Prototyping, low volume, don't want to manage

Performance:

Throughput: Variable (depends on provider)
Latency: Varies (100ms-1s)
Concurrency: Unlimited (they handle it)

Cost: $0.001-0.01 per 1K tokens (expensive for high volume)

-

## Framework Selection Decision Tree

Start: Which inference framework should I use? │

Are you deploying to production at scale? │ │
No → Use lightweight framework or cloud API
(Ollama for local, OpenAI API for cloud) │ │
Yes → Continue below │
Do you need multi-GPU/multi-node? │ │
No → vLLM (easiest, great performance) │ │
Yes → Continue below │
Is latency critical (<100ms)? │ │
Yes → TensorRT-LLM (best latency) │ │
No → vLLM or DeepSpeed (both great) │
Do you need specific hardware support? │ │
NVIDIA GPU → vLLM, TensorRT, DeepSpeed (all good) │ │
AMD GPU → vLLM (best AMD support) │ │
CPU → DeepSpeed or llama.cpp (CPU optimized) │
What's your experience level? │
Beginner → vLLM (simplest production framework) │
Experienced → TensorRT for latency, DeepSpeed for scale │
Expert → Custom setup with best components

-

## Performance Comparison Summary

Framework Ease Latency Throughput Multi-GPU Community ──────────────────────────────────────────────────────────────────── PyTorch Easy Slow Low Built-in Excellent Ollama Easy Medium Medium No Growing llama.cpp Easy Medium Medium No Growing

vLLM Medium Good Excellent Yes Excellent TensorRT-LLM Hard Best Very Good Yes Growing DeepSpeed-MII Hard Good Excellent Yes Growing

OpenAI API Easy Depends Depends Cloud N/A Anthropic API Easy Depends Depends Cloud N/A

Recommendation by scenario:

Single GPU, local development:

Ollama (easiest)

Production, <1000 req/day:

vLLM (great balance)

Production, high throughput:

vLLM (default) or TensorRT-LLM (if latency critical)

Cost-critical, low volume:

Cloud API (pay only for what you use)

Research/experimentation:

PyTorch (familiar, no overhead)

```

Key Takeaways¶

Inference frameworks: 10-100x faster than naive PyTorch Continuous batching: The key innovation enabling high throughput vLLM: Best all-around choice (performance + ease) TensorRT: Best latency, harder to use Choose based on: Scale, latency requirements, team expertise

-

Vllm - Easiest production framework
03 Tensorrt Llm - Best for latency-critical systems
Deepspeed Mii - Distributed training & inference
[Framework Comparison & Selection](/01-modeling/03-inference/frameworks/(framework-comparison-selection/) - Detailed comparison table

Inference Frameworks Fundamentals¶

Overview¶

Why Inference Frameworks Matter¶

Problem: Native Inference is Slow¶

Key Takeaways¶

Related Notes in Inference Frameworks Subdirectory¶