vLLM: Complete Guide to High-Performance LLM Serving¶

Overview¶

vLLM (Virtual LLM) is a high-throughput and memory-efficient inference engine for Large Language Models. It dramatically improves LLM serving performance through innovative memory management and scheduling techniques.

GitHub: vllm-project/vllm
Website: vllm.ai
License: Apache 2.0
Language: Python (C++ backend)
Latest: Actively maintained with frequent updates

Why vLLM? The Problem It Solves¶

Traditional LLM Serving Bottlenecks¶

❌ Memory Inefficiency: Wastes GPU memory with large batch sizes
❌ Slow Generation: Sequential token generation is bottleneck
❌ Low Throughput: Can only serve 1-2 requests concurrently
❌ Memory Fragmentation: KV cache wastes space

vLLM Solutions¶

✅ PagedAttention: Manages KV cache like OS virtual memory (50% memory savings)
✅ Continuous Batching: Serves multiple requests dynamically
✅ GPU Optimization: CUDA kernels for fast inference
✅ Higher Throughput: 10-40x improvement over standard serving

Key Capabilities¶

1. PagedAttention Algorithm¶

The breakthrough innovation that makes vLLM efficient.

How it works: - Divides KV cache into logical pages (512 tokens each by default) - Maps logical pages to physical GPU pages (like virtual memory in OS) - Allows non-contiguous memory allocation - Eliminates memory fragmentation

Benefits: - 2x throughput improvement - 50% memory reduction - Better memory utilization with variable sequence lengths

2. Continuous Batching¶

Dynamically adds/removes requests from batch as tokens are generated.

Traditional approach:

Batch: [Request1, Request2, Request3]
After 50 tokens:
Batch: [Request1, Request2, Request3]  (all still there, even if Request2 is done)

vLLM's continuous batching:

Time 0: [Request1, Request2, Request3]
Time 1: [Request1, Request2, Request3, Request4]  (add new request)
Time 2: [Request1, Request3, Request4]  (Request2 done, removed)
Time 3: [Request1, Request3, Request4, Request5, Request6]  (add 2 new)

3. Multi-GPU & Multi-Node Support¶

Distributed inference across multiple GPUs or nodes.

Tensor Parallelism: Split model across GPUs
Pipeline Parallelism: Different pipeline stages on different GPUs
Ray Integration: Distributed serving framework

4. Diverse Model Support¶

Works with various model architectures: - Llama 2, Llama 3 - Mistral - Falcon - Yi - Qwen - GPT-3/4 (via API) - Phi - And many more

5. Multiple Serving Options¶

OpenAI-Compatible API: Drop-in replacement for OpenAI API
Python API: Direct Python integration
gRPC/REST: Advanced deployment options

6. Quantization Support¶

Reduce model size without significant quality loss: - GPTQ: Post-training quantization - AWQ: Activation-aware quantization - SqueezeLLM: On-device quantization - FP8: 8-bit floating point

Installation & Setup¶

Basic Installation¶

# CPU only (not recommended for LLM inference)
pip install vllm

# GPU support (CUDA 11.8+)
pip install vllm

# Verify installation
python -c "import vllm; print(vllm.__version__)"

Installation with Specific CUDA Version¶

# CUDA 12.1
pip install vllm --index-url https://download.pytorch.org/whl/cu121

# CUDA 11.8
pip install vllm --index-url https://download.pytorch.org/whl/cu118

# ROCm for AMD GPUs
pip install vllm-rocm

From Source (Development)¶

git clone https://github.com/vllm-project/vllm.git
cd vllm

# Install dependencies
pip install -e .  # Install in editable mode

# Build from source
python setup.py build_ext --inplace

Docker Setup¶

# Pull official vLLM Docker image
docker pull vllm/vllm-openai:latest

# Run container with GPU
docker run --gpus all -p 8000:8000 vllm/vllm-openai:latest

# Run with specific model
docker run --gpus all -p 8000:8000 vllm/vllm-openai:latest \
    --model meta-llama/Llama-2-7b-hf

Requirements¶

# Minimum hardware
- GPU with 6GB+ VRAM (8GB+ recommended)
- CUDA 11.8+ or ROCm 5.5+
- 16GB+ RAM

# For large models
- A100/H100 GPUs recommended
- Multi-GPU setup for >70B models

Core Concepts & Architecture¶

KV Cache (Key-Value Cache)¶

What it is: Stores computed attention keys and values from previous tokens to avoid recomputation.

Token 1: Compute K1, V1, store in cache
Token 2: Use K1, V1 from cache + compute K2, V2
Token 3: Use K1, V2, K2, V2 from cache + compute K3, V3
...

Memory impact: - Without cache: O(n²) complexity, huge memory for long sequences - With cache: O(n) complexity, but cache grows with sequence length - vLLM's PagedAttention: Efficient cache management with virtual memory

Model Loading¶

How vLLM loads models:

from vllm import LLM

# Load model from HuggingFace Hub
llm = LLM(
    model="meta-llama/Llama-2-7b-hf",
    tensor_parallel_size=2,  # Distribute across 2 GPUs
    gpu_memory_utilization=0.9,  # Use 90% of GPU memory
    dtype="float16"  # Data type (float16, bfloat16, float32)
)

Models are loaded from: - Hugging Face Hub (automatically downloaded) - Local file paths - Custom model formats

Usage Examples¶

1. Basic Text Generation¶

from vllm import LLM, SamplingParams

# Initialize LLM
llm = LLM(
    model="meta-llama/Llama-2-7b-hf",
    gpu_memory_utilization=0.8
)

# Define sampling parameters
sampling_params = SamplingParams(
    temperature=0.7,
    top_p=0.95,
    max_tokens=256
)

# Generate text
prompts = [
    "Explain quantum computing in simple terms.",
    "Write a Python function to calculate factorial.",
    "What are the benefits of machine learning?"
]

outputs = llm.generate(prompts, sampling_params)

# Process outputs
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt}")
    print(f"Generated: {generated_text}")
    print("---")

2. Batch Processing with Different Sequence Lengths¶

from vllm import LLM, SamplingParams

llm = LLM(model="mistral-7b-v0.1", gpu_memory_utilization=0.9)

# Prompts with varying lengths
short_prompt = "What is AI?"
medium_prompt = "Explain machine learning with examples."
long_prompt = """Write a detailed article about natural language processing. 
Include topics like tokenization, embeddings, transformers, and applications."""

sampling_params = SamplingParams(
    temperature=0.8,
    max_tokens=512
)

# vLLM handles variable lengths efficiently with PagedAttention
outputs = llm.generate(
    [short_prompt, medium_prompt, long_prompt],
    sampling_params
)

for i, output in enumerate(outputs):
    print(f"Output {i+1}:\n{output.outputs[0].text}\n")

3. OpenAI-Compatible API Server¶

Start an OpenAI-compatible API server:

# Launch server
python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-2-7b-hf \
    --tensor-parallel-size 2 \
    --max-model-len 4096 \
    --dtype float16 \
    --port 8000

Use it like OpenAI API:

from openai import OpenAI

# Point to local vLLM server
client = OpenAI(
    api_key="anything",
    base_url="http://localhost:8000/v1"
)

# Use exactly like OpenAI API
completion = client.completions.create(
    model="meta-llama/Llama-2-7b-hf",
    prompt="Explain deep learning",
    max_tokens=256,
    temperature=0.7
)

print(completion.choices[0].text)

4. With LangChain Integration¶

from langchain.llms import VLLM
from langchain.prompts import PromptTemplate
from langchain.chains import LLMChain

# Initialize vLLM
llm = VLLM(
    model="meta-llama/Llama-2-7b-hf",
    trust_remote_code=True,
    max_new_tokens=256,
    top_p=0.95,
    temperature=0.8
)

# Create chain
prompt = PromptTemplate(
    input_variables=["topic"],
    template="Write a short article about {topic}"
)

chain = LLMChain(llm=llm, prompt=prompt)

# Generate
result = chain.run(topic="Artificial Intelligence")
print(result)

5. Advanced Sampling Parameters¶

from vllm import LLM, SamplingParams

llm = LLM(model="mistral-7b-v0.1")

# Different sampling strategies
sampling_params = SamplingParams(
    # Greedy decoding (deterministic)
    temperature=0.0,

    # Top-K sampling (sample from top K tokens)
    # top_k=50,

    # Top-P sampling (nucleus sampling)
    top_p=0.95,
    temperature=0.7,

    # Beam search
    # use_beam_search=True,
    # best_of=3,

    # Repetition penalty (reduce repetition)
    repetition_penalty=1.1,

    # Length penalty
    length_penalty=1.0,

    # Max tokens
    max_tokens=256,

    # Stop tokens (stop generation)
    stop=["Human:", "AI:"]
)

output = llm.generate(
    "Explain quantum computing",
    sampling_params
)

print(output[0].outputs[0].text)

6. Multi-GPU Tensor Parallelism¶

from vllm import LLM, SamplingParams

# Use 4 GPUs (tensor parallelism)
llm = LLM(
    model="meta-llama/Llama-2-70b-hf",  # 70B model needs multiple GPUs
    tensor_parallel_size=4,
    gpu_memory_utilization=0.9,
    dtype="float16"
)

sampling_params = SamplingParams(
    max_tokens=512,
    temperature=0.7
)

# Rest of code is the same
outputs = llm.generate(
    ["What is AGI?", "Explain consciousness"],
    sampling_params
)

7. Using Quantized Models¶

from vllm import LLM, SamplingParams

# GPTQ quantized model (smaller, faster)
llm = LLM(
    model="TheBloke/Llama-2-7B-Chat-GPTQ",
    quantization="gptq"
)

# AWQ quantized model
llm_awq = LLM(
    model="casperhansen/llama-2-7b-orca-200k-awq",
    quantization="awq"
)

# FP8 quantization (on-device)
llm_fp8 = LLM(
    model="meta-llama/Llama-2-7b-hf",
    quantization="fp8"
)

8. Streaming Output¶

from vllm import LLM, SamplingParams

llm = LLM(model="mistral-7b-v0.1")

sampling_params = SamplingParams(
    max_tokens=256,
    temperature=0.7
)

# Get request ID for tracking
request_id = "req-001"

outputs = llm.generate(
    "Write a short story",
    sampling_params,
    use_tqdm=True  # Progress bar
)

# Process streamed output
for output in outputs:
    for choice in output.outputs:
        print(choice.text, end="", flush=True)

9. Python API with Custom Processing¶

from vllm import LLM, SamplingParams
from typing import List, Dict

class LocalLLMService:
    """Wrapper for local vLLM inference"""

    def __init__(self, model_name: str):
        self.llm = LLM(
            model=model_name,
            gpu_memory_utilization=0.8
        )
        self.sampling_params = SamplingParams(
            temperature=0.7,
            max_tokens=512,
            top_p=0.95
        )

    def generate(self, prompts: List[str]) -> List[str]:
        """Generate completions for multiple prompts"""
        outputs = self.llm.generate(prompts, self.sampling_params)
        return [output.outputs[0].text for output in outputs]

    def generate_with_context(self, 
                             context: str, 
                             question: str) -> str:
        """Generate answer based on context"""
        prompt = f"""Context: {context}

Question: {question}

Answer:"""
        output = self.llm.generate(prompt, self.sampling_params)
        return output[0].outputs[0].text

# Usage
service = LocalLLMService("meta-llama/Llama-2-7b-hf")

# Batch generation
results = service.generate([
    "What is machine learning?",
    "Explain neural networks",
    "What is deep learning?"
])

for result in results:
    print(result)
    print("---")

# Context-based generation
context = "The Earth orbits the Sun. The Moon orbits the Earth."
question = "How many moons does Earth have?"
answer = service.generate_with_context(context, question)
print(answer)

10. Performance Benchmarking¶

import time
from vllm import LLM, SamplingParams

def benchmark_vllm():
    """Benchmark vLLM performance"""

    llm = LLM(model="mistral-7b-v0.1", gpu_memory_utilization=0.9)

    sampling_params = SamplingParams(
        max_tokens=256,
        temperature=0.7
    )

    # Generate same prompt multiple times
    prompts = ["Explain AI"] * 100

    # Measure time
    start_time = time.time()
    outputs = llm.generate(prompts, sampling_params)
    end_time = time.time()

    # Calculate metrics
    total_time = end_time - start_time
    requests_per_sec = len(prompts) / total_time
    tokens_generated = sum(
        len(output.outputs[0].token_ids) for output in outputs
    )
    tokens_per_sec = tokens_generated / total_time

    print(f"Total time: {total_time:.2f}s")
    print(f"Requests/sec: {requests_per_sec:.2f}")
    print(f"Tokens/sec: {tokens_per_sec:.2f}")
    print(f"Total tokens: {tokens_generated}")

benchmark_vllm()

Performance Characteristics¶

Throughput Comparison¶

Setup	Requests/Second	Tokens/Second
Standard PyTorch	1-2	5-50
vLLM (Single GPU)	10-20	200-500
vLLM (Multi-GPU)	50+	2000+

Memory Efficiency¶

Model	Memory (Standard)	Memory (vLLM)	Reduction
Llama 2 7B	16GB	8GB	50%
Llama 2 13B	26GB	13GB	50%
Llama 2 70B	140GB	70GB	50%

Latency Comparison¶

Task	Standard	vLLM
Generate 100 tokens	8-10s	0.5-1s
Batch 10 requests	80-100s	5-10s

Configuration Best Practices¶

GPU Memory Utilization¶

from vllm import LLM

# Conservative (more stability, lower throughput)
llm_conservative = LLM(
    model="meta-llama/Llama-2-7b-hf",
    gpu_memory_utilization=0.7
)

# Moderate (balanced)
llm_moderate = LLM(
    model="meta-llama/Llama-2-7b-hf",
    gpu_memory_utilization=0.85
)

# Aggressive (high throughput, risk of OOM)
llm_aggressive = LLM(
    model="meta-llama/Llama-2-7b-hf",
    gpu_memory_utilization=0.95
)

Max Model Length¶

# Short context (faster, less memory)
llm_short = LLM(
    model="meta-llama/Llama-2-7b-hf",
    max_model_len=2048
)

# Medium context
llm_medium = LLM(
    model="meta-llama/Llama-2-7b-hf",
    max_model_len=4096
)

# Long context (more memory)
llm_long = LLM(
    model="meta-llama/Llama-2-7b-hf",
    max_model_len=8192
)

Data Types¶

# FP32 - Highest precision, uses most memory
llm_fp32 = LLM(model="mistral-7b", dtype="float32")

# BF16 - Good precision, less memory (recommended)
llm_bf16 = LLM(model="mistral-7b", dtype="bfloat16")

# FP16 - Faster, less memory, potential precision loss
llm_fp16 = LLM(model="mistral-7b", dtype="float16")

# Auto-detect
llm_auto = LLM(model="mistral-7b", dtype="auto")

Advanced Features¶

1. LoRA Adapters¶

Load fine-tuned LoRA adapters without retraining:

from vllm import LLM

llm = LLM(
    model="meta-llama/Llama-2-7b-hf",
    enable_lora=True,
    lora_modules={
        "qa": "/path/to/qa_adapter",
        "chat": "/path/to/chat_adapter"
    }
)

# Use specific adapter
output = llm.generate(
    "Question: What is AI?",
    lora_request={"id": 1, "lora_name": "qa"}
)

2. Prefix Caching¶

Reuse attention KV cache for common prefixes:

from vllm import LLM, SamplingParams

llm = LLM(
    model="mistral-7b",
    enable_prefix_caching=True  # Enable caching
)

# All these prompts share the first 50 tokens
prompts = [
    "The quick brown fox jumps over the lazy dog. Question 1: ...",
    "The quick brown fox jumps over the lazy dog. Question 2: ...",
    "The quick brown fox jumps over the lazy dog. Question 3: ...",
]

# vLLM reuses cached prefix
outputs = llm.generate(prompts, SamplingParams(max_tokens=256))

3. Structured Output with Guided Decoding¶

Generate output in specific format (JSON, regex pattern):

from vllm import LLM, SamplingParams

llm = LLM(model="mistral-7b")

# Generate JSON output
prompt = """Generate a person object with name and age.
Format: {"name": "...", "age": ...}
Response:"""

# Currently requires custom implementation or external tools
output = llm.generate(prompt, SamplingParams(max_tokens=256))
print(output[0].outputs[0].text)

Comparison with Alternatives¶

Feature	vLLM	TensorRT-LLM	Text Generation WebUI	Ollama
Performance	⭐⭐⭐⭐⭐	⭐⭐⭐⭐⭐	⭐⭐⭐	⭐⭐
Ease of Use	⭐⭐⭐⭐	⭐⭐	⭐⭐⭐⭐⭐	⭐⭐⭐⭐⭐
Memory Efficient	⭐⭐⭐⭐	⭐⭐⭐⭐	⭐⭐⭐	⭐⭐⭐
Scalability	⭐⭐⭐⭐	⭐⭐⭐⭐	⭐⭐	⭐⭐
Model Diversity	⭐⭐⭐⭐	⭐⭐⭐	⭐⭐⭐⭐	⭐⭐⭐

Best for: - vLLM: Production deployment, high throughput - TensorRT-LLM: Maximum performance, NVIDIA-only - Text Gen WebUI: User-friendly experimentation - Ollama: Simplicity, one-liner setup

Troubleshooting¶

Out of Memory (OOM) Error¶

# Solution 1: Reduce GPU memory utilization
llm = LLM(
    model="meta-llama/Llama-2-70b",
    gpu_memory_utilization=0.7  # Reduce from 0.9
)

# Solution 2: Reduce max model length
llm = LLM(
    model="meta-llama/Llama-2-70b",
    max_model_len=2048  # Reduce from 4096
)

# Solution 3: Use quantization
llm = LLM(
    model="TheBloke/Llama-2-70B-Chat-GPTQ",
    quantization="gptq"
)

# Solution 4: Use tensor parallelism across GPUs
llm = LLM(
    model="meta-llama/Llama-2-70b",
    tensor_parallel_size=4  # Use 4 GPUs
)

Slow Generation Speed¶

# Ensure batch processing
outputs = llm.generate(
    ["prompt1", "prompt2", "prompt3"],  # Multiple prompts
    sampling_params
)

# Check GPU utilization
# nvidia-smi

# Increase GPU memory utilization
llm = LLM(
    model="mistral-7b",
    gpu_memory_utilization=0.95  # Max out utilization
)

# Use quantized models
llm = LLM(
    model="TheBloke/Mistral-7B-AWQ",
    quantization="awq"
)

Model Not Found¶

# Check HuggingFace Hub
# https://huggingface.co/models

# Specify local path
llm = LLM(model="/local/path/to/model")

# Ensure HF token is set for private models
huggingface-cli login

Best Practices¶

✅ Do's¶

Batch requests - Process multiple prompts at once
Use quantization - Reduce memory for large models
Monitor GPU - Watch nvidia-smi during inference
Set max_model_len - Limit context for faster inference
Use bfloat16 - Good balance of precision and speed
Cache API server - Keep server running for multiple requests
Warm up GPU - Run inference before benchmarking
Profile performance - Measure tokens/sec and latency

❌ Don'ts¶

❌ Load different models without restarting
❌ Use float32 unless necessary
❌ Ignore GPU memory limits
❌ Process one prompt at a time (kills performance)
❌ Use max_model_len=32000 for short sequences
❌ Change quantization mid-session
❌ Ignore CUDA out-of-memory warnings
❌ Run on GPU without checking VRAM

Use Cases¶

1. High-Throughput Inference Service¶

Serve hundreds of requests per second

2. Real-Time Chat Application¶

Low-latency responses for interactive chat

3. Batch Processing¶

Process thousands of documents efficiently

4. Content Generation¶

Generate articles, emails, code at scale

5. Code Generation¶

Power IDE plugins and code assistant tools

6. Information Extraction¶

Extract structured data from documents

7. Question Answering¶

Q&A over large document collections

8. RAG (Retrieval Augmented Generation)¶

Combine vLLM with vector databases for knowledge-enhanced generation

Resources¶

GitHub: github.com/vllm-project/vllm
Documentation: vllm.readthedocs.io
Paper: "Efficient Memory Management for Large Language Model Serving"
Blog: vllm.ai/blog
Discord Community: Official vLLM Discord

Key Takeaways¶

🚀 vLLM delivers 10-40x throughput improvement over standard LLM serving
💾 PagedAttention reduces KV cache memory by 50%
⚡ Continuous batching dynamically manages inference requests
🔧 Multi-GPU support enables serving large models efficiently
📊 Production-ready with OpenAI-compatible API

Next Steps¶

Install vLLM and try basic examples
Benchmark your hardware to understand performance
Experiment with quantization for memory optimization
Deploy as API server for production use
Integrate with LangChain/RAG for advanced applications