vLLM

Overview

vLLM (Virtual LLM) is a high-throughput and memory-efficient inference engine for Large Language Models. It dramatically improves LLM serving performance through innovative memory management and scheduling techniques.

• GitHub: vllm-project/vllm
• Website: vllm.ai
• License: Apache 2.0
• Language: Python (C++ backend)
• Latest: Actively maintained with frequent updates

---

Why vLLM? The Problem It Solves

Traditional LLM Serving Bottlenecks

Memory Inefficiency: Wastes GPU memory with large batch sizes Slow Generation: Sequential token generation is bottleneck Low Throughput: Can only serve 1-2 requests concurrently Memory Fragmentation: KV cache wastes space

vLLM Solutions

PagedAttention: Manages KV cache like OS virtual memory (50% memory savings) Continuous Batching: Serves multiple requests dynamically GPU Optimization: CUDA kernels for fast inference Higher Throughput: 10-40x improvement over standard serving

-

Key Capabilities

1. PagedAttention Algorithm

The breakthrough innovation that makes vLLM efficient.

How it works:

• Divides KV cache into logical pages (512 tokens each by default)
• Maps logical pages to physical GPU pages (like virtual memory in OS)
• Allows non-contiguous memory allocation
• Eliminates memory fragmentation

Benefits:

• 2x throughput improvement
• 50% memory reduction
• Better memory utilization with variable sequence lengths

2. Continuous Batching

Dynamically adds/removes requests from batch as tokens are generated.

Traditional approach:

Batch: [Request1, Request2, Request3]
After 50 tokens:
Batch: [Request1, Request2, Request3] (all still there, even if Request2 is done)

vLLM's continuous batching:

Time 0: [Request1, Request2, Request3]
Time 1: [Request1, Request2, Request3, Request4] (add new request)
Time 2: [Request1, Request3, Request4] (Request2 done, removed)
Time 3: [Request1, Request3, Request4, Request5, Request6] (add 2 new)

3. Multi-GPU & Multi-Node Support

Distributed inference across multiple GPUs or nodes.

• Tensor Parallelism: Split model across GPUs
• Pipeline Parallelism: Different pipeline stages on different GPUs
• Ray Integration: Distributed serving framework

4. Diverse Model Support

Works with various model architectures:

• Llama 2, Llama 3
• Mistral
• Falcon
• Yi
• Qwen
• GPT-3/4 (via API)
• Phi
• And many more

5. Multiple Serving Options

• OpenAI-Compatible API: Drop-in replacement for OpenAI API
• Python API: Direct Python integration
• gRPC/REST: Advanced deployment options

6. Quantization Support

Reduce model size without significant quality loss:

• GPTQ: Post-training quantization
• AWQ: Activation-aware quantization
• SqueezeLLM: On-device quantization
• FP8: 8-bit floating point

-

Installation & Setup

Basic Installation

# CPU only (not recommended for LLM inference)
pip install vllm

# GPU support (CUDA 11.8+)
pip install vllm

# Verify installation
python -c "import vllm; print(vllm.__version__)"

Installation with Specific CUDA Version

# CUDA 12.1
pip install vllm --index-url https://download.pytorch.org/whl/cu121

# CUDA 11.8
pip install vllm --index-url https://download.pytorch.org/whl/cu118

# ROCm for AMD GPUs
pip install vllm-rocm

From Source (Development)

git clone https://github.com/vllm-project/vllm.git
cd vllm

# Install dependencies
pip install -e. # Install in editable mode

# Build from source
python setup.py build_ext --inplace

Docker Setup

# Pull official vLLM Docker image
docker pull vllm/vllm-openai:latest

# Run container with GPU
docker run --gpus all -p 8000:8000 vllm/vllm-openai:latest

# Run with specific model
docker run --gpus all -p 8000:8000 vllm/vllm-openai:latest \
 --model meta-llama/Llama-2-7b-hf

Requirements

# Minimum hardware
- GPU with 6GB+ VRAM (8GB+ recommended)
- CUDA 11.8+ or ROCm 5.5+
- 16GB+ RAM

# For large models
- A100/H100 GPUs recommended
- Multi-GPU setup for >70B models

-

Core Concepts & Architecture

KV Cache (Key-Value Cache)

What it is: Stores computed attention keys and values from previous tokens to avoid recomputation.

Token 1: Compute K1, V1, store in cache
Token 2: Use K1, V1 from cache + compute K2, V2
Token 3: Use K1, V2, K2, V2 from cache + compute K3, V3
...

Memory impact:

• Without cache: O(n²) complexity, huge memory for long sequences
• With cache: O(n) complexity, but cache grows with sequence length
• vLLM's PagedAttention: Efficient cache management with virtual memory

Model Loading

How vLLM loads models:

from vllm import LLM

# Load model from HuggingFace Hub
llm = LLM(
 model="meta-llama/Llama-2-7b-hf",
 tensor_parallel_size=2, # Distribute across 2 GPUs
 gpu_memory_utilization=0.9, # Use 90% of GPU memory
 dtype="float16" # Data type (float16, bfloat16, float32)
)

Models are loaded from:

• Hugging Face Hub (automatically downloaded)
• Local file paths
• Custom model formats

-

Usage Examples

1. Basic Text Generation

from vllm import LLM, SamplingParams

# Initialize LLM
llm = LLM(
 model="meta-llama/Llama-2-7b-hf",
 gpu_memory_utilization=0.8
)

# Define sampling parameters
sampling_params = SamplingParams(
 temperature=0.7,
 top_p=0.95,
 max_tokens=256
)

# Generate text
prompts = [
 "Explain quantum computing in simple terms.",
 "Write a Python function to calculate factorial.",
 "What are the benefits of machine learning?"
]

outputs = llm.generate(prompts, sampling_params)

# Process outputs
for output in outputs:
 prompt = output.prompt
 generated_text = output.outputs[0].text
 print(f"Prompt: {prompt}")
 print(f"Generated: {generated_text}")
 print("-")

2. Batch Processing with Different Sequence Lengths

from vllm import LLM, SamplingParams

llm = LLM(model="mistral-7b-v0.1", gpu_memory_utilization=0.9)

# Prompts with varying lengths
short_prompt = "What is AI?"
medium_prompt = "Explain machine learning with examples."
long_prompt = """Write a detailed article about natural language processing. 
Include topics like tokenization, embeddings, transformers, and applications."""

sampling_params = SamplingParams(
 temperature=0.8,
 max_tokens=512
)

# vLLM handles variable lengths efficiently with PagedAttention
outputs = llm.generate(
 [short_prompt, medium_prompt, long_prompt],
 sampling_params
)

for i, output in enumerate(outputs):
 print(f"Output {i+1}:\n{output.outputs[0].text}\n")

3. OpenAI-Compatible API Server

Start an OpenAI-compatible API server:

# Launch server
python -m vllm.entrypoints.openai.api_server \
 --model meta-llama/Llama-2-7b-hf \
 --tensor-parallel-size 2 \
 --max-model-len 4096 \
 --dtype float16 \
 --port 8000

Use it like OpenAI API:

from openai import OpenAI

# Point to local vLLM server
client = OpenAI(
 api_key="anything",
 base_url="http://localhost:8000/v1"
)

# Use exactly like OpenAI API
completion = client.completions.create(
 model="meta-llama/Llama-2-7b-hf",
 prompt="Explain deep learning",
 max_tokens=256,
 temperature=0.7
)

print(completion.choices[0].text)

4. With LangChain Integration

from langchain.llms import VLLM
from langchain.prompts import PromptTemplate
from langchain.chains import LLMChain

# Initialize vLLM
llm = VLLM(
 model="meta-llama/Llama-2-7b-hf",
 trust_remote_code=True,
 max_new_tokens=256,
 top_p=0.95,
 temperature=0.8
)

# Create chain
prompt = PromptTemplate(
 input_variables=["topic"],
 template="Write a short article about {topic}"
)

chain = LLMChain(llm=llm, prompt=prompt)

# Generate
result = chain.run(topic="Artificial Intelligence")
print(result)

5. Advanced Sampling Parameters

from vllm import LLM, SamplingParams

llm = LLM(model="mistral-7b-v0.1")

# Different sampling strategies
sampling_params = SamplingParams(
 # Greedy decoding (deterministic)
 temperature=0.0,

 # Top-K sampling (sample from top K tokens)
 # top_k=50,

 # Top-P sampling (nucleus sampling)
 top_p=0.95,
 temperature=0.7,

 # Beam search
 # use_beam_search=True,
 # best_of=3,

 # Repetition penalty (reduce repetition)
 repetition_penalty=1.1,

 # Length penalty
 length_penalty=1.0,

 # Max tokens
 max_tokens=256,

 # Stop tokens (stop generation)
 stop=["Human:", "AI:"]
)

output = llm.generate(
 "Explain quantum computing",
 sampling_params
)

print(output[0].outputs[0].text)

6. Multi-GPU Tensor Parallelism

from vllm import LLM, SamplingParams

# Use 4 GPUs (tensor parallelism)
llm = LLM(
 model="meta-llama/Llama-2-70b-hf", # 70B model needs multiple GPUs
 tensor_parallel_size=4,
 gpu_memory_utilization=0.9,
 dtype="float16"
)

sampling_params = SamplingParams(
 max_tokens=512,
 temperature=0.7
)

# Rest of code is the same
outputs = llm.generate(
 ["What is AGI?", "Explain consciousness"],
 sampling_params
)

7. Using Quantized Models

from vllm import LLM, SamplingParams

# GPTQ quantized model (smaller, faster)
llm = LLM(
 model="TheBloke/Llama-2-7B-Chat-GPTQ",
 quantization="gptq"
)

# AWQ quantized model
llm_awq = LLM(
 model="casperhansen/llama-2-7b-orca-200k-awq",
 quantization="awq"
)

# FP8 quantization (on-device)
llm_fp8 = LLM(
 model="meta-llama/Llama-2-7b-hf",
 quantization="fp8"
)

8. Streaming Output

from vllm import LLM, SamplingParams

llm = LLM(model="mistral-7b-v0.1")

sampling_params = SamplingParams(
 max_tokens=256,
 temperature=0.7
)

# Get request ID for tracking
request_id = "req-001"

outputs = llm.generate(
 "Write a short story",
 sampling_params,
 use_tqdm=True # Progress bar
)

# Process streamed output
for output in outputs:
 for choice in output.outputs:
 print(choice.text, end="", flush=True)

9. Python API with Custom Processing

from vllm import LLM, SamplingParams
from typing import List, Dict

class LocalLLMService:
 """Wrapper for local vLLM inference"""

 def __init__(self, model_name: str):
 self.llm = LLM(
 model=model_name,
 gpu_memory_utilization=0.8
)
 self.sampling_params = SamplingParams(
 temperature=0.7,
 max_tokens=512,
 top_p=0.95
)

 def generate(self, prompts: List[str]) -> List[str]:
 """Generate completions for multiple prompts"""
 outputs = self.llm.generate(prompts, self.sampling_params)
 return [output.outputs[0].text for output in outputs]

 def generate_with_context(self, 
 context: str, 
 question: str) -> str:
 """Generate answer based on context"""
 prompt = f"""Context: {context}

Question: {question}

Answer:"""
 output = self.llm.generate(prompt, self.sampling_params)
 return output[0].outputs[0].text

# Usage
service = LocalLLMService("meta-llama/Llama-2-7b-hf")

# Batch generation
results = service.generate([
 "What is machine learning?",
 "Explain neural networks",
 "What is deep learning?"
])

for result in results:
 print(result)
 print("-")

# Context-based generation
context = "The Earth orbits the Sun. The Moon orbits the Earth."
question = "How many moons does Earth have?"
answer = service.generate_with_context(context, question)
print(answer)

10. Performance Benchmarking

import time
from vllm import LLM, SamplingParams

def benchmark_vllm():
 """Benchmark vLLM performance"""

 llm = LLM(model="mistral-7b-v0.1", gpu_memory_utilization=0.9)

 sampling_params = SamplingParams(
 max_tokens=256,
 temperature=0.7
)

 # Generate same prompt multiple times
 prompts = ["Explain AI"] * 100

 # Measure time
 start_time = time.time()
 outputs = llm.generate(prompts, sampling_params)
 end_time = time.time()

 # Calculate metrics
 total_time = end_time - start_time
 requests_per_sec = len(prompts) / total_time
 tokens_generated = sum(
 len(output.outputs[0].token_ids) for output in outputs
)
 tokens_per_sec = tokens_generated / total_time

 print(f"Total time: {total_time:.2f}s")
 print(f"Requests/sec: {requests_per_sec:.2f}")
 print(f"Tokens/sec: {tokens_per_sec:.2f}")
 print(f"Total tokens: {tokens_generated}")

benchmark_vllm()

-

Performance Characteristics

Throughput Comparison

Setup	Requests/Second	Tokens/Second
Standard PyTorch	1-2	5-50
vLLM (Single GPU)	10-20	200-500
vLLM (Multi-GPU)	50+	2000+

Memory Efficiency

Model	Memory (Standard)	Memory (vLLM)	Reduction
Llama 2 7B	16GB	8GB	50%
Llama 2 13B	26GB	13GB	50%
Llama 2 70B	140GB	70GB	50%

Latency Comparison

Task	Standard	vLLM
Generate 100 tokens	8-10s	0.5-1s
Batch 10 requests	80-100s	5-10s

-

Configuration Best Practices

GPU Memory Utilization

from vllm import LLM

# Conservative (more stability, lower throughput)
llm_conservative = LLM(
 model="meta-llama/Llama-2-7b-hf",
 gpu_memory_utilization=0.7
)

# Moderate (balanced)
llm_moderate = LLM(
 model="meta-llama/Llama-2-7b-hf",
 gpu_memory_utilization=0.85
)

# Aggressive (high throughput, risk of OOM)
llm_aggressive = LLM(
 model="meta-llama/Llama-2-7b-hf",
 gpu_memory_utilization=0.95
)

Max Model Length

# Short context (faster, less memory)
llm_short = LLM(
 model="meta-llama/Llama-2-7b-hf",
 max_model_len=2048
)

# Medium context
llm_medium = LLM(
 model="meta-llama/Llama-2-7b-hf",
 max_model_len=4096
)

# Long context (more memory)
llm_long = LLM(
 model="meta-llama/Llama-2-7b-hf",
 max_model_len=8192
)

Data Types

# FP32 - Highest precision, uses most memory
llm_fp32 = LLM(model="mistral-7b", dtype="float32")

# BF16 - Good precision, less memory (recommended)
llm_bf16 = LLM(model="mistral-7b", dtype="bfloat16")

# FP16 - Faster, less memory, potential precision loss
llm_fp16 = LLM(model="mistral-7b", dtype="float16")

# Auto-detect
llm_auto = LLM(model="mistral-7b", dtype="auto")

-

Advanced Features

1. LoRA Adapters

Load fine-tuned LoRA adapters without retraining:

from vllm import LLM

llm = LLM(
 model="meta-llama/Llama-2-7b-hf",
 enable_lora=True,
 lora_modules={
 "qa": "/path/to/qa_adapter",
 "chat": "/path/to/chat_adapter"
 }
)

# Use specific adapter
output = llm.generate(
 "Question: What is AI?",
 lora_request={"id": 1, "lora_name": "qa"}
)

2. Prefix Caching

Reuse attention KV cache for common prefixes:

from vllm import LLM, SamplingParams

llm = LLM(
 model="mistral-7b",
 enable_prefix_caching=True # Enable caching
)

# All these prompts share the first 50 tokens
prompts = [
 "The quick brown fox jumps over the lazy dog. Question 1:...",
 "The quick brown fox jumps over the lazy dog. Question 2:...",
 "The quick brown fox jumps over the lazy dog. Question 3:...",
]

# vLLM reuses cached prefix
outputs = llm.generate(prompts, SamplingParams(max_tokens=256))

3. Structured Output with Guided Decoding

Generate output in specific format (JSON, regex pattern):

from vllm import LLM, SamplingParams

llm = LLM(model="mistral-7b")

# Generate JSON output
prompt = """Generate a person object with name and age.
Format: {"name": "...", "age":...}
Response:"""

# Currently requires custom implementation or external tools
output = llm.generate(prompt, SamplingParams(max_tokens=256))
print(output[0].outputs[0].text)

---

Comparison with Alternatives

Feature	vLLM	TensorRT-LLM	Text Generation WebUI	Ollama
Performance
Ease of Use
Memory Efficient
Scalability
Model Diversity

Best for:

• vLLM: Production deployment, high throughput
• TensorRT-LLM: Maximum performance, NVIDIA-only
• Text Gen WebUI: User-friendly experimentation
• Ollama: Simplicity, one-liner setup

-

Troubleshooting

Out of Memory (OOM) Error

# Solution 1
llm = LLM(
 model="meta-llama/Llama-2-70b",
 gpu_memory_utilization=0.7 # Reduce from 0.9
)

# Solution 2
llm = LLM(
 model="meta-llama/Llama-2-70b",
 max_model_len=2048 # Reduce from 4096
)

# Solution 3
llm = LLM(
 model="TheBloke/Llama-2-70B-Chat-GPTQ",
 quantization="gptq"
)

# Solution 4
llm = LLM(
 model="meta-llama/Llama-2-70b",
 tensor_parallel_size=4 # Use 4 GPUs
)

Slow Generation Speed

# Ensure batch processing
outputs = llm.generate(
 ["prompt1", "prompt2", "prompt3"], # Multiple prompts
 sampling_params
)

# Check GPU utilization
# nvidia-smi

# Increase GPU memory utilization
llm = LLM(
 model="mistral-7b",
 gpu_memory_utilization=0.95 # Max out utilization
)

# Use quantized models
llm = LLM(
 model="TheBloke/Mistral-7B-AWQ",
 quantization="awq"
)

Model Not Found

# Check HuggingFace Hub
# https://huggingface.co/models

# Specify local path
llm = LLM(model="/local/path/to/model")

# Ensure HF token is set for private models
huggingface-cli login

-

Best Practices

Do's

1. Batch requests - Process multiple prompts at once
2. Use quantization - Reduce memory for large models
3. Monitor GPU - Watch nvidia-smi during inference
4. Set max_model_len - Limit context for faster inference
5. Use bfloat16 - Good balance of precision and speed
6. Cache API server - Keep server running for multiple requests
7. Warm up GPU - Run inference before benchmarking
8. Profile performance - Measure tokens/sec and latency

Don'ts

1. Load different models without restarting
2. Use float32 unless necessary
3. Ignore GPU memory limits
4. Process one prompt at a time (kills performance)
5. Use max_model_len=32000 for short sequences
6. Change quantization mid-session
7. Ignore CUDA out-of-memory warnings
8. Run on GPU without checking VRAM

---

Use Cases

1. High-Throughput Inference Service

Serve hundreds of requests per second

2. Real-Time Chat Application

Low-latency responses for interactive chat

3. Batch Processing

Process thousands of documents efficiently

4. Content Generation

Generate articles, emails, code at scale

5. Code Generation

Power IDE plugins and code assistant tools

6. Information Extraction

Extract structured data from documents

7. Question Answering

Q&A over large document collections

8. RAG (Retrieval Augmented Generation)

Combine vLLM with vector databases for knowledge-enhanced generation

-

Resources

• GitHub: github.com/vllm-project/vllm
• Documentation: vllm.readthedocs.io
• Paper: "Efficient Memory Management for Large Language Model Serving"
• Blog: vllm.ai/blog
• Discord Community: Official vLLM Discord

---

Key Takeaways

vLLM delivers 10-40x throughput improvement over standard LLM serving PagedAttention reduces KV cache memory by 50% Continuous batching dynamically manages inference requests Multi-GPU support enables serving large models efficiently Production-ready with OpenAI-compatible API

-

Next Steps

1. Install vLLM and try basic examples
2. Benchmark your hardware to understand performance
3. Experiment with quantization for memory optimization
4. Deploy as API server for production use
5. Integrate with LangChain/RAG for advanced applications

-

Related Notes

• 00 Readme
• 01 Langchain
• 02 Ollama
• 00 Readme