vLLM: Easy, Fast, and Cheap LLM Serving

Authors: Kwon et al. Year: 2023 ArXiv/Link: https://arxiv.org/abs/2309.06180

Summary

Introduces PagedAttention for efficient memory management in LLM serving, enabling high throughput and low latency.

Key Concepts

• PagedAttention
• Memory management
• Paged KV cache
• High throughput
• Efficient serving

Impact

Practical serving framework achieving high utilization

Category

Serving & Optimization

---