vLLM

vLLM is a high-throughput and memory-efficient inference and serving engine for large language models, developed originally at UC Berkeley's Sky Computing Lab by a team including Woosuk Kwon and Zhuohan Li. It has become one of the most popular open-source LLM serving solutions, widely adopted in production environments for deploying models from LLaMA, Mistral, Qwen, and other model families at scale. The core innovation of vLLM is PagedAttention, a novel attention algorithm inspired by the virtual memory and paging mechanisms in operating systems. Traditional LLM serving systems allocate contiguous memory blocks for each request's KV (Key-Value) cache, leading to significant memory fragmentation and waste, with typical GPU memory utilization rates of only 20 to 40 percent. PagedAttention breaks the KV cache into smaller fixed-size blocks (pages) that can be allocated and deallocated dynamically, similar to how OS page tables map virtual to physical memory. This approach reduces KV cache memory waste from over 60 percent to less than 4 percent, effectively doubling or tripling the throughput of existing LLM serving systems. vLLM supports continuous batching (dynamic in-flight batching) which processes new requests as soon as previous ones complete, maximizing GPU utilization. It also supports tensor parallelism for distributing model inference across multiple GPUs, pipeline parallelism, and seamless integration with popular quantization methods including AWQ, GPTQ, and FP8. The project provides an OpenAI-compatible API server for drop-in replacement. As of 2026, vLLM has over 30,000 GitHub stars.