vLLM LLM Inference Engine

vLLM is a high-throughput, memory-efficient inference and serving engine for Large Language Models (LLMs), designed to maximize throughput and minimize latency in production deployments. Developed at UC Berkeley in 2023 by Woosuk Kwon and Zhuohan Li, vLLM has become the de facto standard for serving LLMs in production. Key features: PagedAttention algorithm that manages attention key and value cache memory like an operating system manages virtual memory with paging, reducing memory waste from traditional KV cache allocation and enabling near-optimal memory utilization. Continuous batching (iteration-level scheduling) that dynamically includes new requests in the current batch while existing requests are still being processed, maximizing GPU utilization compared to static batching. High throughput achieving 2-4x higher throughput than Hugging Face Transformers and up to 24x higher than naive implementations. Quantization support including AWQ, GPTQ, SqueezeLLM, and FP8 (KV cache) for reducing memory and increasing throughput at the cost of minor accuracy loss. Tensor parallelism for distributing model across multiple GPUs with NCCL communication. Pipeline parallelism for distributing across GPU nodes. LoRA (Low-Rank Adaptation) serving for running multiple fine-tuned models from a single base model simultaneously. Streaming generation for real-time token-by-token output. OpenAI-compatible API server enabling drop-in replacement for OpenAI API calls with /v1/completions and /v1/chat/completions endpoints. Supports major LLM families including Llama, Mistral, Qwen, Phi, Gemma, Yi, and hundreds of Hugging Face models. Speculative decoding for reducing latency via draft model verification. Chunked prefill for overlapping prefill and decode operations. Distributed serving via Ray for multi-node deployment. Used by Anyscale, LMSys (ChatBot Arena), and production AI deployments worldwide.