llama.cpp

llama.cpp is an open-source C/C++ inference engine for large language models (LLMs), created by Georgi Gerganov (a former developer at Google). It enables running LLMs locally on consumer hardware, including CPUs without GPU acceleration, making LLM inference accessible to anyone with a standard computer. The project started as a pure C port of Meta's LLaMA model inference code and has since evolved into a comprehensive inference framework supporting dozens of model architectures including LLaMA, LLaMA 2, LLaMA 3, Mistral, Mixtral, Phi, Gemma, Qwen, DeepSeek, Falcon, and many more. Key technical innovations include: GGUF (GPT-Generated Unified Format) as the standard model file format, replacing the older GGML format, which allows storing model weights, tokenizer, and configuration in a single file; quantization support ranging from 2-bit to 8-bit precision (Q2_K through Q8_0), dramatically reducing memory requirements while maintaining acceptable output quality; CPU-based inference using AVX2, AVX-512, NEON, and WASM SIMD instructions for maximum performance on commodity hardware; GPU acceleration via CUDA (NVIDIA), Metal (Apple Silicon), ROCm (AMD), Vulkan, SYCL, and OpenCL backends; a built-in HTTP server mode (llama-server) providing an OpenAI-compatible API endpoint for drop-in replacement of commercial LLM APIs; and support for LoRA adapters for model fine-tuning. As of 2026, llama.cpp has over 75,000 GitHub stars and is the foundation for popular tools like Ollama, LM Studio, and Jan.