LiteLLM

LiteLLM is an open-source AI gateway that functions as an OpenAI-compatible proxy, enabling developers to manage and route requests to over 100 large language models (LLMs) from multiple providers through a single, unified interface. It centralizes authentication, load balancing, spend tracking, and observability, allowing teams to control and monitor LLM usage without modifying their existing codebase. The system supports models from OpenAI, Anthropic, Azure, Google Gemini, Amazon Bedrock, and many others, all accessible via the same OpenAI API format (e.g., `openai.ChatCompletion.create`). One of its core features is dynamic load balancing across multiple model endpoints, which helps distribute traffic, reduce latency, and avoid rate limits or service outages. LiteLLM also provides fallback logic: if a primary model fails or returns an error, the gateway can automatically route the request to a secondary or tertiary model, ensuring reliability. Spend tracking is built in, allowing organizations to set budgets per user, per team, or per model, and to log all requests for cost allocation. The gateway supports rate limiting at the API key or user level to prevent overuse and control access. For security, LiteLLM handles authentication and authorization via API keys, integrates with existing identity providers, and can enforce guardrails (e.g., content moderation, prompt validation) before requests reach the model. It also includes prompt management capabilities, enabling teams to version and reuse templates. Logging can be shipped to external storage like S3 for audit trails and compliance. LiteLLM offers pass-through endpoints for models not yet natively supported, and it supports batching requests for higher throughput. The gateway can be deployed on-premises or in a c