ONNX Runtime Inference Engine

ONNX Runtime is a cross-platform, high-performance scoring engine for machine learning models in the ONNX (Open Neural Network Exchange) format. Developed by Microsoft and first released in 2018, ONNX Runtime has over 16,000 stars as of 2026 and is used in production by Microsoft products including Windows, Office, Bing, Teams, and Azure, serving billions of inferences daily. Key features include: ONNX format support (loading and running models exported from PyTorch, TensorFlow, scikit-learn, Keras, and other frameworks via ONNX converters), cross-platform (Windows, Linux, macOS, Android, iOS, and WebAssembly), hardware acceleration (CUDA and TensorRT for NVIDIA GPUs, DirectML for Windows GPUs, OpenVINO for Intel hardware, CoreML for Apple Silicon, XNNPACK for ARM, ROCm for AMD GPUs, and QNN for Qualcomm Neural Processing), execution providers (pluggable acceleration backends enabling hardware-specific optimization without model modification), quantization (dynamic and static quantization reducing model size and improving inference speed with INT8, UINT8, and float16 support), operator optimization (graph-level optimizations including constant folding, redundant node elimination, and fusion for Conv, MatMul, Attention, and GEMM operators), training support (ONNX Runtime Training for distributed training with gradient computation), performance optimization (graph partitioning, memory planning, and execution plan caching), multi-language bindings (C, C++, C#, Python, Java, JavaScript, Node.js, Ruby, Swift, and Objective-C), and custom operator support (extending with user-defined operators for proprietary or research models).