Unsloth

Unsloth is an open-source library that dramatically accelerates the fine-tuning of large language models, achieving 2x to 5x speed improvements and reducing memory usage by 50 to 80 percent compared to standard Hugging Face Transformers training pipelines. Developed by the Unsloth AI team (Daniel Han and Michael Han), it has become one of the fastest-growing ML tools in the open-source community due to its remarkable performance gains. The library currently supports fine-tuning popular model families including LLaMA (1, 2, 3, 3.1, 3.2, 3.3), Mistral, Mixtral, Gemma (1, 2, 3), Phi (2, 3, 4), Qwen (1.5, 2, 2.5), and Yi. Key technical innovations include: manual derivation and hand-optimized Triton kernels for the forward and backward passes of each model architecture, replacing PyTorch autograd to eliminate overhead; fused kernel implementations for operations like RoPE (Rotary Position Embeddings), RMSNorm, and cross-entropy loss that combine multiple operations into a single GPU kernel launch to reduce memory bandwidth pressure; support for 4-bit and 8-bit quantization (via bitsandbytes integration) to fit large models on consumer GPUs; UnslothGradientCheckpointing for reducing the memory required for intermediate activations during backpropagation; support for LoRA (Low-Rank Adaptation) and QLoRA parameter-efficient fine-tuning; and integration with Hugging Face TRL's SFTTrainer, DPOTrainer, and PPOTrainer for alignment training. Unsloth also provides UnslothX for inference acceleration and supports multi-GPU training via Unsloth's distributed mode. As of 2026, Unsloth has over 25,000 GitHub stars and is used by thousands of developers and researchers for efficient on-premise model training.