DeepSpeed

DeepSpeed is an open-source deep learning optimization library developed by Microsoft, designed to make training of very large models (with hundreds of billions or trillions of parameters) faster, more memory-efficient, and more accessible. Created by the DeepSpeed team at Microsoft (led by Rangan Majumder and Junhua Wang), DeepSpeed has become one of the most important tools in the large-scale AI training ecosystem, used to train models like Megatron-Turing NLG (530 billion parameters), BLOOM (176 billion parameters), and GLM-130B. The library provides a suite of optimizations that can be applied to PyTorch training scripts with minimal code changes, typically just a few lines of initialization code. The flagship technology is the ZeRO (Zero Redundancy Optimizer) family of memory optimizations: ZeRO-Stage 1 partitions optimizer states (Adam moments) across data-parallel workers, reducing memory per GPU by up to 4x; ZeRO-Stage 2 also partitions gradients, achieving up to 8x reduction; and ZeRO-Stage 3 partitions model parameters themselves, enabling training of models with up to 1 trillion parameters on just 32 NVIDIA V100 GPUs. Additional optimizations include: ZeRO-Infinity (offloading optimizer states and parameters to CPU RAM and NVMe SSDs for even larger models), ZeRO-Offload (a lighter version for single-GPU training of large models), 1-bit Adam and 1-bit LAMB (communication compression reducing the network bandwidth required for distributed training by up to 5x), pipeline parallelism (splitting model layers across GPUs in a pipeline), sparse attention (reducing the O(n squared) attention computation to sub-quadratic complexity for long sequences), mixture of experts (MoE) support, curriculum learning, and multi-tensor fusion for faster backward passes. As of 2026, DeepSpeed has over 37,000 stars.