Celery Distributed Task Queue

An asynchronous distributed task queue for Python applications, currently at version 5.6.3 and licensed under the BSD License, originally created by Ask Solem in 2009 and maintained through the Open Collective community funding platform. The system processes vast amounts of messages with a focus on real-time processing while supporting task scheduling through periodic beat schedules with cron-like syntax. Message broker support covers Redis, RabbitMQ, and Amazon SQS, with result backends including Redis, SQLAlchemy, Django ORM, MongoDB, Memcached, Apache Cassandra, Elasticsearch, and IronCache for persisting task outcomes. The Canvas system enables complex workflow composition through primitives including chains for sequential execution, groups for parallel fan-out, chords for barrier synchronization with callback invocation, and chunks for batched data partitioning across worker pools. Worker processes scale horizontally across machines, with prefetch counts, soft and hard time limits, task retry with exponential backoff, and priority-based routing through multiple queues. Monitoring tools include Flower for real-time web-based worker visibility, celery events for event-stream inspection, and Prometheus exporter for metrics integration. The library integrates natively with Django and Flask through documented patterns, with production deployments at Instagram processing billions of tasks daily, Mozilla handling build pipeline orchestration, and Yelp powering internal service communication. The project maintains over 25,000 GitHub stars with active development and extensive documentation.