Celery

Celery is a distributed task queue system for Python, enabling asynchronous execution of time-consuming operations across multiple workers and machines. Originally created by Ask Solem in 2009 and now maintained by the Celery Project, it is the most widely used task queue in the Python ecosystem with over 24,000 stars. Celery follows the producer-consumer pattern: producers (web applications, scripts) submit tasks to a message broker (Redis, RabbitMQ, Amazon SQS, Apache Pulsar, or IronMQ), and worker processes consume and execute these tasks asynchronously. Key features include: task definition via decorators (@app.task) with automatic retry, exponential backoff, and timeout handling, task chaining and groups for complex workflows (chain for sequential execution, group for parallel execution, chord for group-then-callback), task routing to different queues based on task type or priority, scheduled periodic tasks via Celery Beat (cron-style scheduling with timezone support), task result storage in Redis, PostgreSQL, MongoDB, Django ORM, or SQLAlchemy backends, task state inspection and monitoring via Flower (real-time web monitor) or Celery Events, worker concurrency modes (prefork, eventlet, gevent, solo, threads), rate limiting per task type, remote control commands for runtime worker management (inspect active tasks, terminate tasks, adjust pool size), and integration with major frameworks (Django, Flask, FastAPI, Pyramid). Celery supports Python 3.8+ and is licensed under BSD.