Celery: Distributed task queue for Python
Asynchronous task queue with distributed worker execution.
Learn more about celery
Celery is a Python library for managing distributed task queues and asynchronous job execution. It operates by having client applications place task messages onto a broker, which then routes those messages to worker processes for execution. The system supports multiple message brokers (AMQP, Redis, SQS) and allows horizontal scaling through multiple workers and brokers. Common deployment patterns include background job processing, scheduled task execution, and workload distribution across server clusters.
Multi-broker support
Celery integrates with multiple message broker backends including RabbitMQ/AMQP, Redis, and AWS SQS, allowing users to select infrastructure based on their deployment requirements rather than being locked to a single system.
Language interoperability
While written in Python, Celery's message protocol can be implemented in other languages. Implementations exist for Node.js, PHP, Go, and Rust, enabling task queue communication across polyglot systems.
Flexible scheduling and routing
Celery provides task scheduling capabilities, message routing options, and support for task prioritization and rate limiting, allowing fine-grained control over job execution patterns and worker allocation.
from celery import Celery
app = Celery('tasks', broker='redis://localhost:6379')
@app.task
def send_email(recipient, subject, body):
# Send email logic here
return f"Email sent to {recipient}"
# Call task asynchronously
result = send_email.delay('user@example.com', 'Hello', 'Welcome!')Release candidate adds Django connection pooling, fixes nested chords with django-celery-results+Redis, and bumps Kombu to v5.6.0.
- –Upgrade Kombu to v5.6.0 or later; the <5.7 version cap has been removed.
- –Test nested chord workflows if using django-celery-results with Redis backend; a critical fix is included.
Beta 2 drops Python 3.8 support (now requires ≥3.9) and fixes critical MSSQL overflow, Redis prefetch scope, and group signature ordering bugs.
- –Upgrade to Python 3.9+ before deploying; Python 3.8 is no longer supported or tested in this release.
- –Apply MSSQL result backend fix for arithmetic overflow and Redis credential_provider/client_name configuration options if needed.
Beta release fixes memory leaks in exception handling and pending results, adds worker_eta_task_limit config, and resolves chord/chain routing bugs.
- –Set worker_eta_task_limit to cap ETA task memory usage if you schedule many future tasks.
- –Upgrade to Kombu v5.6.0b2; fixes include Pydantic integration with __future__.annotations and CouchDB backend forget() TypeError.
Top in Backend & APIs
Related Repositories
Discover similar tools and frameworks used by developers
gin
Fast HTTP router with JSON binding and middleware support.
whatsmeow
Go client library for WhatsApp web multidevice protocol.
evolution-api
Node.js REST API for programmatic WhatsApp control.
graphiql
Browser-based GraphQL IDE with language service tooling.
beego
MVC framework for Go with integrated ORM and routing.