Web requests should be fast. Sending emails, generating PDFs, resizing images, calling slow APIs, and scheduled jobs belong in background workers. Celery is the standard task queue for Python (and Django), with Redis as a simple, fast message broker. Here’s a production setup on a VPS.

Concepts: background jobs and queues.

Install

sudo apt install -y redis-server
source /opt/app/venv/bin/activate
pip install "celery[redis]"

Secure Redis: bind to localhost (default), or a private network, and set a password if it’s reachable from other servers. See install Redis.

Configure the Celery app

# app/celery.py
import os
from celery import Celery

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "app.settings")   # Django projects

celery = Celery("app")
celery.conf.update(
    broker_url=os.getenv("CELERY_BROKER_URL", "redis://127.0.0.1:6379/0"),
    result_backend=os.getenv("CELERY_RESULT_BACKEND", "redis://127.0.0.1:6379/1"),
    task_acks_late=True,                 # ack after the task finishes (safer on crashes)
    worker_prefetch_multiplier=1,        # fairer distribution of long tasks
    task_time_limit=300,                 # hard kill after 5 min
    task_soft_time_limit=270,
    timezone="UTC",
)
celery.autodiscover_tasks()              # Django: finds tasks.py in installed apps

Write a task

# app/tasks.py
from app.celery import celery

@celery.task(bind=True, autoretry_for=(ConnectionError,), retry_backoff=True, max_retries=5)
def send_invoice_email(self, invoice_id):
    invoice = load_invoice(invoice_id)
    send_email(invoice.customer_email, render_invoice(invoice))

Call it from a view without waiting:

send_invoice_email.delay(invoice.id)

Pass IDs, not big objects. The task loads fresh data from the database.

Run a worker (test)

celery -A app worker --loglevel=info --concurrency=4

Run as systemd services

# /etc/systemd/system/celery-worker.service
[Unit]
Description=Celery worker
After=network.target redis-server.service

[Service]
User=app
WorkingDirectory=/opt/app
EnvironmentFile=/opt/app/.env
ExecStart=/opt/app/venv/bin/celery -A app worker --loglevel=info --concurrency=4
Restart=always
KillSignal=SIGTERM
TimeoutStopSec=300

[Install]
WantedBy=multi-user.target

Scheduled tasks need Celery beat, as its own service (exactly one instance):

# /etc/systemd/system/celery-beat.service
[Service]
User=app
WorkingDirectory=/opt/app
EnvironmentFile=/opt/app/.env
ExecStart=/opt/app/venv/bin/celery -A app beat --loglevel=info --schedule=/opt/app/celerybeat-schedule
Restart=always
sudo systemctl daemon-reload
sudo systemctl enable --now celery-worker celery-beat
journalctl -u celery-worker -f

See systemd unit files. With Docker Compose, run worker and beat as separate services using the same image as the web app. See Docker Compose in production.

Concurrency and memory

  • Default pool: prefork (processes). --concurrency ≈ number of CPU cores for CPU-bound tasks. It can be higher for I/O-bound tasks, or use --pool=gevent/threads.
  • Each worker process holds your app in memory. Watch RAM, and use --max-tasks-per-child=200 to recycle processes that leak memory. See how much RAM a VPS needs.
  • Use separate queues for slow and fast tasks (-Q default,emails), so a backlog of reports doesn’t delay password-reset emails.

Monitoring

  • Flower: a web UI for workers, tasks and queues (pip install flower, then celery -A app flower --port=5555). Keep it behind authentication or an SSH tunnel.
  • Queue length in Redis: redis-cli LLEN celery.
  • Alert when queues grow, or workers stop. See alerting best practices.

Deploys

Restart workers after deploying new code (systemctl restart celery-worker). With task_acks_late=True, tasks interrupted by the restart are re-delivered, so make tasks idempotent (safe to run twice).

Quick reference

Task Command / setting
Broker redis://127.0.0.1:6379/0
Start worker celery -A app worker -c 4
Scheduler celery -A app beat (one instance)
Queue a task task.delay(args)
Retries autoretry_for, retry_backoff
Monitor Flower, redis-cli LLEN celery

Related: deploy Django with Gunicorn and Nginx and Django production settings.

  • Python
  • Redis
  • Background Jobs