When a user clicks “Sign up”, they shouldn’t wait while your server sends a welcome email, resizes their avatar, and syncs them to a CRM. Background jobs move slow or unreliable work out of the web request: the app puts a job on a queue, responds immediately, and separate worker processes do the work.

What belongs in the background

  • Sending emails and notifications.
  • Image and video processing, PDF generation, imports and exports.
  • Calls to slow or flaky third-party APIs (payments webhooks follow-up, CRM sync).
  • Scheduled and periodic work (reports, cleanups).
  • Anything that takes more than a few hundred milliseconds, or can be retried.

The pieces

Piece Role Examples
Producer The web app, enqueuing jobs SendWelcomeEmail.dispatch(user)
Broker / queue Stores jobs until a worker takes them Redis, RabbitMQ, database table, SQS
Worker Long-running process executing jobs Laravel queue worker, Celery, Sidekiq, BullMQ
Scheduler Enqueues jobs on a schedule Laravel scheduler, Celery beat, cron

Brokers: Redis vs RabbitMQ vs the database

Broker Strengths Consider when
Redis Simple, fast, already used for caching Most web apps
RabbitMQ Advanced routing, acknowledgements, durable messaging Complex workflows, many services
Database queue No extra service Low volume, simple setups
Kafka / NATS Event streaming, high throughput Event-driven architectures at scale

See install Redis and Redis persistence. If jobs must survive a Redis restart, enable persistence (AOF).

Tools by language

Language Popular options
PHP / Laravel Laravel Queues (Redis/DB) + Horizon. See Laravel queues with Supervisor
Python / Django Celery, RQ, Dramatiq. See Celery workers
Node.js BullMQ (Redis)
Ruby / Rails Sidekiq, Solid Queue
Go Asynq, River

Reliability rules

  1. Idempotency: a job may run more than once (retries, worker crashes). Running it twice must be safe. Use unique keys, and check whether the work is already done.
  2. Retries with backoff: temporary failures (timeouts, rate limits) are retried with increasing delays. Cap the attempts.
  3. Dead-letter / failed jobs: jobs that keep failing go to a failed list for inspection, not into an endless loop.
  4. Small payloads: pass IDs, not whole objects. Load fresh data in the job.
  5. Timeouts: kill stuck jobs, so they don’t block workers forever.
  6. Separate queues for urgent and bulk work (password resets shouldn’t wait behind 10,000 newsletter emails).

Running workers on a VPS

Workers are long-running processes, and must restart automatically:

  • systemd units, or Supervisor, for each worker type. See systemd unit files.
  • Docker Compose: a worker service using the same image as the web app. See Docker Compose in production.
  • Restart workers on every deploy, so they run the new code.

Sizing: CPU-bound jobs → about one worker per core. I/O-bound jobs (APIs, email) → more concurrency. Watch memory, and recycle workers periodically if they leak. See how much RAM a VPS needs.

Monitoring

  • Queue length and age of the oldest job: growing queues mean workers can’t keep up.
  • Failure rate, and the failed-jobs list.
  • Dashboards: Laravel Horizon, Flower (Celery), Bull Board (BullMQ), RabbitMQ management UI.
  • Alert when the queue backlog or the oldest job age crosses a threshold. See alerting best practices.

Scheduled jobs

Use the framework’s scheduler (triggered by one cron entry, or a scheduler process), so schedules live in code. Run exactly one scheduler instance. See cron jobs for web apps.

Quick reference

Concept Practice
What to queue Slow, external, retryable work
Default broker Redis
Must-haves Idempotency, retries with backoff, failed-job handling
Run workers systemd/Supervisor or Compose service, restart on deploy
Watch Queue length, oldest job age, failures

Related: twelve-factor app and VPS for a SaaS startup.

  • Web Development
  • Redis
  • Architecture