Redis Streams are an append-only log data type built into Redis (since 5.0). Producers add entries, and consumers read them. Unlike Pub/Sub, entries are stored, so consumers that were offline can catch up. Consumer groups spread work across several workers, with acknowledgements and redelivery. For many apps, that’s a lightweight alternative to Kafka or RabbitMQ, using the Redis you already run.

Pub/Sub vs Streams vs Lists

Pub/Sub Lists (LPUSH/BRPOP) Streams
Messages stored No: lost if no one listens Yes, until popped Yes, until trimmed
Multiple consumers of the same message Yes (fan-out) No Yes (multiple groups)
Load balancing between workers No Yes Yes (consumer groups)
Acknowledgement and redelivery No Manual patterns Built in
Replay history No No Yes

See Redis Pub/Sub.

Add entries

redis-cli XADD orders '*' order_id 1001 user 42 total 19.50
# "1730803200000-0"     ← auto-generated ID: milliseconds-sequence

Each entry is a small hash of fields. Cap the stream’s size as you add:

redis-cli XADD orders MAXLEN '~' 100000 '*' order_id 1002 user 7 total 5.00

~ trims approximately, which is much cheaper than an exact cap.

Read entries

redis-cli XRANGE orders - +            # everything (careful on big streams)
redis-cli XRANGE orders - + COUNT 10
redis-cli XLEN orders
redis-cli XREAD COUNT 10 BLOCK 5000 STREAMS orders '$'    # wait up to 5 s for new entries

XREAD is simple fan-out: each reader tracks its own position (the last ID it saw).

Consumer groups

A group delivers each entry to one consumer in the group, and tracks what’s been acknowledged:

redis-cli XGROUP CREATE orders billing '$' MKSTREAM     # '$' = only new entries; '0' = from the start
redis-cli XREADGROUP GROUP billing worker-1 COUNT 10 BLOCK 5000 STREAMS orders '>'
redis-cli XACK orders billing 1730803200000-0

> means “entries never delivered to this group”. Several groups (billing, emails, analytics) can each process the full stream independently.

A Python worker

import redis
r = redis.Redis(decode_responses=True)
STREAM, GROUP, NAME = "orders", "billing", "worker-1"

try:
    r.xgroup_create(STREAM, GROUP, id="0", mkstream=True)
except redis.ResponseError:
    pass                                  # group already exists

while True:
    resp = r.xreadgroup(GROUP, NAME, {STREAM: ">"}, count=10, block=5000)
    for _, entries in resp or []:
        for entry_id, fields in entries:
            process(fields)               # your logic; make it idempotent
            r.xack(STREAM, GROUP, entry_id)

Run several workers with different names, to process in parallel. See background jobs and queues.

Handle failures: pending entries

Delivered but unacknowledged entries stay in the group’s pending entries list (PEL). If a worker crashes, its entries wait there:

redis-cli XPENDING orders billing                       # summary
redis-cli XPENDING orders billing - + 10                # details: IDs, owner, idle time, delivery count
redis-cli XAUTOCLAIM orders billing worker-2 60000 0-0 COUNT 10   # take entries idle > 60 s

Run XAUTOCLAIM periodically in workers, to recover stuck entries. Use the delivery count to detect “poison” messages that always fail, and move them to a dead-letter stream instead of retrying forever.

Because of redelivery, processing is at least once: make handlers idempotent (for example, check whether order 1001 was already billed).

Persistence and memory

Streams live in memory like all Redis data. Enable AOF persistence if entries must survive restarts, and trim streams with MAXLEN or MINID (by age) so memory stays bounded:

redis-cli XTRIM orders MINID '~' 1730000000000     # drop entries older than this timestamp

See Redis persistence: RDB and AOF and Redis maxmemory and eviction. Don’t use an eviction policy that could delete your streams. Run queues on a Redis instance with noeviction.

When to use something else

  • Huge throughput, long retention, many independent consumers, stream processing: Kafka (or Redpanda).
  • Complex routing, per-message TTLs, priority queues: RabbitMQ.
  • Simple background jobs in a framework: libraries like BullMQ (Node), Sidekiq (Ruby), RQ or Celery (Python), which use Redis lists or streams internally.

Quick reference

Task Command
Append XADD s MAXLEN ~ N * field value
Read range XRANGE s - + COUNT n
Wait for new XREAD BLOCK ms STREAMS s $
Create group XGROUP CREATE s g $ MKSTREAM
Consume XREADGROUP GROUP g c STREAMS s >
Acknowledge XACK s g id
Stuck entries XPENDING, XAUTOCLAIM
Trim XTRIM s MAXLEN ~ N / MINID

Related: Redis CLI commands and installing Redis on Ubuntu.

  • Redis
  • Messaging
  • Database