Every subscription gets every message; inside one, workers compete
Billing runs 12 workers and email runs 2. Each order is processed once per service.
Redis Pub/Sub for orders?
Bad: Ephemeral: Redis Pub/Sub
A subscriber restart silently loses everything published meanwhile.
Good: Durable subscriptions
A queue, a Kafka group or a Redis Stream keeps the backlog.
SNS to SQS: a durable queue per subscriber
Analytics falls behind, and only its own queue grows, to 40,000. Email and fraud never notice. Each has its own retries, DLQ and scaling.
6×
the deliveries, with six subscriptions
5,000 events a second at dinner peak become 30,000 deliveries and 60 MB/s. On Google Pub/Sub, reads cost about six times publishing.
Fraud wants only the 2% of orders above ₹5,000. Filter inside a Kafka consumer?
The catch
It still receives all 5,000 a second. An SNS filter policy delivers just 100.
Design a topic like an API
- Keep events that must stay ordered in one topic
- Name it domain.entity.version
- Register schemas; add fields, never rename
- Carry an event ID and a correlation ID
A Kafka group that falls behind retention jumps to the end
Its offset is gone, and the default auto.offset.reset = latest skips everything in between, silently. Alert on lag well before retention.
Remember
A copy per subscription, competing workers inside it. Make subscriptions durable, handlers idempotent, and alert on each one’s lag.