Publish/subscribe (pub/sub)

Publish/subscribe (pub/sub) is a messaging pattern in which a producer publishes each message to a named topic, and the broker hands a copy to every subscription on that topic. The publisher doesn’t know who is listening, so you can add a consumer without touching it. Inside one subscription, workers share the load as on a queue.

Picture it first: a train running late

The evening express is 40 minutes late. The railway control room records that once, and many people act on it: display boards change, booked passengers get an SMS, the cab stand holds its drivers, the tea vendor delays the next batch, and the next station reshuffles platforms. The control room phones none of them. It announces the delay, and whoever cares picks it up.

The news travels two ways. The platform announcement reaches whoever is on the platform right then; a passenger outside buying samosas misses it, and nobody repeats it. The SMS waits in your inbox, even if your phone was off for an hour.

Then there is the porters’ stand. “Coach S4 needs a porter” reaches the stand once, and one porter takes the job.

Map it back:

• Control room = publisher; “train status” = topic; display boards, SMS service, cab stand = subscriptions, each with its own copy.
• A new seat-delivery food service starts listening: a new subscription, publisher untouched.
• Platform announcement = an ephemeral subscription (Redis Pub/Sub); SMS inbox = a durable one (a queue or log that keeps messages).
• Porters’ stand = one subscription with competing consumers.

Point-to-point vs publish/subscribe

A message queue is point-to-point: each message goes to one consumer and is then gone. That suits work (“resize this photo”) but not news (“order 881 was placed”), because billing, email and analytics each need every order.

Pub/sub splits delivery into two levels, with three words:

• Topic: the named channel the publisher writes to, such as order-events. The publisher knows only the topic.
• Subscription: an independent stream of the topic’s messages, with its own position and backlog. Every subscription gets every message unless it filters.
• Subscriber: a process reading from a subscription. Subscribers on the same subscription compete, so each message reaches one of them.

One publish becomes one copy per subscription, handled by one worker in each. Billing can run 12 workers and email 2; each team scales and fails on its own.

Products name the levels differently: a Kafka subscription is a consumer group, a RabbitMQ one is a queue bound to an exchange, and an SNS one usually points at an SQS queue. Redis Pub/Sub has no shared subscription: every client that runs SUBSCRIBE gets every message, so instances can’t split the work.

What decoupling buys you, and what it costs

When checkout calls billing, email and analytics directly, it must know every address, wait on every call and handle every failure, and a new consumer means changing checkout. Pub/sub removes that knowledge:

• Fan-out to independent teams. One “order placed” event drives kitchen notification, rider assignment, the customer SMS, loyalty points, fraud scoring and analytics.
• New consumers without touching the producer. The fraud team subscribes on Tuesday; checkout deploys nothing.
• Decoupling in space. The publisher doesn’t know how many subscribers exist or where they run.
• Decoupling in time. A durable subscriber can be down for a deploy and catch up afterwards.
• Fault isolation. A crashing subscriber affects neither the publisher nor the other subscribers.

You pay for each of these:

• Eventual consistency. Subscribers see the event later, at different times: the SMS may go out before loyalty points appear.
• No reply. The channel is one-way.
• A hidden flow. Fowler warns that logic spread across event handlers appears in no single codebase; often only the live system shows it.
• Duplicates and reordering, because most brokers deliver at least once.
• A bill that grows with every subscriber, worked out below.

See synchronous vs asynchronous communication for the wider trade-off and event-driven architecture for whole systems built this way.

Push or pull delivery

Once a subscription holds a message, it reaches the subscriber in one of two ways.

Push: the broker calls the subscriber. SNS pushes to HTTP/S endpoints, Lambda functions, SQS queues, email and SMS. A Google Pub/Sub push subscription sends each message as an HTTPS POST; a success status is the ack, anything else asks for redelivery. Push suits serverless subscribers and external webhooks, but it can flood the endpoint, so push brokers pace themselves:

• Pub/Sub grows a push window of concurrent requests while the endpoint acks quickly, only linearly beyond 3,000 outstanding messages, and repeated nacks pause the whole subscription for 100 ms to 60 seconds.
• SNS can cap an HTTP/S subscription with maxReceivesPerSecond. It retries 5xx and 429 responses for at most one hour, then discards the message unless that subscription has a dead-letter queue.

Pull: the subscriber asks for the next batch. Kafka consumers fetch from an offset, Pub/Sub pull clients request messages, and SQS workers behind SNS long-poll their queue. Kafka’s designers chose pull because a push broker sets the pace and overwhelms any consumer slower than the producer, while a pull consumer simply falls behind and catches up later. Pull also batches naturally. You pay by running consumers all the time.

Rule of thumb: push for low-volume, latency-sensitive or serverless subscribers; pull for high-volume or bursty consumers that control their own rate. AWS’s common shape uses both: SNS pushes into SQS, and workers pull from SQS. See push vs pull architecture.

Durable vs ephemeral subscriptions

An ephemeral subscription exists only while the subscriber is connected; nothing published while it is away is kept for it. Redis Pub/Sub is the textbook case, and its documentation calls delivery at-most-once: Redis sends a message once, and if the subscriber errors or disconnects, the message is lost for good. Redis also cuts off subscribers that read too slowly: by default, a Pub/Sub client whose output buffer passes 32 MB, or stays above 8 MB for 60 seconds, is disconnected and misses everything until it reconnects.

A durable subscription stores messages until they are acknowledged or retention runs out: an SQS queue behind SNS (up to 14 days), a Google Pub/Sub subscription (unacked messages kept 7 days by default), a Kafka consumer group (its offsets point into a log kept 7 days by default), a Redis Streams consumer group, or a durable RabbitMQ queue.

Ephemeral delivery is fine for signals that matter only now or that another path recovers: typing indicators, presence, cache-invalidation hints backed by a TTL, or relaying an already-saved chat message to the WebSocket servers holding live connections (see real-time updates).

Fan-out with a queue per subscriber

The most common production shape is a topic that only fans out, plus one durable queue per subscriber. On AWS that is SNS to SQS: SNS writes a copy into every subscribed queue, and each team’s workers pull from their own queue as competing consumers. The extra hop buys:

• Isolation. A slow subscriber’s backlog grows in its queue only.
• Retries per subscriber. A message stays in the analytics queue until analytics deletes it.
• Its own dead-letter queue, alarms and scaling, sized on its own backlog.

Three details matter in practice. SNS retries a delivery to SQS or Lambda up to 100,015 times over 23 days, so a briefly unavailable queue loses nothing. SNS wraps each message in a JSON envelope; turn on raw message delivery if workers want just the body. And SNS doesn’t charge for deliveries to SQS, but each queue’s receive and delete calls are billed, so six subscribers pay for six sets of requests.

Quotas are generous but real: 12,500,000 subscriptions per standard topic, 100 per FIFO topic, and a publish quota per account that varies by Region, 30,000 messages a second in us-east-1 but 1,500 in Mumbai by default. A sale-day stream of 5,000 a second in Mumbai needs an increase before the sale.

Kafka consumer groups and RabbitMQ exchanges

Kafka: pub/sub across groups, a queue within a group

A Kafka topic is a partitioned log whose records are not deleted when read. The design documentation states the rule: each partition is consumed by exactly one consumer within each subscribing consumer group at a time. So:

• Across groups it is pub/sub. Every group reads every record and keeps its own offset in every partition. A new group costs the producer nothing.
• Within a group it is a queue. Partitions are split among members, so each record is processed once per group, and the partition count caps the busy members: with 4 partitions, a fifth consumer idles.

Extra groups are cheap for brokers too: Kafka stores each record once per replica however many groups read it, and serves caught-up groups from the operating system’s page cache, so they cause almost no disk reads. Each group still adds network egress.

RabbitMQ: exchanges, bindings and routing keys

In RabbitMQ’s AMQP 0-9-1 model, publishers never write to queues. They publish to an exchange, which copies each message into zero or more queues according to bindings. Each subscribing service declares and binds its own queue; within that queue, consumers compete. The exchange type decides how bindings match:

• Fanout: every bound queue gets a copy; the routing key is ignored.
• Direct: a queue matches if its binding key equals the message’s routing key.
• Topic: routing keys are dot-separated words, and binding keys can use wildcards: * matches exactly one word and # matches zero or more.
• Headers: matches header values instead, with x-match set to all or any.

A message that matches no binding is dropped or returned to the publisher, depending on flags it set, so a typo in a binding key quietly loses messages. And with thousands of subscribers, a full queue each gets heavy; RabbitMQ streams, append-only logs that many consumers read without deleting, exist for large fan-outs and replay.

Filtering and the cost of fan-out

Most subscribers want only part of a topic. Split topics, or let the broker filter per subscription:

• SNS filter policies match message attributes or a JSON body: exact values, prefixes, numeric ranges, anything-but and more. A topic takes up to 200 policies, and a change can take 15 minutes to apply.
• Google Pub/Sub filters match attributes only, up to 256 bytes, and are fixed at creation. Non-matching messages are auto-acknowledged, and you still pay delivery fees for them, though not egress.
• RabbitMQ filters through binding keys or headers.
• Kafka has no broker-side filtering. Consumers drop what they don’t need, or a stream processor writes a narrower topic (orders.high-value). In Kafka, topic boundaries are your filters.

The fan-out bill

One publish × N subscriptions = N deliveries, and each delivery costs bandwidth, consumer compute and, on SNS to SQS, a stored copy. Take a food-delivery order-events topic at a dinner peak of 5,000 events a second, 2 KB each, with 6 subscriptions:

• Deliveries: 5,000 × 6 = 30,000 a second.
• Bandwidth: 5,000 × 2 KB = 10 MB/s published; 30,000 × 2 KB = 60 MB/s delivered.
• Per 30-day month: 60 MB/s × 2,592,000 s ≈ 155.5 TB ≈ 141 TiB delivered. Google Pub/Sub bills throughput on publish and on every subscription’s reads, $40 per TiB at the time of writing, so reads cost about $5,700 a month, six times the roughly $940 for publishing.
• On Kafka, brokers take in 10 MB/s and serve 60 MB/s, but storage ignores the group count: 10 MB/s × 604,800 s (7 days) ≈ 6 TB, × 3 replicas ≈ 18 TB.

Filtering only saves deliveries where the broker does it. A fraud subscription wanting the 2% of orders above ₹5,000 receives 5,000 × 0.02 = 100 a second with an SNS filter policy, but all 5,000 if it filters inside a Kafka consumer.

Designing topics and events

A topic is a public contract between teams, so design it like an API.

Granularity. One topic per domain event (order-placed, order-cancelled) lets subscribers pick exactly what they need; one per entity (orders, with an event-type attribute) keeps the entity’s history together. Ordering decides it: order holds only within one topic’s partition or ordering key, so if “cancelled” must never overtake “placed” for the same order, both belong in an orders topic keyed by order ID. Split when volume, retention or access rules differ sharply. As Azure’s guidance puts it, broad topics push filtering onto subscribers; narrow ones multiply topics to manage.

Naming. Enforce a convention such as <domain>.<entity>.<version>, for example payments.refunds.v1. Redis Pub/Sub channels ignore database numbers, so its docs suggest an environment prefix to keep staging traffic out of production.

Schemas. Publish with a schema (Avro, Protobuf or JSON Schema) in a schema registry, which rejects incompatible versions before subscribers see them. Confluent’s registry defaults to backward compatibility: consumers on the new schema can read data written with the previous one. Add optional fields freely; for a breaking change, publish orders.v2 alongside orders.v1 until every subscriber has moved. See serialization and schema evolution.

Thin or fat events. Martin Fowler’s 2017 article separates two styles:

• Event notification carries little more than an ID, and subscribers call back for details. Messages stay small, but 6 subscribers × 5,000 events a second = 30,000 extra reads a second on the order service, and subscribers stall when it is down.
• Event-carried state transfer carries the changed data, so subscribers keep their own copy. You gain resilience and latency, and pay with bigger messages, many copies and more work per subscriber.

Either way, include an event ID for deduplication and a correlation ID for tracing. And beware Fowler’s passive-aggressive command: an “event” that really expects one service to act should be an explicit command on a queue.

Retention, replay and late subscribers

Durable subscriptions survive downtime; retention decides whether a subscriber can go further back, or start from the past.

• Kafka keeps records for the topic’s retention, 168 hours (7 days) by default, read or not; replay means resetting a group’s offsets. A new group follows auto.offset.reset, which defaults to latest, so it sees only new records. Set earliest to backfill what is retained.
• Google Pub/Sub gives a new subscription only messages published after it was created. With topic retention (up to 31 days) or retained acked messages, you can seek back to a timestamp. A snapshot captures a subscription’s ack state, typically just before a deploy, so you can seek back if new code acks what it shouldn’t; it lives 7 days minus the age of the oldest unacked message.
• SNS standard topics keep nothing, so a new queue starts empty; SNS FIFO topics can archive messages for 1 to 365 days and replay them.
• Redis Pub/Sub keeps nothing, ever.

Backfill beyond retention needs a second path: bulk-load current state from the owner’s database or a compacted topic, then consume the stream from a point before the load began, with idempotent upserts absorbing the overlap. Change data capture uses the same snapshot-then-stream approach.

Ordering and delivery guarantees, briefly

Pub/sub inherits the broker’s guarantees, and most are at-least-once (SNS to standard SQS, Google Pub/Sub by default, Kafka committing after processing), so handlers must be idempotent. Order holds per key at best: per Kafka partition, per Pub/Sub ordering key, per SNS FIFO message group. There is no order across subscriptions: email may handle order 881 a minute before analytics does. See delivery guarantees and DLQs and message ordering.

Going deeper: failure modes

The slow or failed subscriber

With a queue or log per subscription, a slow subscriber hurts only itself, at first. Its real deadline is retention. An SQS queue drops messages older than its retention period. A Kafka group that falls further behind than retention finds its offset gone; auto.offset.reset then applies, and with the default latest the group jumps to the end and silently skips everything between. Alert on each subscription’s lag or oldest-message age, well before retention.

Push and ephemeral systems fail differently: SNS discards an HTTP/S delivery after at most an hour of retries unless the subscription has a DLQ, Pub/Sub push backs off the whole subscription, and Redis disconnects the slow client.

Poison messages, per subscription

A message one subscriber can’t process may be fine for the other five, so dead-lettering is per subscription: SNS attaches its DLQ to the subscription, not the topic, and Pub/Sub sets dead-letter topics per subscription. On Kafka, a record that keeps failing blocks its partition for that group only, until the consumer parks it on a dead-letter topic.

Subscription sprawl

Subscriptions are easy to create and forget. An abandoned SQS queue keeps receiving copies; an abandoned Kafka group shows huge lag on every dashboard. Pub/Sub deletes a subscription after 31 days without subscriber activity by default, backlog included. Tag every subscription with an owner and prune the dead ones.

Unbounded fan-out

Pub/sub assumes a modest number of subscriptions per topic. One per user, chat room or device hits limits (10,000 per Pub/Sub topic, 100 per SNS FIFO topic) and multiplies every publish. In Redis Cluster, a classic PUBLISH goes to every node; sharded Pub/Sub (SPUBLISH, Redis 7.0) keeps a channel within one shard. For millions of recipients, publish once and fan out in the application, as in design a news feed and design a notification system.

Choosing a system, or not using pub/sub

Pub/sub is the wrong tool when:

• You need a reply. Channels are one-way; use a direct call or request-reply.
• Exactly one handler should act. “Charge this card” is a command for a queue, not a broadcast.
• Steps form a workflow that needs step 1’s result and compensation on failure. Use orchestration with a workflow engine (see multi-step workflows).
• The work must be atomic across publisher and consumers; use a database transaction or a saga.
• There are only one or two consumers, where Azure’s guidance notes a broker adds complexity without any scaling benefit.

Publishing reliably is a separate problem: if checkout commits the order and crashes before publishing, nobody hears of it. The transactional outbox closes that gap.

Cheat sheet

How to say it in the interview

“Checkout writes the order and publishes one OrderPlaced event to an order-events topic. Kitchen, rider assignment, notifications, loyalty and analytics each own a subscription, so each sees every event, and inside a subscription the instances compete, so each event is handled once per service. Subscriptions are durable: on AWS, SNS fanning out to an SQS queue per service; on Kafka, a consumer group per service, keyed by order ID for per-order ordering. A slow analytics job grows only its own backlog, and I alert on lag per subscription. Delivery is at-least-once, so handlers dedupe on the event ID. Five thousand events a second times six is 30,000 deliveries a second, so I filter at the broker where I can.”

How do you stop one slow subscriber hurting the others? Give each subscriber its own durable buffer: a queue per subscription or its own Kafka consumer group, so only its backlog grows. Then alert on its lag well before retention runs out, because a Kafka group that falls past retention resets to latest by default and silently skips ahead.

A new service needs last month’s orders. How? If retention covers it, replay: a new Kafka group with auto.offset.reset=earliest, or seek a Pub/Sub subscription back on a retaining topic. If not, bulk-load current state from the owning service, then consume from a point before the load, with idempotent upserts absorbing the overlap.

Kafka or SNS with SQS? SNS with SQS on AWS at moderate volume, when you want per-subscriber filtering, retries and DLQs with no servers to run. Kafka for high volume, replay, long retention, or events that also feed stream processing. Kafka has no broker-side filtering, so topic design matters more.

Should events carry the data or just an ID? An ID keeps messages small, but six subscribers at 5,000 events a second make 30,000 callback reads a second on the source and stall when it’s down. Carrying the state avoids both, at the cost of bigger messages and copies of the data.

When would you not use pub/sub? When the caller needs a reply, when exactly one service must act (a command on a queue), when steps form a workflow needing compensation (orchestration), or when there are only one or two consumers.

Where to go next

• Message queues: point-to-point delivery, leases and competing consumers.
• Delivery guarantees and DLQs: at-least-once, exactly-once and poison messages.
• Message ordering: partition keys, FIFO groups and head-of-line blocking.
• Idempotent consumers and the inbox pattern: making duplicate deliveries harmless.
• Transactional outbox: publishing an event safely with a database write.
• Event-driven architecture: whole systems built on events.
• Kafka: partitions, replication and consumer groups in depth.
• Real-time updates: pub/sub behind WebSockets and live feeds.
• Webhooks: push delivery to endpoints outside your system.

References and further reading

1. Azure Architecture Center: Publisher-Subscriber patternlearn.microsoft.comWhy senders shouldn’t know their consumers, topic granularity, schema versioning through topic names, and when pub/sub is unsuitable.
2. Amazon SNS Developer Guide: Subscription filter policiesdocs.aws.amazon.comFilter policies on message attributes or JSON body, the matching rules, and the up-to-15-minute delay before a policy change takes effect.
3. Amazon SNS Developer Guide: Message delivery retriesdocs.aws.amazon.com100,015 attempts over 23 days to SQS and Lambda, the one-hour cap for HTTP/S policies, maxReceivesPerSecond, and discard unless a DLQ is attached.
4. Amazon SNS endpoints and quotasdocs.aws.amazon.com12,500,000 subscriptions per standard topic, 100 per FIFO topic, 200 filter policies per topic, and per-Region publish quotas.
5. Google Cloud Pub/Sub: Subscription overviewcloud.google.comPull, push and export subscriptions; at-least-once by default; only messages published after a subscription exists; 31-day inactivity expiry.
6. Google Cloud Pub/Sub: Replay and purge messagescloud.google.comSeek to a timestamp or snapshot, snapshot lifetime of 7 days minus the oldest unacked message’s age, and seeking on filtered subscriptions.
7. Redis documentation: Pub/Subredis.ioAt-most-once delivery, pattern subscriptions, database-independent channels, and sharded Pub/Sub in Redis 7.0.
8. Apache Kafka 4.2 documentation: Designkafka.apache.orgWhy consumers pull, one consumer per partition per group, offsets as positions, and page-cache reuse when many consumers read one topic.
9. RabbitMQ: AMQP 0-9-1 Model Explainedrabbitmq.comExchanges, bindings and the direct, fanout, topic and headers exchange types; unroutable messages dropped or returned.
10. What do you mean by “Event-Driven”? (Martin Fowler, 2017)martinfowler.comEvent notification versus event-carried state transfer, hidden flows, and the passive-aggressive command trap.