A message queue is a durable buffer between the code that creates work and the code that does it. A producer drops a message in and moves on; workers take messages out, and each message is handled by one worker, then deleted. You add one to decouple services in time, absorb traffic spikes and push slow or flaky work off the request path.
Picture it first: the ticket rail in a cloud kitchen
A cloud kitchen cooks for three delivery apps. Each order prints as a ticket and is clipped to a rail above the counter. The apps don’t wait on the phone for a cook; they get “order accepted” the moment the ticket is up.
Five cooks work the rail. Whoever is free takes the next ticket, clips it to their station and cooks; when the dish goes out, the ticket is spiked. No two cooks make the same order and nobody assigns work. At the 1 pm rush, orders arrive twice as fast as the cooks can manage: the rail grows, the cooks keep their pace, and by 2:30 it is empty again. To clear it faster, the owner calls in two more cooks, and the apps never notice.
One rule covers accidents: a ticket that has sat at a station for ten minutes without its dish going out goes back on the rail. Now and then that means two plates of the same biryani, so the packing desk checks the order number before sealing a bag.
Map it back:
- Ticket = message, rail = queue, cooks = competing consumers (workers).
- “Order accepted” at once = temporal decoupling; the rail growing and emptying = load levelling.
- The ten-minute rule = the lease (SQS calls it the visibility timeout). The second biryani is at-least-once delivery; checking the order number is an idempotent consumer.
- If billing and the kitchen both needed every ticket, one rail would not do. That is publish/subscribe.
Why put a queue between two services
Without a queue, A calls B and waits: A’s latency includes B’s, and A fails when B fails. A queue breaks that coupling five ways, each with a price.
- Temporal decoupling. Producer and worker need not be up together: during a 10-minute deploy of the invoice service, checkout keeps enqueuing. You pay with eventual results: “invoice on its way”, not an invoice.
- Load levelling. The queue soaks up a burst and workers drain it at their own pace, so you size for near-average load, not the peak (queue-based load levelling). You pay with waiting time during the burst.
- Background work. Resize the photo, send the OTP SMS, render the PDF later; answer the user in 50 ms. You pay with a second code path and a way to report the result.
- Retries. A failed message comes back and is tried again, so a flaky partner API becomes a delay, not an error page. You pay with duplicates and poison messages, which is what dead-letter queues are for.
- Independent scaling. The API scales on requests, the workers on backlog. You pay with one more system to run and watch.
Overall you trade immediacy for resilience: more latency, duplicates, weaker ordering and an asynchronous hop to debug (see synchronous vs asynchronous communication).
This is the competing consumers pattern. A slow message ties up one worker, not the queue, and a crashed worker is replaced by any other. The catch is order: with four workers in parallel, m2 can finish before m1 (see message ordering).
In the interview
Name the job the queue does, not just “add Kafka here”: “Checkout writes the order and returns. The invoice PDF and confirmation email go through a queue, so a 5× sale-day spike grows the backlog by a few minutes instead of timing out checkout.”
The life of a message
Every queue runs the same loop. SQS and RabbitMQ show the two common styles: a pull API with a timer, and a push connection with acknowledgements.
- Send, then store durably. The queue makes the message crash-proof before saying “ok”: SQS stores it across several Availability Zones; RabbitMQ’s publisher confirms ack once a persistent message is on disk, or on a quorum of replicas for quorum queues. Without confirms, a “sent” message can vanish in a broker restart.
- Receive, which takes a lease. The queue hands the message over but does not delete it. SQS hides it from other consumers for the visibility timeout (default 30 seconds, anything from 0 seconds to 12 hours). RabbitMQ pushes it down the worker’s channel and marks it unacked.
- Process it: charge the card, resize the photo.
- Acknowledge, which deletes. SQS needs
DeleteMessagewith the receipt handle; RabbitMQ needsbasic.ack. Only now is the message gone. - Or the lease ends and the message comes back. In SQS, the message becomes visible again when the timeout expires. RabbitMQ requeues every unacked delivery when the worker’s channel or connection closes (crash, lost TCP connection, protocol error) and flags it
redelivered. Its delivery acknowledgement timeout, 30 minutes by default and limited to quorum queues from version 4.3, closes the channel of a consumer that holds a message too long.
Why delivery is at-least-once
If a worker charges the card and crashes before deleting the message, the queue cannot tell that from a crash before the charge, so it redelivers. A queue that never loses messages must sometimes deliver twice; SQS says even the visibility timeout does not guarantee a single delivery. So handlers must be safe to run twice, using a processed-messages table, a conditional write or an idempotency key. Delivery guarantees and idempotent consumers go deeper.
Choosing the lease length
The visibility timeout is a bet on processing time. Too short, and a slow but healthy job reappears and a second worker starts it too. Too long, and a crashed job sits untouched: m1 above idles for 21 seconds, and with a 15-minute timeout it would idle for almost 15 minutes.
Set it a few times above p99 processing time and have long jobs heartbeat with ChangeMessageVisibility. SQS caps the total at 12 hours from the first receive, and extending does not reset that clock, so longer work belongs in a workflow engine. When Lambda consumes the queue, AWS suggests at least six times the function timeout.
Expired leases multiply overload
When a dependency slows, processing time crosses the timeout, the message goes to a second worker, then a third, and the fleet repeats work exactly when it has the least capacity. The Builders’ Library likens it to a fork bomb. Heartbeat the lease, or stop working once it has expired.
How workers get messages: polling and prefetch
Pull, with long polling (SQS). Workers call ReceiveMessage. The default short poll asks a random subset of SQS servers and answers at once, so it can come back empty even when messages exist. Set WaitTimeSeconds (up to 20 seconds) for a long poll: SQS checks all servers and replies as soon as a message arrives. An idle worker short-polling ten times a second makes 10 × 86,400 = 864,000 requests a day; long-polling, at most 86,400 ÷ 20 = 4,320, 200 times fewer, with no extra delay. Batches of up to 10 messages cut calls further.
Push, with prefetch (RabbitMQ). RabbitMQ pushes to long-lived consumers and discourages one-at-a-time basic.get polling. The knob is the prefetch count (basic.qos), the most unacked messages a consumer may hold. Prefetch 1 is fair but pays a round trip per message; unlimited (0) lets one worker hoard messages and memory while others idle. RabbitMQ suggests 100 to 300 for throughput; for jobs that take seconds, go much lower so work spreads evenly. Prefetch is a small form of backpressure.
Redis Streams (XREADGROUP … BLOCK) and Kafka (fetch from an offset) are pull-based too.
Limits that shape the design
Retention and size limits shape what a queue can hold. SQS, as of late 2026:
| SQS setting | Value | Consequence |
|---|---|---|
| Message size | Up to 1 MiB, raised from 256 KiB in August 2025 | Large payloads go to object storage |
| Retention | Default 4 days; 60 seconds to 14 days | A longer consumer outage silently loses work |
| Visibility timeout | Default 30 s; 0 s to 12 hours | Long jobs heartbeat or move to a workflow engine |
| Backlog | Unlimited | Watch age, not just depth |
| In flight | About 120,000 per queue | A bug that never deletes eventually blocks receives |
Retention is the quiet killer. A consumer that breaks at 6 pm on the Friday of a long weekend and is fixed at 9 am Tuesday has been down 87 hours, nine short of the 4-day default. Nothing errors; messages just expire. Use 14 days for real work and alarm on message age well before that.
Send a pointer, not the payload
Put big bodies (a video, a 5 MB CSV) in blob storage and enqueue only the key. This claim check pattern works on every broker; AWS’s extended client libraries for Java and Python do it automatically through S3, up to 2 GB.
Standard and FIFO queues
SQS standard queues take a nearly unlimited call rate, deliver at least once and order on a best-effort basis. FIFO queues keep strict order per message group and drop duplicate sends within 5 minutes, but cap each API action at 300 calls per second per partition (3,000 messages in batches of 10); high-throughput mode raises that to 70,000 calls per second in the largest regions and 9,000 in Mumbai. Default to standard with order-tolerant handlers; message ordering covers groups and head-of-line blocking.
Queue vs log: delete on ack, or keep and track offsets
Kafka gets called a queue, but it is a different design.
A classic queue (SQS, RabbitMQ) tracks every message (waiting, leased, acked) and deletes it on ack. So:
- Parallelism is free. Any free worker takes the next message; run 3 workers or 300.
- No replay. If a bug wrote wrong totals for two hours, those messages are gone.
- One set of readers. If billing and analytics need every order, each needs its own queue fed by a fan-out (SNS to SQS, a RabbitMQ fanout exchange): that is pub/sub.
A log (Kafka) splits a topic into partitions, each an append-only ordered sequence, and keeps records for a retention period (7 days by default) whether anyone read them or not. A consumer group’s progress in a partition is one integer, the offset of the next record, so acking is committing a number. So:
- Replay. Move the offset back and re-read after fixing the bug.
- Many readers. Each group keeps its own offsets; a new reader costs producers nothing.
- Parallelism is capped. Within a group, each partition has exactly one consumer at a time: 12 partitions, at most 12 busy consumers.
- Head-of-line blocking. A committed offset means “everything before this is done”, so one slow record holds up its partition, and there is no per-message lease or built-in dead-letter queue.
That cap bites for slow jobs: the fleet later on this page needs 300 jobs in progress at once, which means 300 partitions or consumers that juggle out-of-order offset commits.
Kafka now has queues too
Kafka 4.2 made share groups (KIP-932, “Queues for Kafka”) production-ready. Share consumers split a topic’s records, can outnumber partitions and ack each record. A fetched record is held under a lock (30 seconds by default) that works like a visibility timeout, and delivery attempts are counted (5 by default) to stop poison records. Records still stay in the log for replay.
Redis Streams sit in between. The stream is a log (entries stay until trimmed, several groups keep their own positions), but inside a group each entry goes to whichever consumer is ready, and a pending entries list holds delivered entries until XACK. There is no automatic lease: a dead consumer’s entries stay pending until another runs XAUTOCLAIM on entries idle past a threshold. Durability is Redis’s: asynchronous replication, so a failover can lose recent writes.
Load levelling: buffering a spike
A worker fleet processes at most 1,200 messages per second; normal traffic is 400. A flash sale sends 2,000 per second from minute 5 to minute 15.
- During the spike the queue grows at 2,000 − 1,200 = 800 per second, reaching 800 × 600 s = 480,000 messages.
- Afterwards it drains at 1,200 − 400 = 800 per second: 480,000 ÷ 800 = 600 seconds, empty by minute 25.
- The worst wait, for a message arriving at minute 15: 480,000 ÷ 1,200 = 400 seconds.
Without the queue you would reject 800 requests a second for ten minutes, or run a fleet 67% bigger (2,000 ÷ 1,200) all day for ten minutes of use. With it, you pay minutes of delay on work nobody was waiting for.
It only works if average arrivals stay below capacity. Here they total 400 × 300 + 2,000 × 600 + 400 × 900 = 1,680,000 messages in 1,800 seconds, 933 per second, under 1,200. If the average exceeds capacity, the queue never drains and only delays the failure; you need backpressure and load shedding instead.
Sizing consumers with Little’s law
Now size a fleet for a steady 1,200 messages per second. Little’s law says that in any stable system, L = λ × W: items inside equal the arrival rate times the time each spends there. For workers, L is messages in progress, λ the message rate, W the handler time.
At λ = 1,200 per second and W = 0.25 seconds, L = 1,200 × 0.25 = 300 messages in progress at any moment, so you need at least 300 handler slots.
- With 20 concurrent handlers per pod, 300 ÷ 20 = 15 pods, all 100% busy: any blip grows a backlog that never shrinks.
- For about 70% utilisation, 300 ÷ 0.7 ≈ 429 slots, so 22 pods (440 slots). They can process 440 ÷ 0.25 = 1,760 per second, leaving 560 spare for draining backlogs.
The trap is that W moves. If the database slows from 0.25 to 2 seconds, the same traffic needs 1,200 × 2 = 2,400 slots. The Builders’ Library makes the same point: 100 messages per second at 100 ms keeps 10 threads busy; at 10 seconds it needs 1,000.
With 440 slots at 2 seconds each, the fleet manages 220 per second, so the backlog grows by 980 every second. More pods help only if the slow database can take more load; Azure’s guidance warns that autoscaling consumers without capping their total rate just moves the overload downstream. Cap the autoscaler at what the dependency can absorb.
Scaling workers on depth and age
Depth (ApproximateNumberOfMessagesVisible in SQS) is what is waiting, but 10,000 messages means little without the fleet size. AWS recommends scaling on backlog per instance, targeting acceptable latency ÷ time per message. A pod above handles 20 ÷ 0.25 = 80 messages per second; if 30 seconds of delay is fine, target 80 × 30 = 2,400 per pod, so 48,000 waiting messages call for 20 pods.
Age of the oldest message (ApproximateAgeOfOldestMessage) is how late you are, which users feel. Alarm on it against the promise (a minute for an OTP SMS, an hour for a report), well below retention. Caveat: SQS standard queues push a message received three or more times to the back and leave such poison messages out of this metric.
On scale-in, let workers finish what they hold; a killed worker’s messages return only when their lease expires (autoscaling done right).
Going deeper: backlogs and other failure modes
The insurmountable backlog
The Builders’ Library calls queue systems bimodal: fast with no backlog, and painfully slow to recover once arrivals outrun processing. Give the 22-pod fleet a one-hour outage at 1,200 per second: 1,200 × 3,600 = 4,320,000 messages. With 560 per second spare, draining takes 4,320,000 ÷ 560 ≈ 7,700 seconds, over two hours, while every fresh message waits behind hours-old ones. Amazon’s rule of thumb: an hour-long outage needs double capacity for another hour. In its harsher example, an unthrottled tenant queues ten times the consumers’ capacity in the 30 minutes before an operator reacts: 300 minutes to recover.
The defences Amazon describes:
- Separate queues per workload or tenant, or hash tenants to a few queues and enqueue into the shortest (noisy neighbours).
- Sideline old or excess traffic into a spillover queue so fresh work goes first, and drop stale messages that a periodic full sync will redo.
- Push back on producers as the backlog grows, and keep spare polling capacity.
Stuck, poisoned and missing messages
- Stuck in flight. A handler that throws and skips the delete leaves the message leased until timeout. At scale that hits the in-flight ceiling: SQS short polls return
OverLimit, long polls return nothing, and the queue looks empty. Always delete, or release by setting the visibility timeout to 0. - Poison messages fail every time and burn a worker each round. Cap attempts and move them to a dead-letter queue.
- Never enqueued. A queue protects only what it acknowledged, so producers check the ack and retry with backoff and jitter. And if the service commits to its database and crashes before enqueuing, the job is lost; the transactional outbox closes that gap.
When not to use a queue
- The caller needs the answer now: login, “is this seat free?”, a price quote. Azure’s pattern rules out callers that need a low-latency synchronous response.
- The user must read their own write. If an address update goes through a queue, the reloaded page may show the old one. Write synchronously; queue only the side effects.
- Every consumer needs every message. Use publish/subscribe.
- Steps depend on each other. Use a workflow engine or a multi-step workflow.
- Load is low and steady, so a queue adds a moving part for no gain. Or you need history (audit, replay), which is a log’s job.
SQS, RabbitMQ, Kafka and Redis Streams as a work queue
All four can feed a pool of workers. They differ in how a message is leased, what an ack does, and what they cost to run.
| Amazon SQS | RabbitMQ | Apache Kafka | Redis Streams | |
|---|---|---|---|---|
| Delivery | Pull; receive takes a lease | Push to consumers, bounded by prefetch | Pull by offset, one consumer per partition per group | Pull with XREADGROUP into a pending list |
| Ack | DeleteMessage deletes it |
basic.ack deletes it |
Commit offset: all before it done (share groups: per record) | XACK clears it; entry stays in the stream |
| Redelivery | Lease expires | Channel closes, or ack timeout | Restart from last committed offset | XAUTOCLAIM on idle entries |
| Ordering | Best effort; FIFO per message group | Per queue, loosened by many consumers | Strict per partition | Per stream, loosened by many consumers |
| Retention | 4 days default, 14 max | Until acked | 7 days default, read or not | Until trimmed; memory-bound |
| Replay | No | No (classic and quorum queues) | Yes, rewind the offset | Yes, by ID range |
| Parallelism | Unlimited consumers | Unlimited consumers | Partition count per group (share groups lift it) | Unlimited consumers |
| Throughput | Standard nearly unlimited; FIFO 300 calls/s per partition | One leader per queue; add queues | Very high; add partitions | In-memory fast; one stream per shard |
| Operations | Fully managed | Run a cluster or buy managed | Heaviest: brokers, partitions, rebalances | Light if you run Redis; you write claim and DLQ logic |
Defaults: SQS for background jobs on AWS with zero operations; RabbitMQ for flexible routing through exchanges or on your own servers; Kafka when the events also feed analytics, need replay or arrive at very high volume (share groups now cover job-style use); Redis Streams for light jobs in a system already on Redis that can live with its durability. To build one yourself, see design a distributed message queue.
Cheat sheet
| Idea | Remember it as |
|---|---|
| Message queue | Durable buffer; each message handled by one consumer, then deleted |
| Why | Decoupling in time, load levelling, background work, retries, independent scaling |
| Cost | Latency, duplicates, weaker ordering, one more system |
| Lifecycle | Send → store → receive (lease) → process → ack; lease expires → redelivered |
| SQS lease | Visibility timeout 30 s default, up to 12 h; heartbeat long jobs |
| RabbitMQ | Manual acks; prefetch 100–300, lower for slow jobs; unacked requeue on channel close |
| Delivery | At-least-once, so handlers are idempotent |
| SQS limits | 1 MiB messages; 4-day default retention, 14 max; long-poll up to 20 s |
| Queue vs log | Delete on ack vs offsets per group: replay and many readers, parallelism capped by partitions |
| Little’s law | Slots = rate × time: 1,200/s × 0.25 s = 300 |
| Scale on | Backlog per worker; alarm on age of oldest message |
| Skip it when | The caller needs the answer now, or must read its own write |
How to say it in the interview
“I’d put a queue between the order API and the invoice and email workers. The API writes the order, enqueues a job and returns. Workers long-poll, take a message under a visibility timeout, process it and delete it; if a worker dies, the message reappears for another worker, so delivery is at-least-once and handlers are idempotent on the order ID. On sale day the queue absorbs the spike and drains afterwards. Little’s law sizes the fleet: 1,200 a second times 0.25 seconds is 300 in flight, so 22 pods of 20 slots for headroom. I’d autoscale on backlog per pod and alarm on the age of the oldest message. If analytics also needed these events, or we needed replay, I’d use Kafka.”
What if a worker crashes halfway through a message? Nothing is lost: it was never acked, so when the lease expires (SQS) or the channel closes (RabbitMQ) another worker gets it. If the crash came after the side effect, that is a duplicate, which is why the handler checks an idempotency key first.
How do you choose the visibility timeout? A few times p99 processing time, plus a heartbeat for long jobs. Too short and slow jobs run twice at once, worst under overload; too long and a crashed job idles for the whole timeout. Beyond SQS’s 12-hour cap, use a workflow engine.
How many consumers do you need? Concurrency = rate × time per message: 1,200 per second × 250 ms = 300. At about 70% utilisation that is 440 slots, 22 pods of 20. Then check the database can take 1,760 messages a second, and cap the autoscaler there.
Why not use Kafka for everything? A log shines at replay and many independent readers, but classic consumer groups cap parallelism at the partition count and stall a partition behind one slow record. For slow, independent jobs a queue (or Kafka’s share groups) fits better, and a managed queue is cheaper to run.
How do you stop a backlog becoming a long outage? Alarm on the age of the oldest message, keep spare capacity, isolate noisy tenants, sideline stale work so fresh messages go first, and push back on producers as the backlog grows.
Where to go next
- Publish/subscribe: every subscriber gets every message.
- Delivery guarantees and DLQs: ack timing, retries and poison messages.
- Idempotent consumers and the inbox pattern: making duplicates harmless.
- Message ordering: FIFO groups, partition keys, head-of-line blocking.
- Backpressure and load shedding: when average load exceeds capacity.
- Transactional outbox: enqueueing safely alongside a database write.
- Kafka: partitions, replication and consumer groups.
- Synchronous vs asynchronous communication: when to go async at all.
- Event-driven architecture: systems built around messages.
- Long-running jobs: queues plus status tracking.
References and further reading
- Amazon SQS Developer Guide: Visibility timeoutdocs.aws.amazon.comReceive hides a message for the visibility timeout (default 30 s, 12-hour cap from first receive); ChangeMessageVisibility heartbeats; in-flight limits.
- Amazon SQS Developer Guide: Message quotasdocs.aws.amazon.comCurrent limits: 1 MiB messages, retention 60 s to 14 days (default 4 days), batches of 10, FIFO throughput per API action.
- Amazon SQS Developer Guide: Short and long pollingdocs.aws.amazon.comShort polling samples a subset of servers; long polling waits up to 20 seconds and cuts empty and false-empty responses.
- Amazon SQS Developer Guide: Available CloudWatch metricsdocs.aws.amazon.comApproximateAgeOfOldestMessage, visible and in-flight message counts, and their caveats for poison messages.
- Amazon EC2 Auto Scaling User Guide: Scaling policy based on Amazon SQSdocs.aws.amazon.comScale workers on backlog per instance, with the target set to acceptable latency divided by processing time.
- RabbitMQ: Consumer Acknowledgements and Publisher Confirmsrabbitmq.comManual acks, prefetch (basic.qos) and its 100–300 sweet spot, and automatic requeue of unacked deliveries when a channel closes.
- Azure Architecture Center: Queue-Based Load Leveling patternlearn.microsoft.comA queue as a buffer between bursty tasks and a steady service, its costs, and when not to use it.
- Avoiding insurmountable queue backlogs (Amazon Builders’ Library)aws.amazon.comBimodal queue behaviour, backlog recovery arithmetic, age-of-message metrics, heartbeating leases and Little’s law for concurrency.
- Apache Kafka 4.2 documentation: Designkafka.apache.orgOne consumer per partition per group, offsets as the consumer position, rewinding to replay, and share groups (queues for Kafka).
- Redis documentation: Redis Streamsredis.ioConsumer groups, the pending entries list, XACK and XAUTOCLAIM, and the durability caveats of asynchronous replication.