Delivery guarantees: at-least-once, exactly-once and DLQs

A delivery guarantee says what can happen to a message when something crashes: at-most-once may lose it, at-least-once may deliver it twice, and exactly-once promises a single effect. Each hop picks its guarantee with one decision: acknowledge before or after the work, and retry or not after a lost acknowledgement. Exactly-once delivery is impossible over a lossy network, but exactly-once effect is routine, and a dead-letter queue catches the messages that will never succeed.

Picture it first: paying the kirana shop by QR code

You scan the QR code at the kirana counter, enter ₹450 and tap Pay. The spinner turns, then: “Something went wrong.” Did the money leave your account?

• Walk away, and if the payment really failed, the shopkeeper is never paid: at-most-once.
• Pay again, and if the first attempt went through, you have paid ₹900: at-least-once.

You can’t tell which, because what got lost was the answer, not necessarily the payment. The fix is a reference number: if the retry carries the same transaction reference and the bank refuses a second debit with a reference it has seen, you can retry freely and pay once. The request arrived twice; the effect happened once.

The shop’s ledger may hear about one payment twice, so it must not count a reference twice. And a payment the bank can never settle, say to a closed account, lands on a manual-review desk after a few retries, which helps only if someone clears it daily.

Map it back: you are the producer, the bank the broker, the vanished success screen a lost ack. The reference is an idempotent producer at the bank and an idempotent consumer at the shop. The review desk is a dead-letter queue.

Three guarantees, decided at every hop

Kafka’s documentation states the three precisely:

• At-most-once: messages may be lost but are never redelivered.
• At-least-once: messages are never lost but may be redelivered.
• Exactly-once: each message is processed once and only once.

A message crosses three hops: producer to broker, broker to consumer, and consumer to whatever it changes (a database row, an email, a payment gateway). Each hop chooses separately, and end to end you get the weakest.

Why exactly-once delivery can’t exist

Two generals on two hills must agree on an attack time, and their messengers cross a valley where they may be captured. A sends “attack at dawn”; B replies “agreed” but can’t know the reply arrived, so A must confirm the confirmation, and so on forever.

A producer and a broker are those generals. The last ack can always be lost, and to the sender a lost message, a lost ack and a slow broker look identical, so it must resend (risking two copies) or not (risking none).

What you can build is exactly-once effect (also called effectively-once): deliver at least once, then make a second delivery change nothing. Every “exactly-once” feature below uses one of three tricks:

1. Naturally idempotent operations: an upsert by primary key, or “set status to PAID”.
2. Remember what you did: record the message ID in the same atomic step as the effect.
3. Commit the effect and the progress marker together, so both happen or neither does.

Consumer side: ack before or after the work

On the consumer, the guarantee is one line of code: where the acknowledgement (SQS DeleteMessage, RabbitMQ basic.ack, a Kafka offset commit) sits relative to the work.

• Ack first, then work: at-most-once. If the process dies mid-work, the broker has already forgotten the message. RabbitMQ’s automatic-ack mode does exactly this, and its docs call it unsafe.
• Work first, then ack: at-least-once. If the process dies after the work but before the ack, the message comes back (lease expiry, closed channel, or a restart from the last committed offset) and the work runs again.

At-least-once is the right default: a duplicate can be made harmless, a lost message can’t be found. The duplicate burst after a crash is everything done since the last ack. Kafka commits a position, and with enable.auto.commit=true (the default) it commits every 5 seconds, so a consumer handling 2,000 records a second can replay up to 2,000 × 5 = 10,000 records. Committing more often shrinks the burst; it never removes it.

Auto-commit hides a second trap: it fires on a timer, not when your code finishes. Hand records to a thread pool and a commit can cover work still in progress, so a crash loses it: at-most-once by accident.

Duplicates also arrive without a crash: a job that outlives its visibility timeout or ack deadline goes to a second worker (see message queues on leases).

Producer side: durable acks and the lost ack

A producer that gets an ack can stop worrying, if the ack means the message is durable.

RabbitMQ publisher confirms. In confirm mode the broker sends basic.ack only once every target queue has accepted the message: on disk for persistent messages on durable queues, on a quorum of replicas for quorum queues. Without confirms, a broker restart can lose messages the publisher thought were sent. Confirms batch disk syncs, so an ack can take a few hundred milliseconds; publish asynchronously and track outstanding confirms.

Kafka acks and min.insync.replicas. acks=1 waits for the partition leader only, so a leader crash before followers copy the record loses it. acks=all waits for every in-sync replica, but that set can shrink to the leader alone. With replication factor 3 and min.insync.replicas=2, every acked write is on at least two brokers, and with two brokers down the partition refuses writes (NotEnoughReplicas). You gain durability; you pay a replication round trip per write (“on the order of 10 ms”, per Kafka’s docs) and lose writes when 2 of 3 replicas are out. min.insync.replicas defaults to 1, so set it.

Then comes the lost ack. When a send times out, the producer can’t tell whether the broker wrote the record. Retry and you may append it twice; don’t, and you may lose it.

The idempotent producer

Kafka 0.11 fixed the retry half. The broker assigns each producer a producer ID, the producer numbers its batches per partition with a sequence number, and the broker keeps the latest sequence numbers in the replicated log, so a new leader knows them too. A retried batch with an already-written sequence number is acknowledged and dropped.

Since Kafka 3.0 (KIP-679) the Java producer defaults to enable.idempotence=true and acks=all (a bug kept idempotence off until 3.0.1, 3.1.1 and 3.2.0). It needs acks=all, retries above 0 and at most 5 in-flight requests per connection, because the broker keeps only the last 5 batches per producer. A conflicting setting (say acks=1) silently turns idempotence off unless you enabled it explicitly, in which case the producer refuses to start. So set enable.idempotence=true yourself.

What it does not cover:

• Application retries. If your code catches an error and calls send() again, that is a new record with a new sequence number.
• Restarts. A restarted producer gets a new producer ID; a transactional.id carries identity across restarts.
• Giving up. Internal retries stop at delivery.timeout.ms (2 minutes by default).

Exactly-once inside Kafka: transactions

Take a consume-transform-produce job: read an order from orders, write a payment instruction to payments. If it crashes after writing the payment but before committing its input offset, the restart writes the payment again; idempotent producers can’t help.

Kafka’s trick is that a committed offset is itself a record in an internal topic, __consumer_offsets, so one transaction can hold the output records and the input offset:

1. Register. initTransactions() registers the transactional.id with a transaction coordinator, which aborts any transaction a previous instance left open and bumps an epoch. A zombie still running with the old epoch is fenced: its writes are rejected.
2. Write. Begin a transaction, send the outputs, and call sendOffsetsToTransaction() with the input position.
3. Commit. The coordinator runs a two-phase commit: “prepare commit” in its replicated transaction log, then a commit marker in every partition the transaction touched.
4. Read. Consumers with isolation.level=read_committed see only committed records, reading up to the last stable offset (just before the first open transaction) and skipping aborted ones.

Configuration: transactional.id on the producer, isolation.level=read_committed and enable.auto.commit=false on the consumer. Kafka Streams wraps it all in processing.guarantee=exactly_once_v2.

The cost: Confluent measured about 3% lower throughput for 1 KB records committing every 100 ms. The overhead is per transaction, so bigger transactions are cheaper per record, but readers can’t see a transaction until it commits, so the commit interval adds straight to end-to-end latency.

What Kafka transactions don’t cover

• Anything outside Kafka. An email, a Postgres row or a payment-gateway call is not in the transaction. An abort hides the Kafka records, but the email has gone, and the retry sends another. Either store the input offset in the same database transaction as the output, or call the external system with an idempotency key derived from the input record.
• Consumers that don’t opt in. isolation.level defaults to read_uncommitted, which also returns aborted records.
• The first hop. Committing to your database and then publishing is a dual write: the transactional outbox’s job.
• Replays you ask for. Rewind a consumer group and the outputs are produced again.

What broker “exactly-once” features promise

Two managed services advertise exactly-once. Both are narrower than the name.

SQS FIFO deduplication. Each FIFO send carries a MessageDeduplicationId, or the queue derives one as a SHA-256 hash of the body (content-based deduplication). A repeat send with the same ID within the 5-minute deduplication interval is accepted but not delivered, so a producer can retry a lost ack safely, for 5 minutes; after that, SQS can’t detect the duplicate. FIFO does not stop a consumer that crashes before DeleteMessage from getting the message again after the visibility timeout, so handlers still need to be idempotent. And content-based dedupe merges two genuinely different events with identical bodies, such as two ₹100 top-ups a minute apart; put an event ID in the body.

Google Pub/Sub exactly-once delivery. Enabled per subscription, it promises no redelivery after a successful ack or while a message is outstanding; after an ack deadline expires, only the newest ack ID works. The fine print: pull subscriptions only (not push or export), and only while subscribers connect to one region. An ack can still fail, and then the message returns, so Google advises tracking processing progress. Pub/Sub removes the broker’s duplicates; a crash between your side effect and your ack is still yours.

Choosing a guarantee

Pick per message type, not per system. A rider’s location ping every 4 seconds can be lost, since the next one supersedes it; a refund must never be lost or applied twice.

Making at-least-once safe

The consumer side has its own page, idempotent consumers and the inbox pattern. In short: give every message a stable producer-set ID, then record it in a processed-messages table in the same database transaction as the effect, make the write conditional (UPDATE … WHERE version = 7), or pass it downstream as an idempotency key.

Retries without a storm

Retries turn transient failures into delays; badly placed, they turn one slow dependency into an outage.

Classify the error first. Timeouts, throttling, 503s and lock conflicts are transient: retry them. A schema violation or a deleted customer is permanent: retrying only delays the alarm, so dead-letter it at once (in RabbitMQ, reject with requeue=false).

Immediate vs delayed. Two or three quick in-process retries absorb a blip but hold the worker. For longer waits, hand the message back with a delay:

• SQS: ChangeMessageVisibility based on ApproximateReceiveCount. Starting at 30 seconds and doubling, 30 + 60 + 120 + 240 = 450 seconds: five attempts span 7.5 minutes.
• Pub/Sub: the default policy redelivers immediately; switch to exponential backoff (up to 600 seconds).
• RabbitMQ: quorum queues (4.3+) offer delayed retry with linear back-off; older setups bounce through a TTL queue.
• Kafka: a consumer group can’t delay one record without stalling its partition, so failures go to retry topics (orders-retry-1m, orders-retry-10m), then orders-dlq. You gain non-blocking retries; you lose per-key order.

The backoff and jitter arithmetic lives in timeouts, retries and backoff with jitter.

Retry at one layer. Retries multiply. Amazon’s Builders’ Library describes a call five services deep with three tries per layer: the database sees 3⁵ = 243 times the load once it starts failing. Queues add a forgotten layer: 3 SDK tries × 3 handler tries × 5 receives = 45 calls per message to a struggling dependency. Pick one layer (see retry storms).

Poison messages and dead-letter queues

A poison message fails every time: malformed JSON, an unknown schema, an input that hits a bug. Without a limit it loops forever, burning SQS receives, spinning a RabbitMQ requeue loop, or, in a Kafka consumer group, stopping the whole partition, since the consumer can’t commit past it. A dead-letter queue (DLQ) is the release valve: after N failed attempts, move the message aside so the line keeps moving, and keep it for a human.

Amazon SQS: redrive policy and redrive

A source queue’s redrive policy names the DLQ (same account and region) and a maxReceiveCount; once a message’s receive count passes it, SQS moves the message. A limit of 1 dead-letters after a single failed receive, so leave room for transient errors.

Retention is the subtle part. On a standard queue a dead-lettered message keeps its original enqueue timestamp. If the source keeps messages 4 days and a message spends 1 day there, a 4-day DLQ deletes it after 3 more days. So give the DLQ longer retention than the source; 14 days, the maximum, is a sane default. (FIFO queues reset the timestamp, and AWS warns that a FIFO DLQ breaks strict order.)

Redrive moves messages back to the source (or another queue of the same type) as a background task, StartMessageMoveTask, with a velocity cap of up to 500 messages per second, for at most 36 hours, without filtering or editing. Redriven messages get new IDs and interleave with new traffic. At 50 per second, a 120,000-message DLQ drains in 120,000 ÷ 50 = 2,400 seconds, 40 minutes, gentle enough for a dependency that has just recovered.

RabbitMQ: dead-letter exchanges

RabbitMQ republishes a message to the queue’s dead-letter exchange (DLX) when a consumer rejects it with requeue=false, its TTL expires, the queue overflows its length limit, or a quorum queue’s delivery limit is exceeded. Each dead letter gains an x-death header recording queue, reason and count. Three defaults to know:

• Quorum queues cap redeliveries at 20 (since 4.0); past it, the message is dropped unless a DLX exists. From 4.3 only failed deliveries (a basic.reject, a crashed consumer) count, not a basic.nack requeue.
• Dead-lettering is at-most-once by default: republished without confirms, so it can be lost if the target is unavailable. Quorum queues offer dead-letter-strategy: at-least-once.
• Missing pieces fail silently: if the DLX doesn’t exist, the message is dropped.

Kafka: no broker DLQ

Kafka keeps records whether or not anyone succeeds with them, so consumer groups have no broker-side DLQ; your consumer decides. Kafka Connect builds one into sink connectors: errors.tolerance=none (the default) stops the task at the first bad record, all skips bad records, and errors.deadletterqueue.topic.name copies them to a DLQ topic, optionally with __connect.errors.* headers naming the original topic, partition, offset and exception. Your own consumers end the retry-topic chain in a DLQ topic with the same context. Share groups, Kafka’s queue mode, stop redelivering a record after 5 attempts by default, but it stays in the log, so log its offset somewhere you will look.

Google Pub/Sub: dead-letter topics

A subscription forwards a message to a dead-letter topic after 5 to 100 delivery attempts (default 5). The count is approximate, and forwarding needs the Pub/Sub service account to have publish permission.

Operating a DLQ

A DLQ is an inbox for engineers. It earns its keep only with an owner, an alarm and a replay procedure.

1. The consumer fails on order 881 and lets it come back with growing delays.
2. After the fifth failure, SQS moves it on the next receive.
3. An alarm on the DLQ’s message count fires at anything above zero.
4. The on-call engineer reads the message and its error, fixes the bug or the data, and deploys.
5. Only then: redrive at a capped rate, and watch whether anything dead-letters again.

What to store with a dead letter. The broker keeps the payload; the context is your job: the original payload unchanged, its source (queue or topic, partition and offset, message ID), the error, the attempt count, first and last failure times, the consumer version and a trace ID. RabbitMQ’s x-death and Connect’s __connect.errors.* headers cover part of this; SQS never sees your exception, so log it with the message ID.

Alarm, own, triage. A healthy DLQ is nearly empty, so alarm on depth above 0 and on the age of the oldest message, well before retention deletes it, and give every DLQ an owning team. Sort dead letters into bugs (fix, then redrive), bad data (correct or discard), dependency outages (redrive after recovery) and expired work (discard).

Replay pitfalls.

• Order. Redriven messages interleave with new traffic, so an old “address = Pune” update can overwrite a newer “address = Mumbai” unless the consumer checks a version (message ordering).
• Duplicates. Some dead letters half-succeeded before failing; replay only into idempotent consumers.
• Staleness. A one-time password redriven three days late does harm. Drop messages past their deadline.
• Load and unfixed bugs. A redrive is a traffic spike at a dependency that may have just recovered, and a fix that didn’t work dead-letters everything again. Cap the rate and redrive a sample first.

Going deeper: what goes wrong

Infinite retry loops. No attempt cap (no redrive policy, a delivery limit of -1), a RabbitMQ consumer that nacks with requeue=true on every error, or a Kafka consumer retrying a bad record in place. One poison message eats a worker, or a whole partition.

The DLQ black hole. A DLQ nobody watches is a slow delete: standard SQS dead letters expire on the source clock, and Connect’s errors.tolerance=all without a DLQ topic skips bad records with nothing logged by default.

Dead-lettering an outage. DLQs are for broken messages, not for dependencies that are down. Say a payment gateway is down for 10 minutes at 1,000 messages a second, with a fixed 30-second visibility timeout and maxReceiveCount=5: each message is moved about 150 seconds after its first receive. Everything first received in the first 600 − 150 = 450 seconds is dead-lettered before the gateway returns: 1,000 × 450 = 450,000 good messages. For dependency errors, back off much longer, pause consumption (backpressure) or trip a circuit breaker.

Cheat sheet

How to say it in the interview

“Each hop picks between loss and duplicates. On the consumer I ack after the work, so delivery is at-least-once; on the producer I use acks=all, min.insync.replicas of 2 and the idempotent producer. Exactly-once delivery is impossible over a lossy network, so I aim for exactly-once effect: the payment consumer records each message ID in the same database transaction as the payment, so a redelivery is a no-op. Kafka to Kafka, I’d use transactions with read_committed consumers. Failures get a few retries with backoff and jitter, then a dead-letter queue with an alarm, an owner, and a redrive after the fix ships.”

Can Kafka give you exactly-once end to end? Between Kafka topics, yes: transactions commit outputs and input offsets atomically, and read_committed consumers never see aborted output. A database write, email or payment call needs its own idempotency key, or the offset stored in the same database transaction as the result.

Why not set maxReceiveCount to 1? Most failures are transient, so a timeout would dead-letter good messages and bury the real poison. Three to five attempts with backoff over a few minutes is typical; permanent errors can skip straight to the DLQ.

How do you replay a DLQ safely? Fix the cause, redrive a small sample, then redrive at a capped rate into idempotent consumers that check versions, dropping messages past their deadline.

When is at-most-once the right choice? When a newer message supersedes the old one or a gap is cheap: metrics, telemetry, location pings. You save retry traffic and dedupe state, and accept occasional silent gaps.

Where to go next

• Effectively-once: idempotent consumers and the inbox pattern: dedupe tables and the inbox.
• Message queues: leases, visibility timeouts, share groups.
• Transactional outbox: the producer-side dual write.
• Idempotency: keys for APIs and payment gateways.
• Timeouts, retries and backoff with jitter: the retry arithmetic.
• Retry storm: retries that sink a recovering service.
• Message ordering: what retries and DLQs do to order.
• Kafka: partitions, replication and consumer groups.
• Design a webhook delivery service: retries and dead letters as a product.

References and further reading

1. Apache Kafka 4.2 documentation: Design (message delivery semantics, transactions, durability)kafka.apache.orgThe three semantics, ack-before vs ack-after on the consumer, the lost producer ack, offsets committed inside transactions, read_committed, acks and min.insync.replicas.
2. Apache Kafka 4.2 documentation: Producer configskafka.apache.orgenable.idempotence and its preconditions (acks=all, retries > 0, at most 5 in-flight requests), acks defaults, delivery.timeout.ms and transactional.id.
3. Transactions in Apache Kafka (Confluent blog)confluent.ioAtomic read-process-write, zombie fencing with transactional.id epochs, the coordinator’s two-phase commit and markers, the 3% overhead figure, and why external side effects aren’t covered.
4. Kafka Connect deep dive: error handling and dead letter queues (Confluent blog)confluent.ioerrors.tolerance (fail fast by default, or skip silently), a DLQ topic for sink connectors, and __connect.errors.* context headers.
5. Amazon SQS Developer Guide: Using dead-letter queuesdocs.aws.amazon.comRedrive policy and maxReceiveCount, the FIFO ordering warning, and why DLQ retention must exceed source retention for standard queues.
6. Amazon SQS Developer Guide: Configure a dead-letter queue redrivedocs.aws.amazon.comRedrive to the source or another queue, velocity up to 500 messages per second, 36-hour task limit, new message IDs, and interleaving with new traffic.
7. Amazon SQS API Reference: SendMessage (MessageDeduplicationId)docs.aws.amazon.comFIFO deduplication: the 5-minute interval, content-based SHA-256 of the body, and duplicates that slip through after the interval.
8. RabbitMQ documentation: Dead Letter Exchangesrabbitmq.comThe four dead-lettering triggers, x-death headers, cycle detection, delayed retry via TTL, and why classic dead-lettering can lose messages.
9. RabbitMQ documentation: Quorum queuesrabbitmq.comDelivery limit (20 by default since 4.0), at-least-once dead-lettering, delayed retry with linear back-off, and which events count as a failed delivery.
10. Google Cloud Pub/Sub: Exactly-once deliverycloud.google.comWhat the feature guarantees, its limits (pull subscriptions only, one region), ack IDs that expire, and why subscribers still track progress.