A message crosses three hops. Each picks loss or duplicates.
End to end, you get the weakest hop. And no broker setting reaches the third: your database write, email or payment call.
Where does the consumer ack?
Bad: Ack, then work
A crash mid-work loses it for good. At-most-once.
Good: Work, then ack
A crash means redelivery: at-least-once, the default. Duplicates can be made harmless; losses can’t.
Can any broker deliver each message exactly once?
The truth
Not over a lossy network, because the last ack can always be lost. Build exactly-once effect. Deliver at least once and make repeats change nothing.
Kafka’s fix for the lost ack: number every batch
Each producer gets an ID and numbers its batches per partition. A retry with a sequence number the broker already has is acked and dropped.
Every “exactly-once” feature has a boundary
- Idempotent producer: one producer session, one partition
- Kafka transactions: Kafka to Kafka, read_committed readers only
- SQS FIFO: producer retries within 5 minutes
- Pub/Sub: pull subscribers in one region
One poison message can stall a whole Kafka partition
It fails every time, so the consumer can’t commit past it. After a few attempts, move it to a dead-letter queue for a human.
450K
good messages dead-lettered in a 10-minute outage
Say 1,000 a second, a 30 s visibility timeout, maxReceiveCount=5. DLQs are for broken messages; for an outage, back off longer or pause.
Remember
Exactly-once delivery is impossible. Deliver at least once, make repeats change nothing, and dead-letter what never succeeds.