A dual write has a gap. A crash can land in it.
“Did the database commit?” and “did the broker accept it?” are two facts. You need one fact that decides both.
Retries or two-phase commit?
Bad: Retry the publish; use XA
Retries die with the process, and Kafka can’t join an XA transaction.
Good: One local transaction
Order and outbox row commit together, or not at all.
Write the event as a row. A relay publishes it later.
The request ends at COMMIT. If Kafka is down for an hour, orders still succeed and events wait in the outbox.
The relay remembers “last sent seq was 1,041” and next asks for seq > 1,041.
The trap
T1 took 1,040 but committed after 1,041. The watermark skipped it forever. Select by published_at IS NULL.
The relay can send twice. Consumers must dedupe.
It published e-71, then crashed before marking it, and sent it again. Payments checks the event id before charging ₹450.
Keep each order’s events in order
- Key messages by aggregate id
- One relay per key: a lock or a hash shard
- Never skip a failed row for that key
- Version events so consumers drop stale ones
518M
outbox rows a day, never cleaned up
6,000 rows a second × 86,400 s, about half a terabyte. Delete in the relay, drop hourly partitions, or insert-and-delete with log tailing.
Remember
Commit the change and its event together. The relay is at least once, so dedupe by event id and key by aggregate.