31k
requests in flight at the peak
Healthy merchants reply in about 300 ms: 69k a second × 0.3 s ≈ 21k open. If 1% hang until the 15 s timeout, they add 10.5k.
Where does the event come from?
Bad: Commit, then publish
A crash in between: a real payment, no webhook.
Good: Outbox row, same commit
CDC tails the log and publishes it. Committed means published, maybe twice.
Merchant C’s server hangs while it gets 1,000 deliveries a second.
Blocking workers would wait 15 s on each, 15,000 stuck. Give every endpoint its own lane and cap, and C only slows C.
An endpoint dies while 1,000 deliveries a second keep coming. Trying each one holds 15,000 sockets open.
The trick
Retry the endpoint, not the message. Park its backlog and send one probe a minute.
Anyone can POST to a merchant’s URL. Why trust yours?
Sign the event ID, a timestamp and the raw body with HMAC-SHA256. A captured request is useless five minutes later.
An attacker registers 169.254.169.254 as its webhook URL.
Your workers call it from inside your data centre. If they reach the cloud metadata service, the delivery log may hand over your credentials.
The fix: resolve, check, pin, isolate
- Resolve once; refuse private, loopback and mapped IPs
- Connect to that checked IP; never follow redirects
- Egress proxies in a subnet with no route inside
Remember
Commit the event with the payment. Give every endpoint its own lane, retry endpoints not messages, sign everything, never reach inside.