2,880
polls a day to catch 8 orders
That’s one merchant polling every 30 s. Assume 10,000 merchants and it’s 28.8 million requests a day to learn of 80,000 orders. Webhooks send 80,000 POSTs.
Is the POST the hard part?
Bad: POST after commit
A crash in between loses it; a slow endpoint slows checkout.
Good: Outbox, then a pipeline
Commit the event with the order; a relay queues it per endpoint.
One queue per endpoint, so a hung shop waits alone
With one shared queue, the slowest receiver sets everyone’s pace. With a lane each, Shop C’s backlog grows in its own queue, capped at 2 workers.
Nobody home? Retry for days, with widening gaps
Back off after each failure and add jitter, so thousands of failures don’t return in the same second. When retries run out, dead-letter the event for replay.
Inside the request: four quick steps
- Verify the HMAC over the raw bytes
- Reject timestamps more than 5 minutes old
- Store the event ID under a unique key
- Reply 2xx; do the real work later
Delivery is at least once and unordered: payment.refunded can land before payment.succeeded. Some events never arrive.
The trick
Treat each event as a hint: re-fetch the object, apply only if newer, and reconcile on a schedule.
Webhooks win between companies. Elsewhere, pick another pipe.
Inside one company, prefer a queue or log. For a user watching a screen, SSE or WebSockets. With no public HTTPS endpoint, poll.
Remember
Webhooks are at least once and unordered. Send with an outbox, a queue per endpoint and retries for days. Receive by verifying, storing the ID and replying fast; then re-fetch and reconcile.