Design a webhook delivery service

When a shopper pays ₹1,499 over UPI, the merchant’s server needs to know within seconds so it can confirm the order and pack the parcel. A payments gateway tells it with a webhook: an HTTPS POST from the gateway to a URL the merchant registered. One request is easy. This problem is about sending 750 million of them a day to 60,000 servers you do not own. Some answer in 50 ms, some take 14 seconds, some were switched off last Diwali, and a few point at your own internal network on purpose. You must never lose an event the payment system committed, never let one broken endpoint slow everyone else, retry for days without causing a storm, prove every request is genuine, and show merchants exactly what happened to each one. For the basics, read the concept article on webhooks; this page designs the system that sends them.

Understand the problem

The product is a payments gateway with about 40,000 merchants who use webhooks: D2C brands, food delivery apps, school-fee portals, SaaS billing. The gateway emits events such as payment.captured, payment.failed, refund.processed and settlement.processed. A webhook is a push API: you decide when to call, and the merchant’s server decides how it answers. Everything that normally protects a service — timeouts, retries, isolation, authentication — now has to work in the other direction, against receivers that are slow, flaky and sometimes hostile.

Functional requirements

1. Merchants should be able to register endpoints and choose event types. An HTTPS URL, the events it wants and the payload version it understands. A merchant may have a few endpoints: the orders service, the ERP, a chat bot.
2. Every event should reach every subscribed endpoint within seconds. A payment.captured committed at 20:00:01 should normally land by 20:00:02.
3. Failed deliveries should be retried with backoff for about three days, then marked failed, with the merchant told. An endpoint that has not succeeded for three days is disabled.
4. Merchants should be able to verify each request came from the gateway, was not altered, and is not an old request replayed — including while a secret is being rotated.
5. Merchants should be able to see every attempt and resend or replay events: status code, latency, error and response snippet; resend one event, or replay a time window after an outage.

Non-functional requirements

• Scale. 500 million events a day, with peaks of 8× the average on sale evenings and at month-end settlement. On average each event goes to 1.5 endpoints.
• Latency. From the payment’s commit to the first delivery attempt: p50 under 1 s, p99 under 5 s.
• At-least-once. Once the payment transaction commits, its events are delivered or visibly failed — never silently lost. Duplicates are acceptable; receivers drop them by event ID.
• Isolation. One slow, failing or very busy endpoint must not delay anyone else’s deliveries.
• Bounded attempts. 3 s to connect, 15 s per attempt in total, no retries after 72 hours.
• Security. Every request signed. Delivery traffic can never reach internal networks. Requests leave from a small, published set of IP addresses.
• Observability. Attempt logs kept for 30 days and searchable by endpoint or event within seconds; replay possible for 30 days.

Out of scope

• Producing the events. The payment state machine is someone else’s; our job starts at the commit.
• Inbound webhooks from banks and card networks. That is an ingestion API, a different problem.
• Exactly-once processing at the receiver. We deliver at least once and give receivers the IDs to drop duplicates.
• SMS, email and push. Those belong to a notification system.

The set-up

Plan the approach

Say the shape early: events in, deliveries out, logs on the side. A durable intake (outbox to Kafka), a fan-out, a delivery engine (dispatcher, workers, retry store), a guarded exit to the internet, and a read side for logs and replay. Build each requirement simply, then spend the deep dives on what is hard: never losing an event, isolation, retries, signatures, SSRF, duplicates and ordering, and storing the logs.

Core entities

• Endpoint (we_7Hq2). Merchant, URL, subscribed event types, API version, status (enabled or disabled), signing secrets (current and previous, with expiry), a concurrency cap and a health state.
• Event (evt_9KzP3m). Type, merchant, creation time, the resource it describes (pay_4Lm8) with a version number, and the payload. Immutable.
• Delivery. One per event and endpoint: status (pending, delivered, failed), attempt count, next_attempt_at and a deadline 72 hours after creation. Its ID is derived from the pair, so creating it twice is harmless.
• Attempt. One per HTTP try: start time, egress IP, status code or error class (timeout, TLS, DNS, refused, blocked), latency and the first 1 KB of the response body.
• Replay job. Endpoint, time window, filter (failed only, or everything), a cursor and a rate.

API

There are three surfaces: managing endpoints, the webhook request itself, and logs and replay.

Register an endpoint. The gateway generates a 32-byte random secret and shows it once.

POST /v1/webhook_endpoints
Authorization: Bearer sk_live_…
Content-Type: application/json

{
  "url": "https://api.shop.example.in/hooks/payments",
  "events": ["payment.captured", "payment.failed", "refund.processed"],
  "api_version": "2026-08-01"
}

{
  "id": "we_7Hq2",
  "status": "enabled",
  "url": "https://api.shop.example.in/hooks/payments",
  "events": ["payment.captured", "payment.failed", "refund.processed"],
  "secret": "whsec_AQIDBAUGBwgJCgsMDQ4PEBESExQVFhcYGRobHB0eHyA="
}

The webhook. Use the header names from the Standard Webhooks specification, so receivers can use existing libraries. The signature below is the real HMAC of this body with the secret above.

POST /hooks/payments HTTP/1.1
Host: api.shop.example.in
Content-Type: application/json
webhook-id: evt_9KzP3m
webhook-timestamp: 1793090423
webhook-signature: v1,cUnL8dxdpm+3a2IPOK4rAVHYGn/Iwp9WwXPhSZS5VsI=

{"id":"evt_9KzP3m","type":"payment.captured","created_at":"2026-10-27T08:40:22Z","data":{"payment_id":"pay_4Lm8","order_id":"ORD-4711","amount":149900,"currency":"INR","method":"upi","version":3}}

The contract: any 2xx within 15 seconds means delivered. Everything else — 3xx, 4xx, 5xx, timeouts, TLS and connection errors — is a failure and gets retried. Two codes are special: 410 Gone disables the endpoint, and 429 or 503 with Retry-After pushes the next attempt back.

Logs and replay.

GET  /v1/webhook_endpoints/we_7Hq2/attempts?status=failed&since=2026-10-26T00:00:00Z
GET  /v1/events/evt_9KzP3m/attempts              every try of one event, on every endpoint
POST /v1/events/evt_9KzP3m/resend                body: endpoint = we_7Hq2
POST /v1/webhook_endpoints/we_7Hq2/replay        body: from, to, only_failed → 202 + job id
POST /v1/webhook_endpoints/we_7Hq2/rotate_secret body: keep_old_for = 24h
POST /v1/webhook_endpoints/we_7Hq2/enable        after fixing a disabled endpoint
GET  /v1/events?type=payment.captured&created_after=…   list events, to reconcile

Back-of-envelope estimates

These numbers decide the worker fleet and the log store (method: back-of-envelope estimation).

Events. 500M ÷ 86,400 s ≈ 5.8k events/s on average; × 8 ≈ 46k events/s at peak.

Deliveries. × 1.5 endpoints per event: 8.7k deliveries/s on average, 70k/s at peak, 750M a day.

Retries. Assume 3% of first attempts fail. Two-thirds are blips that succeed after one or two retries (≈ 1.5 extra attempts); one-third hit endpoints that stay down for hours and use most of the schedule (≈ 8 extra). Average: 2/3 × 1.5 + 1/3 × 8 ≈ 3.7 extra attempts per failure, so 3% × 3.7 ≈ 11% more attempts: 750M × 1.11 ≈ 830M attempts a day, about 9.6k/s on average and 77k/s at peak.

Concurrency. Little’s law: requests in flight = arrival rate × time each one takes. Healthy merchants reply in about 300 ms. If 99% of peak deliveries are healthy: 69k × 0.3 s ≈ 21k in flight. If the other 1% hang until the 15 s timeout: 700/s × 15 s ≈ 10.5k in flight. Total ≈ 31k concurrent requests — and 1% of the traffic holds a third of them.

Workers. Budget 2,000 concurrent requests per pod for a non-blocking HTTP client (about 100 KB of buffers and TLS state each, so 200 MB). 31k ÷ 2,000 ≈ 16 pods, doubled for a lost zone and bursts: about 32 pods. With a blocking thread per request and 200 threads per pod you would need 155 pods, and one dead endpoint at 1,000 deliveries a second would pin 1,000 × 15 = 15,000 threads — 75 idle pods.

Egress. 70k/s × about 3 KB (2 KB body plus headers and TLS) ≈ 210 MB/s ≈ 1.7 Gbit/s at peak through the egress proxies.

Storage. Payloads average 2 KB: 500M × 2 KB = 1 TB a day raw, about 250 GB compressed 4×, so 7.5 TB for 30 days — stored once per event, not per endpoint. Attempt records of about 400 B: 830M × 400 B ≈ 330 GB a day, 10 TB for 30 days, 30 TB with three replicas. The Kafka event log: 250 GB a day × 3 days × 3 replicas ≈ 2.3 TB.

Skew. 60,000 endpoints average 750M ÷ 60,000 ÷ 86,400 ≈ 0.14 deliveries/s each, but a large marketplace with 5% of traffic receives 3,500/s at peak and needs about 3,500 × 0.3 ≈ 1,050 requests in flight on its own.

High-level design

Build one requirement at a time. By the end of this section the design works for one region at modest traffic; the deep dives make it survive the real world.

Merchants register endpoints and choose events

A small endpoint service owns the control plane. When a merchant registers a URL, it checks the URL (HTTPS only, a public hostname, no user:password@), generates a 32-byte random secret, encrypts it with a key from the key management service (see encryption and key management) and sends a signed test ping. Only a 2xx enables the endpoint. The endpoints database is tiny — 60,000 rows — so one PostgreSQL primary with a replica is plenty. Every change is pushed to the fan-out service’s in-memory cache within seconds, so the hot path never queries this database.

Every event reaches every subscribed endpoint within seconds

For now, the payments service publishes each event to a Kafka topic called events right after its database commit, keyed by merchant. (The first deep dive shows why “right after” is not good enough.)

1. The fan-out service reads payment.captured for merchant m_311, finds that merchant’s endpoints subscribed to that type in its cache, and renders the payload in each endpoint’s API version.
2. It writes the event to the event store once (for logs, replay and the list-events API), then writes one message per endpoint to a deliveries topic, keyed by endpoint ID. Each message carries the rendered payload, so workers never read a database on the hot path.
3. Workers consume deliveries, sign each request, POST it with a 15-second deadline and record the outcome.

Here the workers are an ordinary consumer pool. That is the part the second deep dive tears up.

Failed deliveries are retried for three days, then dead-lettered

A merchant’s server goes down at 07:55 for a deploy that goes wrong. Its deliveries must wait and try again, without blocking a worker and without being lost if a worker restarts.

On failure, the worker computes the next attempt time from the schedule and writes the delivery to a retry store. A timer claims rows whose time has come and puts them back on deliveries. When a delivery passes its 72-hour deadline it moves to a dead letter state, still visible and replayable, and a notifier emails the merchant. An endpoint with no 2xx for 72 hours is disabled and the merchant is told. The exact schedule, and how not to cause storms, is the third deep dive; the general patterns are in delivery guarantees and DLQs.

Merchants can verify that a request came from us

Each worker signs every attempt with the endpoint’s secret: an HMAC-SHA256 over the event ID, the attempt’s Unix timestamp and the exact body bytes, sent in the webhook-signature header next to webhook-id and webhook-timestamp. Each retry gets a fresh timestamp and signature. The gateway also publishes the small set of IP addresses its requests leave from, so cautious merchants can allow-list them. The fourth deep dive covers verification, replay windows and rotation.

Merchants can see every attempt and replay

Workers write one record per attempt to an attempt log. The management API reads it for the dashboard (“3 failed attempts, last one 503 after 412 ms”), and turns resend and replay requests into new messages on a replay lane, reading payloads from the event store.

This design works: a few thousand deliveries a second, healthy merchants, nobody attacking it. Now push on it.

Deep dives

How do we never lose an event between the payment and the webhook?

The payment row and the event live in two systems: the payments database and Kafka. Any design that writes to both separately — a dual write — has a moment where one has happened and the other has not.

Bad: commit, then publish. The payments service commits the capture, then publishes payment.captured. If the process crashes, is redeployed or loses its network between those two lines, the payment exists and the event never does. At 5.8k events a second, a pod killed mid-deploy drops a handful every time. The merchant never ships an order the customer paid for, and nobody notices until the customer calls.

Also bad: publish inside the transaction. Publish first, then commit. Now a rolled-back transaction leaves a phantom event — you told the merchant a payment was captured when it was not, which is worse than silence for a payments company. And a slow broker holds a database transaction, and its row locks, open.

Good: a transactional outbox with a poller. In the same transaction as the payment, insert a row into an outbox table: event ID, type, merchant, payload. A relay process polls the table every 100 ms, publishes new rows to Kafka and marks them sent. Because the event is written in the same commit, it exists exactly when the payment does (the pattern has its own page: transactional outbox). The cost is polling load on the payments database: at 46k events a second across, say, 16 shards, each relay reads and updates about 2,900 rows a second, and the table needs regular cleanup.

Great: an outbox read by change data capture. Instead of polling, a CDC connector tails the database’s write-ahead log and publishes each new outbox row — Debezium, for example, ships an outbox event router for exactly this. There is no extra query load, events appear within a few hundred milliseconds, and they come out in commit order per shard. Rows can be deleted right after insert (the log still has them) or dropped in daily partitions. You pay for running CDC infrastructure and for watching its lag; more in change data capture.

Either relay is at least once: if it crashes after publishing but before recording its position, it publishes again. So make everything downstream idempotent. The event ID is created inside the payment transaction and never changes. The fan-out derives each delivery ID from the event ID and endpoint ID, so a re-published event upserts the same deliveries instead of creating new ones. Kafka keeps three days of events, enough to rebuild the fan-out’s output after a bad deploy.

How do we stop one slow or dead endpoint from delaying everyone else?

This is the heart of the problem. Your receivers are 60,000 independent servers, and at any moment hundreds of them are slow or down.

Bad: one shared queue and blocking workers. All deliveries sit in one FIFO queue (or a Kafka topic whose partitions mix endpoints), and each worker thread takes the next message and waits for the HTTP response. Picture merchant C’s server hanging during a sale while C receives 1,000 deliveries a second. Each attempt holds a worker for the full 15 seconds: 1,000 × 15 = 15,000 workers stuck, more than the whole fleet. Within seconds every worker is waiting on C, and A and B — healthy merchants — get nothing. This is head-of-line blocking, and it turns one merchant’s outage into everyone’s.

Good: partition the work by endpoint. Key the deliveries topic by endpoint ID, or give every endpoint its own queue. Partitioning shrinks the blast radius but does not remove it: 60,000 endpoints over 600 partitions puts about 100 endpoints in each, and a consumer reading a partition in order still stalls C’s 99 neighbours. A queue per endpoint isolates perfectly, but 60,000 queues averaging 0.14 messages a second are expensive to poll, and you still need per-endpoint caps and fair sharing of workers.

Great: per-endpoint lanes, a scheduler and non-blocking workers. Separate reading deliveries from sending them:

• Lanes. A dispatcher reads deliveries partitions as fast as they arrive and puts each message in an in-memory lane for its endpoint. Reading never waits on HTTP.
• Concurrency caps. Each endpoint may have at most N requests in flight — 50 by default. By Little’s law an endpoint answering in 300 ms gets 50 ÷ 0.3 ≈ 166 deliveries a second, far above the 0.14 average. The cap adapts like TCP: +1 for each second without errors, halved on a timeout, 429 or 503. The marketplace that needs 1,050 in flight grows its cap up to a contracted ceiling.
• Fair turns. The scheduler takes ready lanes in round-robin (deficit round-robin, weighted by plan), so a merchant with 3,500 deliveries a second cannot starve the 59,999 small ones. More in noisy neighbours and fairness.
• Non-blocking workers. An event-loop HTTP client holds 2,000 requests per pod and keeps connections alive per host. A request that hangs costs a socket and some memory, not a thread.
• A slow lane. Endpoints that are erroring or slow move to a separate, small worker pool — a bulkhead — so their 15-second timeouts can never use up the main pool’s connection budget.
• Parking. When a lane holds more than 2,000 messages or its oldest has waited 30 seconds, the dispatcher writes the backlog to the retry store and drops it from memory.

Parking is what makes this safe with Kafka. The dispatcher commits a partition’s offset only up to the oldest unsettled delivery, where settled means delivered or safely written to the retry store. A dead endpoint’s messages are parked within 30 seconds, so they never pin the offset and a restart never loses them.

Each endpoint also carries a health state, computed from a rolling two-minute window of its outcomes. This is a circuit breaker per endpoint, with one extra state for gentle recovery.

What you gain: a dead endpoint costs one probe a minute instead of 15,000 sockets, and a merchant’s outage stays the merchant’s. What you pay: a stateful dispatcher (lanes, caps, health) that must hand its partitions over cleanly on a rebalance, and careful offset bookkeeping. Lanes are cheap — an empty lane is a few hundred bytes — so 60,000 of them fit easily in memory.

How do we retry for three days without retry storms or lost state?

About 260 deliveries fail every second on an average day (3% of 8.7k). Their retries must survive deploys and must not hurt a merchant who is trying to recover.

Bad: sleep in the worker, or keep timers in memory. A worker that sleeps between retries holds its slot for minutes or hours, and an in-memory timer wheel holds millions of pending retries that vanish on the next deploy. Retries without jitter are worse still: when a hosting provider that runs 5,000 merchant sites blips at 20:00:00, every one of their deliveries retries at 20:00:15, then again at 20:02:15 — a synchronised retry storm aimed at a provider that is trying to come back.

Good: delay queues with fixed tiers. Create topics such as retry-15s, retry-2m, retry-10m … retry-24h. A failed delivery goes to the next tier; each tier’s consumer waits until a message is due, and because every message in a tier has the same delay, arrival order is due order. It is durable and simple. But the delays are fixed, so you cannot honour Retry-After: 120 exactly. You cannot pause or cancel: a disabled endpoint’s messages keep cycling through the tiers. And a dead endpoint with a million queued deliveries has every one of them re-read and re-written nine times.

Great: a durable timer keyed by next_attempt_at, and retry the endpoint, not the message.

• Durable schedule. The retry store is a table sharded by endpoint, indexed on (shard, next_attempt_at). A timer per shard claims due rows in batches with a short lease, so two timers never take the same row, and hands them to the dispatcher. A crash just means the lease expires and another timer picks them up. This is a small job scheduler specialised for one job.
• Backoff with jitter. After attempt n fails, next_attempt_at = now + delay[n] × random(0.8, 1.2), with delays of 15 s, 2 min, 10 min, 1 h, 3 h, 6 h, 12 h, 24 h and 24 h: ten attempts over about 70 hours, then the dead letter. The ±20% jitter turns a cliff of retries into a smear (see timeouts, retries and backoff with jitter).
• Listen to the receiver. Retry-After on a 429 or 503 moves the next attempt to at least that time (capped at 24 hours) and halves the endpoint’s cap. 410 Gone disables the endpoint at once, as the Standard Webhooks spec suggests. A 3xx is a failure, never followed. A request the egress proxy blocked is a permanent failure that alerts the merchant.
• Retry the endpoint. Once an endpoint is Failing, stop scheduling each delivery separately. New deliveries are parked under the endpoint, and only one probe — the oldest parked delivery — is scheduled, every minute at first and every 30 minutes later. A dead endpoint receiving 1,000 deliveries a second costs one request a minute instead of 15,000 sockets. When the probe gets a 2xx, the endpoint enters Recovering and its backlog drains oldest first while its cap doubles each minute. Each delivery still keeps its own 72-hour deadline; anything past it goes to the dead letter.

The size of the retry store stays modest. Most of the 260 new failures a second leave within minutes. The big case is a large merchant down for a day: 1% of traffic × 8.7k/s × 86,400 s ≈ 7.5 million parked rows, which is a few gigabytes.

Short windows (Shopify’s four hours) keep queues small and surface problems fast, but punish a merchant whose server is down overnight. Three days survives a long weekend, at the cost of a bigger retry store and events that arrive very late — so every payload carries its own created_at.

How do receivers know a request is really from us — and not a replay?

Anyone on the internet can POST JSON to a merchant’s webhook URL. If the merchant’s server trusts it, an attacker can mark unpaid orders as paid.

Bad: a shared token in the URL, or nothing. https://shop.example.in/hooks?token=abc123 leaks into access logs and screenshots, says nothing about whether the body was altered, and never changes. IP allow-listing alone is not enough either: cloud IP ranges are shared.

Good: an HMAC of the body. Sign the raw body with HMAC-SHA256 and a per-endpoint secret. GitHub sends X-Hub-Signature-256: sha256= plus a hex digest; Shopify sends a base64 digest in X-Shopify-Hmac-SHA256. This proves who sent it and that nothing changed. But a captured request stays valid forever: an attacker who sees one payment.captured can replay it next month. GitHub’s advice is to use the per-delivery X-GitHub-Delivery ID to spot replays.

Great: sign the ID and a timestamp too, and support rotation. The Standard Webhooks scheme signs webhook-id.webhook-timestamp.body; Stripe signs timestamp.body and sends t= and v1= in its Stripe-Signature header. The receiver rejects any timestamp outside a tolerance window — Stripe’s libraries default to five minutes — so a captured request is useless after five minutes, and the ID lets it drop a duplicate within the window.

The details that make it hold up:

• Per-endpoint secrets of 24 to 64 random bytes, shown with a whsec_ prefix so they are easy to recognise in a leak scan. Store them encrypted; workers keep decrypted keys in memory for a few minutes.
• A fresh timestamp and signature on every attempt, but the same webhook-id. Stripe does the same for its retries.
• Rotation without downtime. During an overlap window, sign with both the new and the old secret and send both signatures, space-separated, in webhook-signature. The receiver accepts if any one matches. Stripe lets the old secret live for up to 24 hours; so do we, by default.
• An asymmetric option. With ed25519 (v1a in the spec), we sign with a private key and merchants verify with a published public key, so a leak on the merchant’s side cannot be used to forge our requests. It costs more CPU per signature, so offer it as an opt-in.
• Alternatives for strict customers. Mutual TLS with our client certificate (see service-to-service auth and mTLS), or an OAuth access token in the Authorization header: stronger, but harder for small merchants to set up. Fixed egress IPs add a layer for those who can allow-list.

Receiver-side verification is short. This version handles rotation and the replay window:

import base64, hashlib, hmac, time

def verify(secret: str, headers: dict, body: bytes, tolerance_s: int = 300) -> bool:
    key = base64.b64decode(secret.removeprefix("whsec_"))
    msg_id = headers["webhook-id"]
    ts = int(headers["webhook-timestamp"])
    if abs(time.time() - ts) > tolerance_s:          # stale or from the future: treat as a replay
        return False
    signed = f"{msg_id}.{ts}.".encode() + body         # the raw bytes, never re-serialised JSON
    expected = base64.b64encode(hmac.new(key, signed, hashlib.sha256).digest()).decode()
    for entry in headers["webhook-signature"].split(" "):   # several during secret rotation
        version, _, sig = entry.partition(",")
        if version == "v1" and hmac.compare_digest(sig, expected):
            return True
    return False

How do we stop merchants from using us to attack our own network?

A webhook service is a machine that sends HTTP requests to any URL a stranger types in, from inside your data centre. That is the textbook setup for server-side request forgery (SSRF), and the OWASP cheat sheet names custom webhooks as a case. If an attacker registers http://169.254.169.254/latest/meta-data/ and your workers can reach the cloud metadata service, the “response snippet” in the delivery log may hand them your instance credentials.

Bad: validate the URL when it is registered. A regex that rejects 10., 192.168. and localhost misses almost everything:

• Other spellings of the same address: http://2130706433/ is 127.0.0.1 in decimal, and [::ffff:a9fe:a9fe] is the metadata address written as an IPv4-mapped IPv6 address.
• Names that resolve inside: meta.evil.test can simply have an A record of 169.254.169.254.
• DNS rebinding: the name resolves to a public IP when you check it, with a zero TTL, and to 127.0.0.1 when you send.
• Redirects: a perfectly public endpoint answers 302 Location: http://10.0.3.7/admin.
• Parser disagreements: OWASP shows http://example.com\@evil.com, which two common URL parsers read as two different hosts.

Good: resolve and check at send time. Before each attempt, resolve the hostname and reject the request if any A or AAAA answer is in a blocked range. OWASP’s minimum list is a sensible start: 127.0.0.0/8, 0.0.0.0/8, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 169.254.169.254, ::1, fc00::/7, fe80::/10 and multicast. Normalise IPv4-mapped IPv6 addresses before checking. The hole: your HTTP client resolves the name again when it connects, and the attacker’s DNS server can answer differently the second time.

Great: resolve once, pin the IP, refuse redirects, and isolate the network.

• Resolve, check, connect to that IP. Resolve the name once, check every answer, then open the TCP connection to the checked address while keeping the original hostname for the Host header, TLS SNI and certificate verification — exactly the fix OWASP describes for rebinding. Apply the same check to every retry and fallback address.
• Never follow redirects. Treat every 3xx as a failure and show the Location in the delivery log so the merchant can update the URL. Stripe, Svix and the Standard Webhooks spec all treat redirects as failures.
• Isolate the network. Workers and proxies run in their own subnet with no route to internal services and no access to the metadata endpoint. All delivery traffic goes through an egress proxy fleet. Stripe open-sourced its own, Smokescreen: an HTTP CONNECT proxy that resolves each host, refuses non-public addresses, and gives partners stable egress IPs. A forward proxy is also the natural place to publish four fixed IPs per region.
• Limit what comes back. Read at most 64 KB of any response and store only the first 1 KB. Report blocked attempts as “destination not allowed”, without detailed connection errors, and rate-limit test pings, so the test button cannot become a port scanner.

You pay for an extra hop (a fraction of a millisecond inside a zone) and a proxy fleet to run at 1.7 Gbit/s. You gain a guarantee that does not depend on every code path remembering a check.

How do we handle duplicates and ordering?

Duplicates are guaranteed. At-least-once means the same event can arrive twice: the relay re-publishes after a crash; a worker times out at 15 seconds while the merchant finishes at 16; a worker gets a 200 and crashes before recording it; a merchant presses resend. We remove the duplicates we can — delivery IDs are derived from event and endpoint, so re-fanning out is an upsert — but the last three are invisible to us. So every request carries a webhook-id that never changes across retries, resends and replays (GitHub’s X-GitHub-Delivery behaves the same way on redelivery). Receivers record processed IDs under a unique constraint in the same transaction as their side effect — the inbox pattern — and our docs say so in the first paragraph. More on the idea in idempotency.

Ordering is not guaranteed — on purpose. Stripe and Shopify both state that events can arrive out of order. Retries reorder by definition: if payment.captured fails once and refund.processed succeeds first, the refund arrives before the capture. You could promise strict order per endpoint, but look at the cost.

Strict order allows one request in flight: at 300 ms each, about 3 deliveries a second, and one failing event blocks everything behind it for up to three days — the sequential convoy, explained in message ordering and per-key queues. So give receivers what they need to cope without order:

• A version and a timestamp in every payload. pay_4Lm8 at version: 3 with created_at. The receiver stores the highest version it has applied per resource and ignores anything older. A late payment.captured cannot undo a refund.processed that came first.
• Thin events as an option. A thin event says only “pay_4Lm8 changed”; the receiver fetches the current state from the API, so order stops mattering. Stripe now recommends thin events for new integrations. You gain smaller payloads, no stale data and less personal data on the wire; you pay with an API call per event — up to 70k extra reads a second at peak, which your read API must be sized for.
• Versioned payloads. Each endpoint pins an API version; the fan-out renders the canonical event in that version. Adding fields is safe, breaking changes need a new version, and old renderers are kept until no endpoint uses them (see serialization and schema evolution).

How do we store and query 830 million attempts a day?

The log has a very particular shape. Writes are append-only, time-ordered and heavy: up to 77k attempts a second, 330 GB a day. Reads are narrow: “failed attempts for we_7Hq2 in the last 24 hours, newest first”, “every attempt for evt_9KzP3m”, “failed deliveries between 09:00 and 13:00 for replay”. And everything expires after 30 days.

Bad: one relational table with indexes. 830 million inserts a day becomes 25 billion rows in a month, every insert updates two or three indexes, and expiring old data means 830 million deletes a day — with the vacuum and bloat that follow.

Good: time-partitioned relational tables. Daily partitions turn expiry into dropping a partition (see table partitioning). It works up to a few billion rows, but one primary taking 77k inserts a second plus indexes is near its limit, so you end up sharding by endpoint anyway.

Great: a wide-column store, shaped to the queries.

• Partition by endpoint and day, cluster by time. In a wide-column store such as Cassandra or DynamoDB, the key (endpoint_id, day) with rows sorted newest first makes the dashboard query a single partition read. Writes are cheap appends spread across the whole cluster.
• Expire by TTL, not by delete. Write each row with a 30-day TTL and use time-window compaction, so whole files of expired rows are dropped at once instead of being tombstoned row by row.
• Split hot partitions. The marketplace endpoint, with 5% of traffic, writes about 40 million attempts a day (16 GB); one (endpoint, day) partition would be far too big. Add a bucket — (endpoint_id, day, bucket 0–15) — for endpoints above a threshold, and read the 16 buckets in parallel.
• Look up by event. A second small table maps event_id to its delivery IDs, for support (“where did evt_9KzP3m go?”).
• Payloads once. The event store is keyed by event_id, about 7.5 TB compressed for 30 days; deliveries and attempts point to it instead of copying 2 KB per row.
• Metrics separately. Attempts also stream into a columnar store (see OLTP vs OLAP) for per-endpoint success rates and p95 latency per minute. That powers the dashboard charts and an email to merchants whose failure rate jumps — before their endpoint gets disabled.

Replay is then a paged scan. The replay job walks the endpoint’s (endpoint, day) partitions in the window, picks failed deliveries (or every event, if asked), and puts them on a replay lane that the dispatcher serves at low priority with its own rate cap — 50 a second to start, ramping up while the endpoint stays healthy. Live traffic never waits behind a replay. Events created while an endpoint was disabled have no delivery rows, so “replay missing” reads the event store’s per-merchant index instead and regenerates them.

The final design

Put the pieces together. The hot path runs from the payments database to the merchant’s URL; everything that waits — retries, dead letters, logs, replays — sits beside it.

Follow one payment.captured through the system:

1. Commit. The payments service writes the payment row and an outbox row in one transaction. A CDC relay tails the log and publishes the event to the events topic, keyed by merchant.
2. Fan out. The fan-out service looks up subscribed endpoints in its cache, stores the payload once in the event store, and writes one delivery per endpoint to deliveries, keyed by endpoint, with an ID derived from event and endpoint.
3. Dispatch. The dispatcher puts each delivery in its endpoint’s lane and serves ready lanes in weighted round-robin, within each endpoint’s cap and health state; degraded endpoints use the slow lane.
4. Send. An async worker signs the request with every active secret and sends it through the egress proxy, which resolves, checks and pins the destination IP and leaves from one of the fixed egress IPs. A 2xx within 15 seconds marks the delivery done.
5. Fail. On any other outcome, the worker writes the delivery to the retry store with a jittered next_attempt_at, honouring Retry-After. If the endpoint is now Failing, its new deliveries are parked and only a probe is scheduled.
6. Retry. Timers claim due rows and probes and hand them back to the dispatcher. A successful probe moves the endpoint to Recovering and its backlog drains at a rising rate.
7. Give up. A delivery past its 72-hour deadline becomes a dead letter; the notifier emails the merchant, and an endpoint with no 2xx for 72 hours is disabled.
8. Record. Every attempt goes to the attempt log, keyed by endpoint and day, and to a columnar store for success rates and alerts.
9. Inspect and replay. Merchants use the management API to manage endpoints and secrets, read attempts, resend single events and replay windows through the low-priority replay topic.

Check it against the requirements. Events are durable from the commit, and stable IDs make duplicates harmless. Commit to first attempt is typically under a second: CDC in a few hundred milliseconds, two Kafka hops and an in-memory lane. About 32 worker pods carry 31k concurrent requests at peak, and a dead endpoint costs one probe a minute. Retries last about 70 hours, then a dead letter and an email. Requests are signed, replay-protected and unable to reach inside, and every attempt is searchable and replayable for 30 days.

What interviewers look for by level

Follow-up questions

What if a top merchant wants 2,000 events a second in strict order?

Strict order per endpoint allows one request in flight, so at 300 ms you get about 3 a second, far short of 2,000. Ask what they really need ordered. Usually it is per payment or per order, not global: key lanes by resource within the endpoint, so thousands of payments flow in parallel while each payment’s events stay in order, and one stuck payment stalls only itself. If they truly need a total order at that rate, push is the wrong tool: offer a pull API with a cursor over the event log, or stream into their own Kafka or EventBridge, where ordering per partition is native.

What if a merchant’s endpoint was down for four days and they want everything?

Retries stopped at 72 hours and the endpoint was disabled, but deliveries and payloads stay for 30 days. Once they fix the server and re-enable the endpoint, a replay job for the four-day window, failed and missing deliveries only, feeds the replay lane. A merchant with 100,000 events a day has 400,000 deliveries to catch up; at a ramp that settles near 200 a second, that is about 33 minutes, while live traffic keeps flowing in its own lane. Some of those events were delivered before the outage, so the receiver must drop duplicates by webhook-id. Past 30 days, only the list-events API for reconciliation remains, and the docs should say so.

What if a merchant wants events in their Kafka, SQS or EventBridge instead of HTTP?

Make the destination type pluggable. The lane, retry, logging and replay machinery stays the same; only the sender changes: HTTPS, Amazon EventBridge, SQS, Google Pub/Sub. Stripe offers EventBridge and Azure Event Grid destinations, and Shopify supports Pub/Sub and EventBridge. The merchant gains a durable, scalable buffer with no public endpoint to run, and no HMAC to check, because cloud IAM authenticates the sender. You pay with per-merchant cross-account credentials to manage, new failure modes such as throttling and quota errors, and ordering rules that differ by destination.

What if we deliver from several regions and merchants allow-list our IPs?

Run a full delivery stack in each region — say Mumbai and Hyderabad — with four static egress IPs per region. Publish all eight through the docs and an API and ask merchants to allow-list every one, so a failover never changes what they see. Each endpoint has a home region that owns its lane, so two regions never deliver in parallel; if a region fails, the other takes over from the replicated event log and retry store, accepting some duplicates. Changing the list needs months of notice. More in multi-region and active-active.

What if the receiver returns 200 but never processes the event?

Then, as far as the gateway can know, it was delivered: the contract ends at the 2xx. Receivers should persist, then acknowledge: write the event to their own inbox table or queue, reply 200, and process it asynchronously. A crash after the reply then loses nothing. For anything that matters, they should also reconcile on a schedule: Shopify recommends periodic reconciliation jobs, and Stripe’s list-events API can return events whose delivery failed. Our part is making that easy: a list-events API with a cursor, stable IDs, and a resend button.

What if the payload contains personal data?

Payment events carry names, phone numbers, UPI IDs and partial card numbers. Send less: thin events, or per-endpoint field selection, and never a full card number. Require HTTPS with TLS 1.2 or later — Stripe accepts only TLS 1.2 and 1.3. Encrypt payloads at rest with per-merchant keys, so deleting a key erases that merchant’s data (crypto-shredding). Keep only the first 1 KB of responses in logs, audit dashboard access, and let the 30-day TTL do the rest. Data-protection law, such as India’s DPDP Act, makes this a requirement, not a nicety.

How to say it in the interview

“I’ll design the webhook platform of a payments gateway: 500 million events a day, peaking at 46 thousand a second, delivered to about 60 thousand merchant endpoints we don’t control. The hard parts are never losing an event, stopping one bad endpoint from hurting the rest, retrying politely for days, and security.

Intake first. The payments service writes each event to an outbox table in the same transaction as the payment, and CDC publishes it to Kafka, so a committed payment always produces an event. A fan-out service looks up subscriptions, stores the payload once and writes one delivery per endpoint, with an ID derived from event and endpoint so re-processing is harmless.

Delivery is where the design lives. Little’s law says 70 thousand deliveries a second at 300 milliseconds is 21 thousand requests in flight, plus another 10 thousand if just 1% of endpoints hang until our 15-second timeout. So workers are non-blocking, a dispatcher keeps a lane per endpoint with an adaptive concurrency cap and fair round-robin, slow endpoints go to a separate pool, and failing ones are parked and probed once a minute instead of retried message by message.

Retries live in a durable store keyed by next attempt time: ten attempts over about 70 hours, with jitter, honouring Retry-After and 410 Gone, then a dead letter and an email.

Security has two sides. Every request carries webhook-id, a timestamp and an HMAC-SHA256 over all three plus the body, so receivers reject forgeries and replays older than five minutes, and we sign with two secrets during rotation. And because merchants choose the URLs, everything leaves through egress proxies in an isolated network that resolve, block private addresses, pin the IP and never follow redirects.

Delivery is at-least-once and unordered, so payloads carry stable IDs and resource versions. Every attempt goes into a wide-column log keyed by endpoint and day with a 30-day TTL, which powers the dashboard and a rate-limited replay lane.”

Concepts used in this design

• Webhooks — the concept this system delivers at scale.
• Transactional outbox — writing the event in the same commit as the payment.
• Change data capture — publishing outbox rows from the database log.
• Kafka — the durable event log between producers and the fan-out.
• Delivery guarantees, at-least-once and DLQs — why duplicates are the price of never losing events.
• Idempotency — stable IDs that make duplicates harmless.
• Effectively-once and the inbox pattern — what receivers do with those IDs.
• Message ordering and the sequential convoy — why strict order stalls a lane.
• Timeouts, retries and backoff with jitter — the retry schedule and its spread.
• Retry storm — what synchronised retries do to a recovering receiver.
• Circuit breaker — the per-endpoint health states.
• Bulkheads — the separate slow lane.
• Noisy neighbours and fairness — weighted round-robin across merchants.
• Service-to-service auth and mTLS — the stronger alternative to HMAC.
• Cassandra and DynamoDB — the attempt log keyed by endpoint and day.
• Multi-region and active-active — regional stacks with fixed egress IPs.

References and further reading

1. Standard Webhooks specification (v1.0.0)github.comSigning id.timestamp.body with HMAC-SHA256 (v1) or ed25519 (v1a), the webhook-id/-timestamp/-signature headers, space-separated signatures for rotation, 24–64 byte secrets, a 15–30 s timeout, status-code handling and a three-day example retry schedule.
2. Stripe: Receive Stripe events in your webhook endpointdocs.stripe.comThree days of retries with exponential backoff, the Stripe-Signature t= and v1= scheme, a 5-minute default tolerance, a fresh signature per retry, 24-hour secret overlap, 3xx as failure, no ordering guarantee, thin payloads recommended.
3. Stripe: Manage webhook endpoints and process undelivered eventsdocs.stripe.comRecovering from downtime by listing events with delivery_success=false over the last 30 days, and skipping events already processed while automatic retries continue.
4. Svix documentation: Retry scheduledocs.svix.comAttempts immediately, then after 5 s, 5 min, 30 min, 2 h, 5 h, 10 h and 10 h; a 30 s response window; endpoints disabled after five days of failures; recover-failed and replay-missing.
5. Shopify: Verify webhook deliveriesshopify.devBase64 HMAC-SHA256 of the raw body in X-Shopify-Hmac-SHA256, X-Shopify-Webhook-Id for duplicates, a 1 s connect and 5 s total timeout, and 8 retries over 4 hours before the subscription is removed.
6. GitHub Docs: Best practices for using webhooksdocs.github.comReceivers must reply within 10 seconds; publishing sender IPs for allow-lists; X-GitHub-Delivery stays the same on redelivery and helps detect replays.
7. GitHub Docs: Validating webhook deliveriesdocs.github.comX-Hub-Signature-256 as sha256= plus a hex HMAC of the body, and why a plain == comparison must be replaced by a constant-time one.
8. GitHub Docs: Redelivering webhooksdocs.github.comGitHub does not retry failed deliveries automatically; deliveries from the past three days can be redelivered by hand or through the API.
9. OWASP Server-Side Request Forgery Prevention Cheat Sheetcheatsheetseries.owasp.orgWebhooks as a named SSRF case, why domain checks alone miss DNS rebinding, connecting to the validated IP while keeping Host and SNI, disabling redirects, and the minimum ranges to block.
10. Smokescreen, Stripe’s egress proxy (GitHub)github.comAn HTTP CONNECT proxy for outbound traffic such as webhooks that resolves each host and refuses non-public addresses, and centralises egress so partners get stable IPs.