Webhooks

A webhook is an HTTP callback that a customer registers with a provider: when something happens (a payment succeeds, a pull request opens, an order ships), the provider POSTs a small event to the customer’s URL instead of waiting to be asked. It turns polling into push. The POST is the easy part; the design work is in everything around it: signing, retrying for days, deduplicating, coping with events that arrive out of order, and catching up on the ones that never arrive.

Picture it first: the doorbell, not the gate

You have ordered a phone and a charger online. You could walk to the gate every 30 seconds to see whether the courier has come: that is polling, tiring and mostly pointless. Or the courier rings the bell: that is a webhook. The bell brings its own rules:

• Is it really the courier? You check the OTP the shop sent you. The receiver checks a signature.
• Sign now, unpack later. The receiver replies 2xx at once and does the work in the background.
• Nobody home. The courier tries a few more times, then leaves the parcel at the depot. The sender retries with backoff, then dead-letters the event for replay.
• The charger before the phone, or the same parcel twice. Events arrive out of order and more than once, so the receiver dedupes on the event ID.
• You still check the order page, in case the bell rang while you were in the shower. That is a reconciliation job against the provider’s API.

What a webhook is, and what it replaces

Jeff Lindsay popularised the name in a 2007 post: “user defined callbacks made with HTTP POST”. The subscriber registers a URL and the event types it wants; the provider calls it when one happens, a “reverse API”.

Typical events: a payment gateway telling a merchant that a UPI collect request was approved, minutes after the API call returned “pending”; GitHub telling a CI server about a push; a courier aggregator telling a shop that its parcel shipped. Each is an asynchronous outcome that only the provider sees happen. The alternative is to keep asking.

Take a payment provider with 10,000 merchants polling every 30 seconds, where a typical merchant gets 8 orders a day:

• Per merchant: 86,400 s ÷ 30 s = 2,880 polls a day, at least 2,872 of them empty.
• In total: 2,880 × 10,000 ≈ 28.8 million requests a day (about 333 a second) to learn about 8 × 10,000 = 80,000 orders: 360 requests per piece of news.
• Delay: 15 seconds on average, 30 at worst. Halving it doubles the load.

Webhooks send 80,000 POSTs a day, about one a second, each moments after its event. GitHub’s docs make the same case: polling many resources burns through your API rate limit.

Polling still has its place: a handful of resources, a consumer that can’t expose a public HTTPS endpoint (a phone app, a laptop behind NAT), and the backstop that catches what webhooks miss. To push to users watching a screen, use long polling, SSE or WebSockets instead (real-time connections).

Anatomy of an event

A webhook request has a body describing the event and headers describing this delivery attempt.

POST /webhooks/payments HTTP/1.1
Host: shop.example.in
Content-Type: application/json
webhook-id: evt_01J9Z6Q4K7
webhook-timestamp: 1793092445
webhook-signature: v1,K3m0…= v1,9fQx…=

{"id":"evt_01J9Z6Q4K7","type":"payment.succeeded","created_at":"2026-10-27T09:14:05Z",
 "api_version":"2026-09-01","data":{"id":"pay_8812","status":"succeeded","amount":49900,
 "currency":"INR","version":7}}

• id: unique per event and identical on every retry, so it is the receiver’s dedupe key.
• type: a dot-separated name such as payment.succeeded; subscribers filter on it, and each type has one fixed schema.
• created_at: when the event happened, not when this attempt was sent.
• api_version: which shape data has, so the schema can evolve without breaking old receivers.
• Headers: the Standard Webhooks spec puts the ID, the time of this attempt and the signature in webhook-id, webhook-timestamp and webhook-signature. Stripe, GitHub and Shopify use their own names for the same three ideas.

Two efforts try to stop every provider inventing its own envelope. Standard Webhooks fixes the headers and signature scheme. CloudEvents, a CNCF spec, fixes the event attributes (id, source, specversion and type are required) across HTTP, Kafka and more; source plus id identifies an event.

Thin or fat?

The biggest payload decision: send the object, or a pointer to it?

Stripe offers both and recommends thin events for new integrations, warning that a snapshot may be outdated by the time you handle it. Standard Webhooks agrees that thin is more future-proof (you can fatten a thin payload later, never the reverse) and suggests staying under about 20 KB. Fat payloads suit audits and tools that can’t call back; a common middle ground is thin plus the field every handler needs, such as status.

Version payloads like any public API. Stripe renders snapshot events in the API version pinned to the endpoint and never edits an event after creation; breaking changes get a new version or a new event type (serialization and schema evolution).

The sender: from state change to POST

A naive sender commits the order, then calls http.post(url, event) in the same request handler. A crash between the two loses the event, a slow customer endpoint slows your checkout, and a failure has nowhere to wait for a retry. A real sender is a small pipeline:

1. Capture the event atomically. Insert it into an outbox table in the same transaction as the state change; a relay publishes it afterwards (transactional outbox).
2. Fan out. Match the event against subscriptions by account and type, creating one delivery task per endpoint.
3. Queue per endpoint, each with its own concurrency cap.
4. Deliver. A worker signs the body, POSTs it with tight timeouts and records the outcome.
5. Retry or give up. Failures go to a retry scheduler; exhausted tasks go to a dead-letter store.

Why one queue per endpoint

With one shared queue, the slowest receiver sets everyone’s pace. Suppose a large merchant gets 50 events a second and its endpoint starts hanging until your 10-second timeout. Keeping up would take 50 × 10 = 500 workers stuck waiting; with a shared pool of 200, every other merchant queues behind it. That is head-of-line blocking.

Per-endpoint lanes are a bulkhead: Shop C’s backlog grows in its own queue, and its cap (2 instead of 10) stops it hogging workers. Good senders lower a lane’s cap automatically while its error rate is high, a per-endpoint circuit breaker. Without lanes, one customer becomes everyone’s noisy neighbour.

One delivery attempt

• Timeouts are short. Shopify allows 1 second to connect and 5 for the whole request; GitHub waits 10 seconds, Slack’s Events API 3. Standard Webhooks suggests 15 to 30. Short timeouts free workers sooner but fail receivers that work before answering, which is why every provider says: answer first, work later.
• Only 2xx is success. Stripe, Shopify and the CloudEvents webhook spec treat a 3xx as a failure and don’t follow it: a redirect means a stale URL, and following it lets someone steer your traffic. (Slack follows up to two.)
• Some failures carry instructions. Standard Webhooks treats 410 Gone as “disable this endpoint”, 429, 502 and 504 as “slow down”, and honours Retry-After.
• Reuse connections. Shopify’s sender uses HTTP keep-alive, saving a TLS handshake per event (HTTP and HTTPS).

Ordering is not guaranteed

Two workers in one lane can finish in either order, and a retried event lands after newer ones; Stripe and Shopify both say plainly that they don’t guarantee order. Strict per-object order is possible (one delivery in flight per key), but then one failing event blocks everything behind it for days, the sequential convoy. Most providers choose throughput and leave ordering to the receiver.

Retries, dead letters and disabled endpoints

Receivers fail for ordinary reasons: a deploy, an expired certificate, a bug in one event type. The sender retries with exponential backoff and jitter, waiting longer after each failure and randomising each wait so thousands of failed deliveries don’t return in the same second. Standard Webhooks adds a webhook-specific reason: sometimes the receiver failed because of the webhook load, and synchronised retries would recreate the spike. The theory is in timeouts, retries and backoff with jitter. How long to keep trying is a product decision, and providers differ by orders of magnitude.

The Standard Webhooks example waits 5 seconds, 5 minutes, 30 minutes, then 2, 5, 10, 14, 20 and 24 hours: about 75½ hours in all, so a weekend outage costs each event only ten attempts.

When retries run out, put the event in a dead-letter store the customer can see, with the last status and error, and offer replay of one event or a time range, as Stripe and GitHub do (delivery guarantees and DLQs).

Then disable endpoints that fail for days: they waste workers and bury real failures in noise. Shopify does it after 8 consecutive failures, and Slack when over 95% of attempts fail within an hour (apps under 1,000 events an hour are exempt). Standard Webhooks says to tell the customer through another channel, such as email. Never disable silently, and keep the events listable so the customer can catch up.

The receiver: verify, acknowledge, then work

A receiver’s HTTP handler should do four quick things and nothing else.

Verify the signature, on the raw bytes

Anyone can POST to your URL, and an unverified payment.succeeded is a free order. Providers sign each delivery with a per-endpoint secret, almost always with HMAC-SHA256:

• Stripe signs timestamp + "." + body and sends Stripe-Signature: t=…,v1=….
• Standard Webhooks signs id + "." + timestamp + "." + body, so the event ID is authenticated too.
• GitHub signs only the body (X-Hub-Signature-256: sha256=…). With no signed timestamp, your replay defence is the X-GitHub-Delivery ID.

Three details break real integrations:

1. Use the raw bytes. Verify the body exactly as received. A framework that parses and re-serialises the JSON first changes whitespace or key order, and the HMAC fails; Stripe’s docs single out Express apps that register express.json() before the webhook route.
2. Compare in constant time. A plain == returns faster the earlier the strings differ, leaking the signature byte by byte. Use hmac.compare_digest or crypto.timingSafeEqual.
3. Check the timestamp. Otherwise a captured, validly signed request can be replayed forever. Stripe’s libraries default to a 5-minute tolerance (0 disables the check), so keep clocks in sync with NTP. A retry days later still passes, because each attempt is re-signed with a fresh timestamp.

import base64, hashlib, hmac, time

TOLERANCE_S = 5 * 60
# All active secrets for this endpoint: two of them while a rotation overlaps.
KEYS = [base64.b64decode(s.removeprefix("whsec_")) for s in load_active_secrets()]

def verify(raw: bytes, headers) -> bool:
    msg_id, ts = headers["webhook-id"], headers["webhook-timestamp"]
    signed = f"{msg_id}.{ts}.".encode() + raw          # the exact bytes received
    sent = [s[3:] for s in headers["webhook-signature"].split() if s.startswith("v1,")]
    ok = any(
        hmac.compare_digest(base64.b64encode(hmac.new(k, signed, hashlib.sha256).digest()).decode(), sig)
        for k in KEYS for sig in sent
    )
    return ok and abs(time.time() - int(ts)) <= TOLERANCE_S

@app.post("/webhooks/payments")
def receive():
    raw = request.get_data()                           # bytes, before any JSON parsing
    if not verify(raw, request.headers):
        return "", 401
    db.execute(                                        # unique key = dedupe; the row is the work item
        "INSERT INTO webhook_inbox (event_id, body, status) VALUES (%s, %s, 'new') "
        "ON CONFLICT (event_id) DO NOTHING",
        (request.headers["webhook-id"], raw),
    )
    return "", 200                                     # duplicates get 200 too, so retries stop

Acknowledge fast, work later

Do the minimum inside the request: verify, persist, reply. GitHub fails deliveries that take over 10 seconds, Shopify allows 5 and Slack 3, and Stripe is blunt: return the 200 before marking the invoice paid in your accounting system. Traffic is bursty too (Stripe warns of spikes at the start of each month, when subscriptions renew), and a queue lets you work through a burst at your own pace, a form of backpressure.

Order matters: persist, then reply. Reply 200 and crash before writing the event anywhere, and it is gone, because the sender believes it was delivered. The CloudEvents webhook spec even has a status for “received, not yet processed”: 202 Accepted. In the code above, the inbox row is both the dedupe record and the work queue.

Dedupe on the event ID

Delivery is at least once: duplicates come from senders that timed out after you committed, manual resends, and your own reconciliation job overlapping with retries. A unique constraint on the event ID, the inbox pattern, makes repeats harmless; the reasoning is in idempotency.

Pick the right ID. Shopify sends one delivery per subscription, each with its own X-Shopify-Webhook-Id but a shared X-Shopify-Event-Id: dedupe on the first, correlate on the second. Stripe occasionally creates two Event objects for one change, so for effects that must never repeat, such as shipping, also key on the object ID and event type.

Out-of-order and missing events

Stripe’s docs give the classic example: creating a subscription can emit customer.subscription.created, invoice.created, invoice.paid and charge.created, in any order. Shopify warns that products/update can arrive before products/create. Design the receiver so that order does not matter:

• Re-fetch current state. Treat the event as a hint, “payment pay_8812 changed”, and GET the payment from the API. Whatever arrives last, you store the current truth. This is what thin events are built for. The cost is one API call per event, so watch your rate limit during bursts.
• Guard with a version. If the payload carries a per-object version or updated_at, apply it only if it is newer than what you hold. Don’t order by event creation time: Stripe’s created is in whole seconds, so distinct events can share a value. Shopify suggests its X-Shopify-Triggered-At header or the payload’s updated_at.
• Make states move forward only. A payment that is refunded should ignore a late succeeded.
• Fetch what is missing. If invoice.paid arrives for an invoice you have never seen, fetch the invoice rather than failing.

UPDATE payments
SET    status = 'refunded', version = 9, updated_at = now()
WHERE  id = 'pay_8812' AND version < 9;   -- 0 rows updated: an older event arrived late; ignore it

Reconciliation: the safety net you must build

Shopify’s guidance is unusually direct: “Your app shouldn’t rely on receiving data from Shopify webhooks.” Delivery isn’t always guaranteed, so run reconciliation jobs that periodically fetch what changed. Events go missing in ways retries can’t fix: your endpoint was down longer than the retry window, a handler acknowledged and then hit a bug, someone deleted a subscription, or, rarely, the provider failed to create the event at all (Stripe documents this case and sends a separate health event about it).

Providers give you the tools:

• Stripe lists events from the last 30 days; delivery_success=false returns only those that failed to reach an endpoint, and ending_before with the last good event ID anchors the list to your outage window.
• Shopify lets many queries filter on updated_at, so a job can fetch everything changed since its last run.
• GitHub has an API to list recent deliveries and redeliver failed ones, and its docs include sample scripts to run on a schedule.

A job that runs every 10 minutes bounds the delay for any missed event to about 10 minutes, for six small list calls an hour (more pages when many events changed). Run it through the same dedupe path as live events, because it will overlap with retries.

Security on both sides

For the receiver

Layer the defences; no single one is enough.

• HTTPS only. A signature proves who sent the body but doesn’t hide it; Stripe requires HTTPS in live mode.
• One strong secret per endpoint. Standard Webhooks asks for 24 to 64 random bytes, unique per endpoint, because a secret shared across customers lets one forge events for another.
• Rotate without downtime. During a rotation the sender signs every delivery with both the old and the new secret, and the receiver accepts either (that is why the code above loops over keys). Stripe lets the old secret live for up to 24 hours.
• Prefer asymmetric signatures where offered. Standard Webhooks defines v1a signatures with ed25519: receivers verify with a public key, so a leaked receiver config can’t forge events. The cost is more CPU per signature than HMAC.
• Keep secrets out of URLs. GitHub warns against putting API keys or credentials in the payload URL, because URLs end up in logs, dashboards and support tickets.
• IP allowlists are an extra layer. GitHub publishes its source ranges at GET /meta, and Stripe publishes a list too. Ranges change, so automate updates and keep the signature check.
• Mutual TLS is the strongest transport-level identity when both sides can manage certificates; see service-to-service auth and mTLS.
• Verified is not the same as true. For money and access, act on state you fetched from the provider’s API, not on fields in the body. With thin events you do that anyway.

For the sender

The sender’s biggest risk is server-side request forgery. Customers type in a URL and your servers call it, so a customer can type http://169.254.169.254/ (a cloud metadata service that hands out credentials) or an internal admin address. OWASP’s SSRF cheat sheet lists custom webhook URLs as a classic entry point. The defences, in short: send every delivery through an egress proxy, such as Stripe’s Smokescreen, that refuses internal addresses; run delivery workers in a subnet that can’t reach internal services; check the resolved IP at connection time, not only at registration, because DNS answers can change (DNS rebinding); and never follow redirects. Proxy vs reverse proxy covers egress proxies and fixed egress IPs in depth.

The second risk is being used as a weapon: an attacker points thousands of subscriptions at a victim and lets your retries flood it. The CloudEvents webhook spec answers with a validation handshake: the sender first sends an OPTIONS request with a WebHook-Request-Origin header, and the target must grant permission, optionally stating a WebHook-Allowed-Rate. Either way, cap the delivery rate for new or unverified endpoints.

Webhooks compared with other push styles

The last two columns are converging: Stripe can deliver events to Amazon EventBridge or Azure Event Grid, and Shopify to Google Pub/Sub or EventBridge, with no public endpoint or HMAC check. If you share a cloud with the provider, that is often the better pipe. Inside your own company, prefer a log or queue between services (publish-subscribe, event-driven architecture) and keep webhooks for the edge.

Going deeper: what breaks at scale

The recovery stampede

A receiver that comes back after an outage meets its own backlog. Suppose an endpoint normally gets 20 events a second and is down for 2 hours: 20 × 7,200 s = 144,000 events are waiting. If the sender releases them as fast as its workers allow, the receiver, which has only just recovered, falls over again, and the cycle repeats.

The fixes are on both sides. The sender caps each lane’s rate, ramps it up gradually after an endpoint recovers, and honours 429 with Retry-After (the CloudEvents spec says the sender MUST wait). The receiver puts a queue behind the endpoint, so a burst costs it disk rather than CPU. Jitter keeps retry timers from lining up. This is a retry storm in a new costume, and a per-endpoint rate limit is the cure.

Webhook storms

One bulk operation can emit a flood: a merchant imports 50,000 products and you send 50,000 products/update events; every subscription renews on the 1st of the month. Senders protect receivers with per-endpoint rate caps (Slack stops at 30,000 events per workspace per app per hour and sends an app_rate_limited event instead), and receivers protect themselves by subscribing only to the event types they handle, which both GitHub and Stripe recommend.

Poison events and silent loss

An event that always crashes the handler fails for the whole retry window, then lands in the dead-letter store. Because lanes don’t enforce order, it doesn’t block the events behind it. The opposite failure is quieter: a handler that returns 200 on every error makes the sender’s dashboard look perfect while events are dropped. Alert on processing failures in your worker, not only on HTTP status codes.

Testing and local development

Providers can’t reach localhost. Use a tunnel or the provider’s forwarder: stripe listen --forward-to localhost:4242/webhook streams events to your machine. Check your HMAC code against GitHub’s published test vector: the secret It's a Secret to Everybody and the body Hello, World! must give a signature starting 757107ea. Two traps: web frameworks that demand a CSRF token on every POST will reject webhooks unless you exempt the route, and new event types appear over time, so log unknown types and return 200 rather than failing.

Cheat sheet

How to say it in the interview

“When the payment completes, the payment service writes the status change and an outbox row in one transaction. A relay fans the event out to a queue per merchant endpoint, and workers POST it with a 10-second timeout. The body has an event ID, a type, a version and the payment ID, and it’s signed with HMAC-SHA256 over the ID, a timestamp and the body. Anything but a 2xx is retried with exponential backoff and jitter for three days, then dead-lettered for replay, and we email the merchant and disable endpoints that fail for days. Delivery is at least once and unordered, so we tell merchants to verify, dedupe on the event ID, reply fast, re-fetch the payment from our API, and reconcile against our list-events endpoint.”

How do you stop one slow customer endpoint from delaying everyone? A queue and a concurrency cap per endpoint, short timeouts, and a breaker that slows a lane when its error rate climbs. A hung endpoint then holds only its own few workers, and its backlog grows in its own queue.

Can you guarantee exactly-once delivery? No. A timeout leaves the sender unsure whether the receiver committed, so it must retry and duplicates are inevitable. We deliver at least once; the receiver makes the effect happen once by storing the event ID under a unique constraint in the same transaction as the work.

What if payment.refunded arrives before payment.succeeded? The receiver doesn’t trust arrival order. It either re-fetches the payment and stores the current state, or applies an update only when its version is higher, and the payment’s states move forward only, so a late succeeded can’t undo a refund.

The merchant’s server was down all weekend. What happens? Our retries cover about three days with growing gaps, so most events land once it is back, through a per-endpoint rate cap so the backlog doesn’t knock it over again. Anything past the window is in the dead-letter store for replay, and the merchant’s reconciliation job can list failed events from our API.

How does the merchant know the request really came from you? Each endpoint has its own secret, and we sign the event ID, timestamp and raw body. The merchant recomputes the HMAC, compares it in constant time and rejects timestamps older than five minutes. During a rotation we send two signatures, and we can offer ed25519 so merchants hold only a public key.

Where to go next

• Design a webhook delivery service: the whole sender at scale, with storage, scheduling, multi-tenancy and capacity numbers.
• Idempotency: why duplicates are inevitable and how to make repeated effects harmless.
• Effectively-once: idempotent consumers and the inbox pattern: the receiver’s dedupe table in depth.
• Transactional outbox: how the sender avoids losing events between commit and publish.
• Delivery guarantees: at-least-once, exactly-once and DLQs: the vocabulary behind retries and dead letters.
• Timeouts, retries and backoff with jitter: choosing retry schedules that don’t cause storms.
• Real-time connections: long polling, SSE and WebSockets: push to browsers and apps instead of servers.
• Proxy vs reverse proxy: egress proxies, fixed egress IPs and SSRF defences.
• Design a payment system: where payment webhooks and reconciliation fit in a full design.

References and further reading

1. Stripe docs: Receive Stripe events in your webhook endpointdocs.stripe.comRetries for up to three days in live mode, manual resend windows, no ordering guarantee, the t.body HMAC-SHA256 signature, the 5-minute default tolerance, secret rolling and the advice to return 2xx before doing work.
2. Stripe docs: How events work (thin and snapshot events)docs.stripe.comThin notifications that carry IDs versus snapshot payloads that can be stale, and why thin events are recommended for new integrations.
3. Stripe docs: Manage webhook endpointsdocs.stripe.comRecovering missed events with the List Events API (30 days, delivery_success=false), raw-body pitfalls, and rare event-generation failures.
4. GitHub Docs: Best practices for using webhooksdocs.github.comReply with a 2xx within 10 seconds, keep secrets out of payload URLs, allowlist the IP ranges from GET /meta, and use X-GitHub-Delivery against replays.
5. GitHub Docs: Validating webhook deliveriesdocs.github.comX-Hub-Signature-256 is an HMAC-SHA256 of the body alone, compared in constant time, with a published test vector.
6. GitHub Docs: Redelivering webhooksdocs.github.comGitHub does not retry failed deliveries automatically; you can redeliver anything from the past 3 days.
7. Shopify docs: Verify webhook deliveriesshopify.devA 1-second connect and 5-second total timeout, 8 retries over 4 hours, subscriptions deleted after 8 consecutive failures, and webhook ID versus event ID for deduplication.
8. Shopify docs: About webhooksshopify.devNo ordering guarantee within or across topics, and the instruction to run reconciliation jobs because delivery is not guaranteed.
9. Standard Webhooks specification v1.0.0github.comThe id.timestamp.body signature, HMAC versus ed25519, multi-signature rotation, thin versus full payloads, a multi-day retry schedule with jitter, status-code handling and SSRF advice.
10. CloudEvents: HTTP 1.1 Web Hooks for Event Deliverygithub.comHTTPS and POST only, no redirects, 202 for accepted-but-unprocessed, 410 and 429 handling, and the OPTIONS handshake that stops a sender being abused to flood a site.