Design a payment system

A payment system is the part of a marketplace that turns “Place order” into money in the bank. It does not move money itself — banks, card networks and NPCI do — but it picks the payment provider, remembers every attempt, tells the order service whether the customer paid, refunds, pays sellers and keeps the books. The design is hard because the network keeps lying to you. A timeout does not mean the charge failed. The customer taps “Pay” twice. A webhook arrives forty minutes late, or twice. The provider reports success but the bank credit is ₹24 short. At 30 million payments a day and about ₹1,200 an order, that is ₹3,600 crore of other people’s money a day, and a 0.01% double-charge rate is 3,000 angry customers every single day.

Understand the problem

First pin down which “payment system” you are building. A card network (Visa, RuPay) or NPCI for UPI switches messages between banks. A payment service provider (PSP) such as Stripe, Razorpay or Adyen connects merchants to those networks and banks. A merchant’s payment platform sits inside a large company and talks to several PSPs. You are building the third: the payments platform of a marketplace selling in India and abroad, accepting cards, UPI, netbanking and wallets through external PSPs.

Every payment has two flows on very different clocks: messages that authorise it in a second or two, and money that moves later, in batches, net of fees.

Functional requirements

1. Customers should be able to pay for an order with a card, UPI, netbanking or a wallet, through one of several external PSPs.
2. The order service should get one definitive outcome per payment — succeeded or failed — even when a PSP is slow or the customer approves on another device.
3. Customers should be able to get full or partial refunds to the method they paid with.
4. Sellers should be paid out on a schedule, net of the marketplace’s commission and fees.
5. Every money movement should be recorded and reconciled against PSP and bank records.

Non-functional requirements

• Correctness over availability for money. No double charge, no lost payment, no “failed” payment that actually succeeded — and when the outside world forces one, automatic repair within hours. Messages may arrive twice; effects happen exactly once.
• High availability. 99.99% for create and confirm — about 4.3 minutes a month (0.01% × 30 days × 1,440 minutes). One PSP’s outage must not stop payments.
• Latency we control, latency we don’t. Our own overhead under 150 ms at p99 for create and confirm, excluding PSP time; status reads under 50 ms. Assume PSP calls take about 1 s at p50 and several seconds at p99, and human steps (an OTP, a UPI PIN) ten seconds to a few minutes.
• Scale. About 30 million payments on a normal day, with a festive-sale opening at 15× the daily average (estimates below).
• Durability and audit. An acknowledged state change is never lost on failover. Every change is immutable, attributable and kept for years.
• Security. PCI DSS scope as small as possible; raw card numbers never stored outside a vault, in line with RBI card-on-file rules.
• Consistency. Strong for a payment’s state and amounts; eventual for dashboards and analytics.

Out of scope

• Being a PSP or acquirer. We never connect to Visa or NPCI directly; PSPs do.
• Ledger internals. Accounts, postings and balances belong to the double-entry ledger; we only hand it events.
• Stored-value wallets. Top-ups and wallet-to-wallet transfers are a digital wallet problem.
• Cart, pricing and checkout orchestration. The order and its amount arrive from checkout.
• Fraud models. We call a risk service before confirming; its internals are separate.
• Subscriptions and mandates (UPI AutoPay, e-mandates), cross-border FX and disputes — touched on in the follow-ups.

The set-up

Plan the approach

Say the shape early: “a state machine around unreliable external calls, with a ledger and reconciliation behind it.” Build the five requirements simply, then spend the time on failure handling — duplicates, timeouts, asynchronous approvals, atomic recording, reconciliation, provider outages and card data.

Core entities

• Payment (intent). One per order payment: pay_7Hq2, order ID, amount in paise as an integer (₹1,249.50 is 124950, never a float), currency, status, captured and refunded amounts, and a version for safe updates.
• Attempt. One per try with a method and a PSP: att_9, PSP, method, the PSP’s reference, status, error code. Normally at most one attempt per payment succeeds.
• Refund. Payment ID, amount, reason, status, PSP refund reference.
• Idempotency record. Caller scope plus key, a hash of the request, progress, and the saved response.
• Webhook event. PSP, the PSP’s event ID, payload, processed time — the inbox.
• Payout. Seller, period, amount, status and the bank’s transaction reference (UTR).
• Settlement row. One line from a PSP settlement file or bank statement, loaded for reconciliation.

API

Create a payment — server to server from the order service, so the amount never comes from the customer’s device:

POST /v1/payments
Idempotency-Key: 01JB8Z3K7Q4MZ0F3T9W2ABCDEF
Content-Type: application/json

{
  "order_id": "ord_8812",
  "amount": 124950,
  "currency": "INR",
  "customer_id": "cus_42",
  "capture": "automatic"
}

{ "id": "pay_7Hq2", "status": "requires_payment_method", "amount": 124950, "currency": "INR" }

Confirm with a method — from the app; for a card it sends a vault token, never the card number:

POST /v1/payments/pay_7Hq2/confirm
Idempotency-Key: 01JB8Z4C1N6PXJ2V0R5YGHIJKL

{ "method": { "type": "upi", "flow": "intent" } }

{
  "id": "pay_7Hq2",
  "status": "requires_action",
  "attempt_id": "att_5kx",
  "next_action": {
    "type": "upi_intent",
    "uri": "upi://pay?pa=shopco@psp&pn=ShopCo&tr=att_5kx&am=1249.50&cu=INR"
  }
}

It can also return a 3-D Secure challenge, processing, succeeded, or requires_payment_method again with a decline code.

Wait for the outcome — a long poll that returns when the status changes, or after 25 seconds:

GET /v1/payments/pay_7Hq2?wait=25

Refund, with the return’s own ID as the key:

POST /v1/payments/pay_7Hq2/refunds
Idempotency-Key: rfd-ret_3391

{ "amount": 50000, "reason": "item_returned" }

Events such as payment.succeeded, payment.failed and refund.succeeded go to a payment-events topic with an event ID and the payment’s version. Signed PSP webhooks arrive at POST /internal/webhooks/{psp}.

Back-of-envelope estimates

These numbers decide the shard count, the connection pools and the case for long polling (method: back-of-envelope estimation).

Payments. 30M a day: 30,000,000 ÷ 86,400 ≈ 350 payments/s on average. A festive-sale opening runs at about 15× that: 347 × 15 ≈ 5,200 payments/s at peak.

PSP calls. Declines and method switches give about 1.25 attempts per payment: 5,200 × 1.25 ≈ 6,500 attempts/s at peak.

Database writes. Each payment needs one transaction to create it, and each attempt about three (record it, record the result, apply a webhook): 1 + 1.25 × 3 ≈ 4.75 transactions per payment, so 5,200 × 4.75 ≈ 24,700 write transactions/s at peak. Budget about 4,000 per primary, because each commit also waits for a synchronous replica: 24,700 ÷ 4,000 ≈ 6.2, so 8 shards, each near 3,100/s at peak.

Calls in flight. By Little’s Law, in-flight = rate × time. At 6,500 attempts/s and 1.5 s average PSP time: about 10,000 concurrent PSP calls. If the PSP carrying half the traffic slows to 10 s, its share alone is 3,250 × 10 = 32,500 in flight — enough to exhaust any uncapped pool.

Customers on the phone. About 60% of payments wait about 30 s for a person (3DS or a UPI PIN): 5,200 × 0.6 × 30 ≈ 94,000 payments waiting. Polling every 2 s would be 47,000 reads/s; a 25-second long poll needs 94,000 ÷ 25 ≈ 3,800 reads/s.

Storage. About 4 KB of durable rows per payment (payment, attempts, state history): 30M × 4 KB = 120 GB a day, about 44 TB a year before replication. Keep 90 days hot — about 11 TB, 1.4 TB per shard — and archive the rest. Idempotency records add 30M × 2 calls × 2 KB = 120 GB a day; kept 72 hours, that is 360 GB.

Webhooks. About two per attempt: 6,500 × 2 ≈ 13,000 webhooks/s at peak.

Late answers. One large Indian gateway documents that under 0.5% of its payments are authorised late, after it had stopped waiting. At 30M a day that is up to 30,000,000 × 0.005 = 150,000 payments a day whose first answer was not the final one — routine, not an incident.

High-level design

Build one requirement at a time. By the end, the design works end to end in one region at moderate traffic.

Customers should be able to pay for an order

The order service creates a payment; the app confirms it with a method. Card details go straight into fields hosted by a small card vault, which returns a token. The payment service writes an attempt, picks a PSP and calls it through a PSP adapter that translates our model into that PSP’s API.

1. The order service calls POST /v1/payments with a key derived from the order; the payment starts in requires_payment_method.
2. For a card, the app posts the card details to the vault and gets a token such as tok_v_93xb. UPI and netbanking need no token.
3. The app calls confirm through the gateway.
4. The payment service writes attempt att_9, picks a PSP, and the adapter calls it with att_9 as the PSP’s idempotency key and merchant reference.
5. It stores the answer and returns the new status.

Store payments in a relational database such as PostgreSQL, sharded by order ID, with the shard number embedded in the payment ID so later calls find it directly. You need multi-row transactions (payment, attempt, idempotency record and outbox row together), conditional updates and unique constraints far more than raw write throughput. Distributed SQL such as Spanner or CockroachDB also works if you prefer one logical database to manual sharding.

The order service should get one definitive outcome

A PSP answers in the synchronous response, later through a webhook, or when we ask; we use all three. A webhook receiver checks the signature, stores the event in an inbox table and answers 200. A status resolver asks the PSP about attempts in flight too long. Either way, the payment service applies the result through one state machine and, in the same transaction, writes an outbox row that an outbox relay publishes to Kafka for the order service.

1. Synchronous reply — many card payments finish within the confirm call.
2. Webhooks — for UPI approvals, 3DS completions, late authorisations and refunds.
3. Inbox — a unique constraint on the PSP’s event ID makes a duplicate delivery a no-op.
4. Status queries — the resolver asks about attempts still in flight after a few seconds.
5. Outbox — every state change writes an event row; the relay publishes it.
6. Order service — consumes payment.succeeded or payment.failed by event ID and confirms or cancels the order.
7. App — long-polls the payment and shows the result as soon as it changes.

The rule that keeps this sane: only the payment service changes payment state, and only through allowed transitions. Webhooks, the resolver and the reply are just evidence.

Customers should be able to get refunds

A refund is a new money movement with its own ID, key and states (pending → processing → succeeded | failed). The returns service uses the return’s ID as the key, so a retried return never refunds twice. A conditional update — “add ₹500 to refunded only if refunded + 500 ≤ captured” — stops two concurrent partial refunds from exceeding what was paid. The refund goes to the PSP that captured the payment, back to the original instrument, and its outcome arrives the same three ways; some methods take days.

Sellers should be paid out on schedule

A payout service runs daily. For each seller it reads the payable balance from the ledger, holds back amounts still inside the return window, and sends a payout through a bank’s payout API (IMPS or NEFT in India) with the payout ID as the idempotency key, recording the bank’s UTR when it completes. A payout that times out is treated like a charge that times out: unknown, then resolved by asking. With 500,000 sellers and 40% on daily payouts, that is about 200,000 payouts a day — small, but every one is money leaving.

Every money movement should be recorded and reconciled

Each money event — captured, refunded, settled, paid out — becomes an immutable double-entry journal entry in the ledger service, which owns balances such as “owed to seller 4471” (see the double-entry ledger design). Each morning a reconciliation engine matches PSP settlement files and bank statements against our attempts and the ledger. Anything that does not match becomes an exception, fixed by a rule or routed to operations.

That is a working payment platform. Everything that makes it hard is in the deep dives.

Deep dives

How do we make sure a customer is never charged twice?

Duplicates come from every layer: a double tap on “Pay”, an app retry when the phone hops from Wi-Fi to 4G, the order service retrying after its own timeout, a worker that crashes after calling the PSP, a redelivered webhook. Each is a different door, and each needs a lock.

Bad: check before you charge. “No successful attempt yet? Then charge.” Two requests 20 ms apart both read “no” and both charge — the read-then-act race behind the lost update problem.

Good: idempotency keys behind a unique constraint. The caller generates one key per logical operation and sends it on every retry. The server inserts (caller scope, key) in the same transaction as the payment’s first write; the unique constraint is the lock, so of two concurrent requests exactly one insert succeeds. Then:

• Key finished → replay the saved status code and body, nothing else. Stripe saves the first result once execution begins and returns it on every retry.
• Key in progress → 409 Conflict, “retry shortly”. Making the duplicate wait would hold a connection for a 10-second PSP call.
• Same key, different request → reject it. Compare a stored hash of method, path and body, as Stripe compares parameters.
• Nothing saved for validation errors or a lost race, so those retries simply run again.

CREATE TABLE idempotency_keys (
  scope         text        NOT NULL,   -- the caller: 'svc:checkout', 'app'
  key           text        NOT NULL,
  request_hash  bytea       NOT NULL,   -- SHA-256 of method, path, body
  state         text        NOT NULL,   -- 'in_progress' or 'finished'
  locked_until  timestamptz,            -- a crashed holder's lock expires
  response_code int,
  response_body jsonb,
  payment_id    text,
  created_at    timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (scope, key)
);

Scope keys to the caller, never globally. Keep them long enough: Stripe allows pruning after 24 hours; a reaper at about 72 hours, as Brandur Leach suggests, still covers a bug shipped on Friday and fixed on Monday. Read them from the primary, as Airbnb’s payments library does: a replica 200 ms behind will cheerfully say “never seen this key”.

Great: a chain of keys, end to end. The caller’s key protects our API, not our call to the PSP. So add a rule: every external call carries an ID we created and committed before the call. Attempt ID att_9 goes to the PSP as its idempotency key where supported, and always as the merchant reference — Brandur’s reference implementation likewise sends Stripe its own ID, not the client’s key. A worker that crashes mid-call is now harmless: its successor resends with the same att_9 and the PSP replays its first answer; without PSP idempotency, it queries by att_9 first. A new attempt ID is minted only after the previous attempt has definitively failed.

Server-to-server callers derive keys from their data — ord_8812:pay:1, so a restarted checkout finds the same payment. Apps generate a random key per tap.

What you pay: an indexed write per call (the 120 GB a day), clients that handle 409, and discipline. Every error must be classed as retryable or not; Airbnb’s write-up spells out the stakes — mislabel a transient error and the payment fails forever, mislabel a dangerous one and you can pay twice. When in doubt, do not retry the money call; ask.

How do we handle a PSP call that times out?

The adapter sends the charge for att_9 and waits 8 seconds. Nothing. Either the request never reached the PSP, or it was declined, or it succeeded and the reply was lost. From our side all three look identical — the most important idea in payments: a timeout is an unknown, not a failure.

Bad: treat the timeout as a failure. Show “Payment failed, try again”. If it was the third case, the bank has already debited ₹1,249.50, the order is canceled, and the customer pays again — the familiar “money debited but order failed” complaint. One large Indian gateway documents that when a bank does not answer it marks the payment failed after 10 minutes, keeps polling the bank for three days, and that under 0.5% of payments end up authorised late: up to 150,000 a day at our volume.

Worse: fail over to another PSP. Retry as att_10 on PSP B, and if it was the third case, the customer is charged by both. No idempotency key helps: PSP B has never heard of PSP A.

Good: mark the attempt unknown and ask. The attempt becomes unknown; the payment stays processing. The resolver queries the PSP by att_9 at 5 s, 15 s, 45 s, 2 min and 5 min, then every 15 minutes for two hours, then hourly up to 72 hours — roughly the window in which PSPs keep chasing banks. “Succeeded” or “failed” goes through the state machine; “not found” a minute after the call means the request never arrived, so resending with the same att_9 is safe. A webhook can settle it at any moment. Most unknowns resolve within a minute; if the PSP itself gives up and reports “failed”, apply that, but keep listening — a late authorisation can still arrive, and takes the late-success path below. Add jitter and a per-PSP cap on queries: a PSP carrying 3,250 calls a second that times out 30% of them creates about 1,000 unknowns a second, and querying them all five seconds later is a retry storm of your own making.

Great: a state machine that makes illegal moves impossible. Every transition is a conditional update on status and version, so a late message cannot overwrite a newer fact:

UPDATE payments
SET    status = 'succeeded', captured_amount = 124950, version = version + 1
WHERE  id = 'pay_7Hq2'
  AND  status IN ('processing', 'requires_action')
  AND  version = 41;
-- 0 rows updated: another writer moved it first. Reload, then decide.

This is optimistic locking applied to money. The states follow Stripe’s documented PaymentIntent lifecycle: a decline sends the payment back to “needs a method”, a challenge parks it in “needs action”, and with separate capture it waits in “requires capture”.

Terminal states — failed, canceled, refunded — are never left; a contradicting fact triggers a compensating action. Say the payment expired at 15 minutes, the order was canceled, and at 40 minutes a webhook says att_9 succeeded. Do not flip “failed” to “succeeded” — the order service already acted. Record the success on the attempt, emit payment.late_success, and let a rule reinstate the order if stock remains or refund within minutes. Meanwhile tell the customer the truth: after two minutes, “We’re confirming with your bank. If money was deducted, your order will be confirmed or the amount refunded” — which also stops a second payment.

What you pay: a resolver, more states to test, and customers who sometimes see “pending” for a minute. What you get: no blind retries and a definitive answer for every payment.

How do we handle payments the customer approves somewhere else?

Most Indian payments need the customer to act outside our app — an OTP or biometric for a card, a UPI PIN, a netbanking login — and the answer comes back by a different road.

3-D Secure. When the issuer wants to authenticate the cardholder, confirm returns requires_action with a challenge; the customer completes it on the issuer’s page or in an SDK and returns to our URL. Two traps. The return redirect is not proof of payment — its parameters can be stale or forged — so the server confirms with the PSP first. And customers close the tab right after the OTP, a documented cause of late authorisation: the bank debits, we never see the redirect, and only the webhook or resolver brings the news.

UPI intent. Our server asks the PSP for a UPI transaction with our attempt ID as its reference and gets back a upi://pay link. The app opens it in the customer’s UPI app; the customer enters their PIN; their bank debits the account and NPCI switches the payment to the PSP’s bank; the PSP sends us a webhook. The older collect flow, where the customer types a UPI ID and approves a pop-up request, is deprecated per NPCI guidelines from 28 February 2026 for most merchants, with listed exemptions — so intent and QR are the default. Keep the flow as data on the attempt, and switching is a configuration change.

Expiry. Every attempt waiting for a person gets a deadline, say 10 minutes. When it passes, the resolver asks the PSP once more and, if nothing was paid, cancels the attempt where the PSP allows and marks it failed. Races still happen — a customer gives up on UPI, pays by card, and the UPI approval lands later. Two successful attempts on one payment is a detectable state: keep the first, refund the second automatically.

Webhooks are late, duplicated and out of order. Stripe documents retries for up to three days with exponential backoff, no ordering guarantee and occasional duplicates, and advises logging event IDs. So the receiver:

1. Verifies the signature on the raw body before parsing anything.
2. Inserts the event into an inbox with a unique key on (PSP, event ID); duplicates are dropped.
3. Answers 200 within about 100 ms, before real work, so the PSP does not redeliver.
4. Applies events through the state machine, where only forward moves succeed: a payment.failed after payment.captured changes nothing and raises an alert.
5. Re-fetches the object from the PSP’s API when an event is thin or surprising.

This is the inbox pattern on the receiving side of webhooks.

Long polling keeps the waiting cheap: 94,000 waiting payments at peak cost about 3,800 reads a second instead of 47,000. The API node subscribes to a change notification for the payment and answers the moment it fires; server-sent events also work (see real-time connections).

How do we record each money movement exactly once?

One confirm touches four systems: our database, the PSP, the ledger and the order service. No transaction spans them, and a PSP will never take part in two-phase commit with you.

Bad: one database transaction around everything. Each open transaction holds a connection and row locks for the whole PSP call: about 10,000 open transactions at 6,500 attempts a second, 32,500 when a PSP slows down, and the pool is gone. And if the commit fails after the PSP said “yes”, the charge exists and our record does not. Airbnb’s payments team made it a rule: no network calls inside database transactions, and no database work during the network call.

Bad: dual writes. Commit the state, then publish to Kafka, then call the ledger. A crash in between leaves an order never confirmed or a ledger missing an entry; publish first and you may confirm an order for a payment that rolls back.

Good: atomic phases around the external call. Split confirm into short local transactions around one foreign call — Airbnb’s pre-RPC, RPC and post-RPC phases, Brandur’s atomic phases with recovery points:

• Tx 1, about 5 ms. Insert the idempotency record and attempt att_9 (pending); move the payment to processing. Commit.
• The PSP call, 0.5–10 s. No transaction open; att_9 is the PSP’s key.
• Tx 2, about 5 ms. Write the attempt’s result and the payment’s new state with a compare-and-set, plus one outbox row per event. Commit.

Every crash point now has a defined recovery.

Great: outbox, idempotent consumers, and the ledger as a consumer. The outbox row from Tx 2 is the only way the rest of the company learns of the change. A relay — polling the table or tailing it with change data capture — publishes each row at least once (transactional outbox), and every consumer dedupes by event ID. The ledger posts a balanced journal entry whose unique source key is the event ID, so a redelivery changes nothing: exactly-once effects from at-least-once delivery.

For a ₹1,249.50 order with a 10% commission and a 2% PSP fee (ignoring GST), the ledger receives:

PSP receivable is debited ₹1,249.50 and credited ₹1,224.51 + ₹24.99 = ₹1,249.50, so it returns to zero once the money lands — if not, reconciliation notices. Storage, balances and hot accounts belong to the double-entry ledger design.

What you pay: the ledger trails the payment by a second or so, and you run a relay. Writing ledger rows inside Tx 2 on the same shard is stronger but couples two systems with different scaling needs; choose it only if balances must be exact at every instant.

How do we prove our records match the money?

Everything so far makes our records probably right. Reconciliation makes them provably right by comparing them with records we did not write — as Shopify’s payments team puts it, checking individual charges and refunds and aggregates such as unpaid balances against their financial partners’ records.

There are three sources and two levels of matching:

• Our records: attempts, refunds, payouts and ledger entries.
• PSP settlement reports: one row per transaction — our attempt ID (sent as the merchant reference for exactly this), gross, fee, tax and net — grouped into batches.
• Bank statements: one credit per settlement batch with its UTR, one debit per payout.

At the transaction level, attempts match PSP rows by attempt ID; at the batch level, each batch’s net total matches a bank credit by UTR and amount.

Bad: monthly spreadsheets. At 40 million rows a day, impossible — and a bug found after 30 days has already touched 30 days of money.

Good: a daily automated batch. Normalise each file (paise, one time zone, our IDs), join on attempt ID — a hash join over 40 million rows takes minutes — and turn every difference into a typed exception:

• Success at the PSP, failure with us — a late success. Reinstate the order or refund automatically.
• Success with us, missing at the PSP — often a cut-off difference. Wait two days, query the PSP, then escalate.
• Amount mismatch — a partial capture, a fee outside the contract, a rounding bug.
• Duplicate — two successful attempts for one payment. Refund one.
• Batch mismatch — net total differs from the bank credit. Hold and investigate.

Great: continuous reconciliation with automatic fixes and ageing. Pull PSP report APIs during the day so a late success is caught in hours. Record every exception even when a rule fixes it — Shopify records each anomaly, remediates where it can, and prefers preventing anomalies to fixing them. Give each type an owner, a deadline and a rupee value, and watch the auto-match rate, open exceptions by age, and unmatched rupees.

The numbers make the case for automation. About 40 million rows a day (37.5 million attempts plus refunds and payouts) at a 99.95% match rate leaves 40,000,000 × 0.0005 = 20,000 exceptions a day, mostly timing differences and late successes. If rules clear 95%, about 1,000 a day reach people — a small team’s work. Without the rules, nobody could keep up.

How do we keep taking payments when a PSP fails?

Our target allows 4.3 minutes of downtime a month; a PSP at 99.9% has 43 minutes of its own, so one provider can never meet it. And PSPs seldom fail cleanly: more often one bank’s netbanking breaks, one issuer starts declining, or UPI through one partner bank slows down.

Bad: one PSP. Simple, and its bad day is yours.

Good: a primary and a backup, switched by hand. Someone notices the graph and flips a flag — ten to twenty minutes of lost sales, and all traffic moves for a fault that affects one issuer.

Great: success-rate routing with scoped circuit breakers and bulkheads.

• Route on evidence. The router scores the PSPs that support the attempt’s segment — method, card network, issuing bank (from the BIN), amount band — on rolling 10-minute success rate, latency and cost. About 5% of traffic explores the others so their scores stay fresh.
• Break circuits narrowly. A circuit breaker per (PSP, method, issuer) opens when success falls well below baseline or timeouts spike, then half-opens with probes. Shopify scopes its breakers by merchant country for the same reason: a local outage should not trip everything.
• Cap what each PSP can hold. A PSP that slows to 10 seconds would soak up 32,500 in-flight calls. A bulkhead of, say, 5,000 per PSP stops one slow provider starving the rest.

Put numbers on it. Normally PSP A carries 70% of traffic at 94% success and B carries 30% at 91%: 0.7 × 94 + 0.3 × 91 ≈ 93.1% overall. If A falls to 60% and nobody reacts, the blend drops to 0.7 × 60 + 0.3 × 91 ≈ 69.3%. Over ten minutes at 5,200 payments a second, that is (93.1 − 69.3)% × 5,200 × 600 ≈ 740,000 extra failed attempts. With the breaker holding A to 5% probes and B at 89% under the extra load: 0.05 × 60 + 0.95 × 89 ≈ 87.6%, and the loss shrinks to (93.1 − 87.6)% × 5,200 × 600 ≈ 170,000.

Two rules keep routing from creating double charges:

• Route new attempts, never in-flight ones. An unknown attempt stays with its PSP until resolved. Only a definitive soft failure (an issuer or bank error) frees the payment for a new attempt elsewhere; hard declines such as “insufficient funds” or “card reported stolen” are never retried on another PSP.
• Make saved cards portable. A card saved as one PSP’s token can only be charged through that PSP. Portability needs network tokens held for the marketplace, or a vault that can send card data to any PSP — the next deep dive.

How do we keep card numbers out of most of our systems?

PCI DSS applies to every system that stores, processes or transmits card numbers (PANs), and to systems connected to them. The PCI Security Standards Council’s tokenisation guidance states the trade: tokens can shrink scope, but any system that can turn a token back into a PAN stays in scope, and the vault becomes your most attractive target.

Bad: card numbers flow through the stack. Gateway, payment service, Kafka, logs and traces all see them. Everything is in scope, and one debug log line is a reportable incident.

Good: PSP-hosted card fields. The card goes into the PSP’s iframe or SDK and we receive its token; our scope nearly vanishes. The cost is lock-in: that token works only with that PSP, so routing stops working for saved cards.

Great: a small vault of our own, plus network tokens. The vault is one service in its own network segment, with its own accounts, keys in an HSM or cloud KMS, mTLS everywhere and a separate audit trail (see encryption and key management). The app posts card details into vault-hosted fields and gets back tok_v_93xb, the last four digits and the issuer. For a card attempt, the adapter sends the request through the vault’s forwarding proxy, which swaps the token for card data and passes it to the chosen PSP over mTLS. The payment service never sees a PAN, yet any PSP can be used.

India goes further. The RBI’s card-on-file tokenisation circular says that from 1 January 2022 nobody in the card payment chain except card issuers and networks may store actual card data; others may keep only the last four digits and the issuer’s name for tracking and reconciliation. Saved cards become tokens from the network’s or issuer’s token service, created with explicit consent and an additional factor of authentication, and unique to the card, the token requestor and the merchant. So for Indian cards the vault holds network tokens and keeps a PAN only in memory for the transaction in flight; elsewhere, where rules allow, it may also keep encrypted PANs.

UPI and netbanking carry no card data, and the UPI PIN never leaves the customer’s UPI app. UPI IDs and bank details are still personal data: encrypt them and mask them in logs.

What you pay: one high-security service to run and audit, and a forwarding hop of a few milliseconds. What you get: an audit of one small service instead of hundreds, and saved cards that work with any PSP.

The final design

The hot path is the gateway, the payment service, its database and the PSP adapters. Outcomes arrive through the resolver and leave through the outbox; the ledger, payouts and reconciliation run off the hot path.

Follow one ₹1,249.50 UPI or card payment through it:

1. Create. The order service calls POST /v1/payments with the key ord_8812:pay:1; the payment lands on its shard.
2. Tokenise. For a card, the app posts the card details to the vault and receives a token; for UPI or netbanking it just picks the method.
3. Confirm. The app calls confirm through the gateway with its own key. Tx 1 commits the key, attempt att_9 and the move to processing.
4. Route and charge. The router picks a PSP from success rate, cost and open breakers for this method and issuer; the adapter calls it with att_9 as key and reference, through the vault’s proxy for cards.
5. Learn the outcome. The synchronous reply, a webhook from the inbox, or the resolver’s status query — whichever comes first is applied; the rest are no-ops.
6. Publish. Tx 2 writes the state and an outbox row; the relay publishes payment.succeeded; the order service confirms the order and the app’s long poll returns.
7. Book and pay out. The ledger posts a balanced entry per money event; each morning the payout service pays sellers from their payable balances, keyed by payout ID.
8. Reconcile. The recon engine matches settlement rows and bank credits against attempts and the ledger; rules fix late successes and duplicates, and the rest go to operations with an owner and a deadline.

Check it against the requirements. No double charge: keys at our API, attempt IDs at the PSP, forward-only states, duplicate detection. No lost payment: unknowns are resolved by asking; late successes are caught by webhooks, the resolver or the next file. Availability: stateless services, replicated shards, several PSPs behind breakers. Latency: two short transactions and a routing decision add tens of milliseconds. Audit and security: immutable history, a double-entry ledger, three-way reconciliation, card data in one vault.

What interviewers look for by level

Follow-up questions

What if a customer is charged twice anyway?

It will happen — a UPI approval lands after a card payment, or a PSP confirms one charge twice. Detect it three times over: the state machine sees a second successful attempt and emits payment.duplicate_success; webhooks and the resolver catch it minutes later if that missed; reconciliation catches the rest next morning. Each detection triggers a refund of the later attempt keyed dup-<attempt_id>, so three detections still make one refund, plus a message to the customer. Track the duplicate rate per PSP and method; a rise usually means a bug.

What if a PSP does not support idempotency keys?

Put the attempt ID in the field every PSP has — the merchant order ID or reference — and query before you resend: after any ambiguous failure, look up that reference and resend only on a definitive “not found”. If the PSP cannot be queried by reference either, never retry automatically: mark the attempt unknown, let webhooks and the settlement file decide, and route new payments to a better-behaved PSP. Brandur Leach gives the same advice for foreign calls without idempotency: when you cannot know a retry is safe, take the conservative path.

What if the region hosting the payments database fails?

Each shard commits only after a synchronous replica in another availability zone has the write, so a zone failure loses nothing and a replica takes over within a minute. For a whole region, keep asynchronous replicas elsewhere and accept a short window of possible loss, repairable from the PSPs’ records because every attempt carried our ID. After failover, the resolver re-queries every attempt that was in flight. Never let two regions write the same payment; for active-active, give each customer a home region or cell. See multi-region and active-active and disaster recovery.

What if a customer disputes a charge?

A dispute (chargeback) arrives as a PSP webhook with a response deadline. Create a dispute record, move the amount out of the seller’s payable balance so it is not paid out, and gather evidence — delivery proof, invoice — before the deadline. A win releases the hold; a loss posts the debit and the fee. It is the same machinery again: own IDs, idempotent webhooks, ledger entries and reconciliation against dispute reports.

What if the sale peak is 50×, not 15×?

At 350 × 50 = 17,500 payments a second, three things break first: PSP capacity (each PSP has contracted throughput), database writes (17,500 × 4.75 ≈ 83,000 transactions a second, against the 32,000 that eight shards are budgeted for — about 21 shards at 4,000 each) and the order pipeline in front. Start upstream with admission control and a waiting room at checkout, as in a flash sale, so only as many customers reach payment as the PSPs can take. Agree peak capacity with each PSP and enforce it with per-PSP rate limits in the router. Add shards weeks ahead, and defer payouts and reconciliation runs until the peak passes.

How to say it in the interview

“I’ll design the payment platform of a large marketplace. We don’t move money ourselves; we orchestrate several PSPs for cards, UPI, netbanking and wallets, refund, pay sellers and prove our books are right. Thirty million payments a day, about 5,200 a second at a sale peak, and correctness before availability.

Each payment is an intent with an explicit state machine in sharded SQL with synchronous replicas. Every write takes an idempotency key behind a unique constraint, with a stored request hash and response, and 409 for a concurrent duplicate. Each try is an attempt with our own ID, committed before the call and passed to the PSP as its idempotency key — so a retry at any layer can’t charge twice.

The key insight: a timeout is an unknown, not a failure. The attempt stays in processing while a resolver queries the PSP with backoff, and webhooks — deduped in an inbox, applied with compare-and-set — settle it. I never fail an unknown attempt over to another PSP. 3-D Secure and UPI intent work the same way: the customer approves elsewhere, the app long-polls, and only a signed webhook or status query counts.

Confirm runs as atomic phases: record the attempt, call the PSP with no transaction open, then write the state and an outbox row together. The order service and a double-entry ledger consume the events idempotently, and daily three-way reconciliation against settlement files and bank statements catches late successes, duplicates and missing money.

For resilience, success-rate routing across PSPs with narrow circuit breakers and bulkheads. For security, card numbers live only in a small vault that holds network tokens under RBI’s card-on-file rules, so almost nothing else is in PCI scope.”

Concepts used in this design

• Idempotency — keys and stored responses at every layer.
• Webhooks — signed, late, duplicated, unordered PSP events.
• Transactional outbox — events committed with the state change.
• Effectively-once and the inbox pattern — deduplicating webhooks and events.
• Delivery guarantees — exactly-once effects from at-least-once delivery.
• Distributed transactions: 2PC vs sagas — why confirm is a forward-recovering saga.
• ACID transactions — the short transactions around the PSP call.
• Optimistic vs pessimistic locking — compare-and-set on state and version.
• Database sharding — eight shards for the write load.
• Timeouts, retries and backoff with jitter — the resolver’s query schedule.
• Circuit breaker — narrowly scoped breakers per PSP, method and issuer.
• Bulkheads — capping in-flight calls per PSP.
• Real-time connections — the long-polling payment page.
• Change data capture — one way to run the outbox relay.
• Encryption and key management — the vault’s keys.
• Design a double-entry ledger — where events become balanced postings.

References and further reading

1. Stripe API reference: Idempotent requestsdocs.stripe.comClient-generated keys up to 255 characters; the first status code and body are saved and replayed on retry; parameters are compared with the original; keys can be pruned after 24 hours; validation failures and concurrent conflicts are not saved.
2. Stripe: How PaymentIntents and SetupIntents workdocs.stripe.comThe intent states (requires_payment_method, requires_action, processing, requires_capture, succeeded, canceled) and the rule that a failed attempt returns the intent to requires_payment_method.
3. Stripe: Receive Stripe events in your webhook endpointdocs.stripe.comLive-mode delivery retried for up to three days with exponential backoff, no ordering guarantee, occasional duplicates, deduplication by event ID, signature checks and a fast 2xx.
4. Avoiding double payments in a distributed payments system (Airbnb Engineering)medium.comThe Orpheus library: pre-RPC, RPC and post-RPC phases, no network calls inside database transactions, retryable versus non-retryable errors, idempotency data on the sharded primary.
5. Implementing Stripe-like idempotency keys in Postgres (Brandur Leach)brandur.orgA unique key per user, stored request parameters, 409 for a locked key, atomic phases and recovery points, passing your own ID to the PSP, and the completer and reaper (about 72 hours) processes.
6. 10 tips for building resilient payment systems (Shopify Engineering)shopify.engineeringLow timeouts, circuit breakers scoped per country, Little’s Law, ULID idempotency keys (50% faster inserts than UUIDv4), and reconciliation that records every anomaly.
7. Razorpay docs: Late payment authorisationsrazorpay.comWith no bank response a payment is marked failed after 10 minutes, the bank is polled for 3 days, and under 0.5% of payments are authorised late; closing the window after the OTP is a common cause.
8. Razorpay docs: S2S UPI Collect Flowrazorpay.comPer NPCI guidelines the UPI collect flow is deprecated from 28 February 2026 for most merchants, with listed exemptions; merchants move to UPI intent or QR.
9. RBI circular: Tokenisation – Card Transactions: Permitting Card-on-File Tokenisation (CoFT) Services (7 September 2021)rbi.org.inFrom 1 January 2022 only card issuers and networks may store card data; others may keep the last four digits and issuer name; a token is unique per card, token requestor and merchant; tokenisation needs consent with an additional factor.
10. PCI DSS Tokenization Guidelines (PCI Security Standards Council information supplement)pcisecuritystandards.orgTokens can shrink PCI DSS scope, any system that can retrieve a PAN stays in scope, and the card data vault is the most attractive target.