Idempotency

An operation is idempotent if doing it twice has the same effect as doing it once. It is what makes a retry safe: when a request times out and you cannot tell whether it worked, you can simply send it again. Reads, PUT and DELETE get it almost for free; creates and payments need help, usually a client-chosen ID or an idempotency key that the server remembers.

Picture it first: the lift button and the numbered cheque

Press a lift’s call button and the lift comes. Press it ten more times and nothing extra happens. The button sets a fact, “someone is waiting on floor 4”, and setting a fact again changes nothing. That is natural idempotency.

Now pay a contractor ₹20,000 by cheque number 004512 and hear nothing for a week. If that cheque reaches your bank twice, say deposited at two branches, the bank pays once: its register already shows 004512 as paid. But if you worry and write a new cheque, 004513, the bank cannot know it is the same payment. If both clear, you have paid ₹40,000.

Map it back:

• The lift button is an operation that sets state (status = PAID): safe to repeat by design.
• The cheque number is an idempotency key, chosen once by the payer before the first attempt.
• The bank’s register is the server’s key store, checked before money moves.
• Writing a fresh cheque is the most common real bug: a new key on every retry.
• The silent week is a timeout. It says nothing about whether the payment happened.

What idempotent really means

In maths, f(f(x)) = f(x). In systems, say it with side effects: running an operation n times, for any n ≥ 1, leaves the system in the same state as running it once. Three clarifications matter.

Same effect, not the same response. RFC 9110 defines idempotence by the intended effect and says the response might differ: DELETE /orders/42 returns 204, then 404. Callers notice, though: AWS’s Builders’ Library notes that answering a retried create with “ResourceAlreadyExists” leaves the caller unsure whether its request made the resource, so AWS returns a semantically equivalent response to the first one. Good APIs replay the original answer.

Only the intended effect counts. Logs, metrics and rate-limit counters may move on every retry; the order table and the wallet may not.

Idempotent is not safe. Safe means read-only; DELETE is idempotent but hardly safe. Nor is idempotent “harmless whenever it arrives”: a late duplicate can still undo a newer write.

In HTTP, GET, HEAD, OPTIONS, PUT and DELETE are idempotent; POST and PATCH are not (full table in HTTP and HTTPS). A proxy must not retry a non-idempotent request on its own, so making a POST retryable is your job.

The unknown outcome

You pay ₹500 at a pharmacy from a phone with one bar of signal. The spinner turns for 30 seconds, then the app says “Something went wrong”. The shopkeeper asks you to try again, so you tap Pay. Two SMS arrive: ₹500 debited, ₹500 debited.

The first request reached the bank and the debit committed; the response died on the way back. From the phone, three different failures look identical:

1. The request never arrived. Retrying is safe.
2. The server failed halfway. Retrying may be safe, depending on what committed.
3. The work committed and the response was lost. Retrying blindly does it twice.

A timeout is not a failure. It is the absence of information.

Retries come from everywhere, and most of them are not your code:

• People double-tap, refresh and press “Try again”.
• SDKs retry with backoff on their own; the AWS CLI even generates a client token for operations that support one and reuses it on each retry.
• Infrastructure resends on connection errors. RFC 9110 bans proxies from doing that for POST, yet notes that some clients resend a POST when an idle keep-alive connection closes before any response.
• Deadlines expire after the work is done: a gRPC call can end in DEADLINE_EXCEEDED after the server committed (see REST vs GraphQL vs gRPC).
• Message brokers redeliver a message whose acknowledgement was lost.

You cannot remove retries; they are how you survive packet loss and crashes. You can only make them harmless: at-least-once delivery + idempotent processing = an effect that happens once. How to retry politely is covered in timeouts, retries and backoff with jitter; this page is about making the retry safe.

Four ways to make a write safe to repeat

Think of these as a ladder. Each rung covers more cases and costs more, so climb only as high as the operation needs.

Rung 1: design it to be naturally idempotent

Phrase the write as “make it so” rather than “do it again”:

• Set, don’t increment. UPDATE orders SET status = 'PAID' WHERE id = 42 is harmless twice; SET balance = balance - 500 debits twice.
• Replace with PUT. PUT /users/42/address with the full address converges on the same row.
• Upsert by a natural key. INSERT … ON CONFLICT (user_id, day) DO UPDATE SET steps = 8000 records the day’s total, not “add 8,000”.
• Delete by ID. The second delete finds nothing.

You gain retry safety with no extra storage. You pay in two ways. Moving money, sending an SMS and creating an order are additive; they can’t be phrased as “set”. And a late duplicate can overwrite newer data: Asha’s phone sends PUT name = "Asha", which stalls; she fixes it to “Asha K” on her laptop; the stalled request lands and puts “Asha” back. Each PUT is idempotent, yet the interleaving is a lost update.

Rung 2: make updates conditional

Attach the version you read, and let the write succeed only if it is still current. Over HTTP the client sends If-Match with the ETag, and the server answers 412 Precondition Failed if the resource changed. In a database, a version column does the same job:

UPDATE accounts
SET    balance_paise = 450000, version = 8
WHERE  id = 9 AND version = 7;   -- a retry or a racing writer matches 0 rows

Most stores offer the same compare-and-set. You gain protection from duplicates and stale retries: Asha’s stalled PUT carries version 7 and is refused. You pay with a read before each write and an ambiguity on retry: a 412 (or 0 rows) may mean someone else won, or that your own first attempt succeeded and bumped the version. The client must re-read and check whether the record holds its values. This is optimistic locking seen from the retry side.

Rung 3: let the client choose the ID for creates

POST /orders is the classic non-idempotent call, because the server assigns a fresh ID each time. Flip it: the client generates the order ID before the first attempt, and the primary key does the deduplication.

INSERT INTO orders (id, user_id, amount_paise, status)
VALUES ('0190f4c2-6b1e-7a3c-9d55-2f8e41c0aa17', 7, 50000, 'CREATED')
ON CONFLICT (id) DO NOTHING;     -- 0 rows inserted means “seen before”

If nothing was inserted, load the existing order, check that it matches this request (same user, same amount), and return it. The same idea appears as PUT /orders/{id}, as EC2’s ClientToken (up to 64 ASCII characters, on RunInstances and dozens of other actions) and as the request_id field in Google’s AIP-155.

Use IDs with enough entropy. A version 4 UUID carries 122 random bits (RFC 9562), so even after a billion IDs the chance of any collision is about n² / 2¹²³ = 10¹⁸ / 1.06 × 10³⁷ ≈ 10⁻¹⁹. UUIDv7 puts a millisecond timestamp first, so new IDs append to a B-tree index instead of landing on random pages (see unique ID generators).

You gain idempotent creates from an index you already have. You pay with narrow coverage: it protects one insert in one database, not a card charge or an email, and doesn’t replay the original response.

Why not hash the request body instead? Because identical requests can be intentional: two ₹100 recharges in a row look alike, and so do two requests for identical EC2 instances, which is why AWS rejected parameter hashing. Only the client knows its intent, so the client names the operation. A hash is still useful as a fingerprint alongside the key.

Rung 4: idempotency keys with a stored result

The general tool for any non-idempotent POST, especially one that touches systems you can’t roll back. The client sends a unique key; the server remembers key → outcome and answers retries from memory. Stripe popularised it, Adyen and Worldpay use the same header, and the IETF is standardising it.

Idempotency keys, step by step

The header and the key

POST /v1/payments HTTP/1.1
Content-Type: application/json
Idempotency-Key: "8f14e45f-ceea-467a-9575-0d6f8a1b2c3d"

{"payee": "pharmacy-2231", "amount_paise": 50000}

The IETF Idempotency-Key draft (still a draft in late 2026) makes the value a structured-field string, hence the quotes; Stripe and Adyen accept a bare token of up to 255 and 64 characters. The client generates a unique key (a UUID is recommended) and never reuses it with a different payload; the server must publish how long keys live.

Scope it and fingerprint it

Make the key unique per caller: the constraint is on (account_id, key), so two customers who pick the same string don’t collide. The draft’s security section asks for this composite, because with global, low-entropy keys one client could guess another’s key and read its cached response. Add the endpoint to the scope too.

Store a fingerprint of the request, such as a SHA-256 of the canonical body, so a reused key with a different payload is caught instead of answered with someone else’s result.

CREATE TABLE idempotency_keys (
  account_id     BIGINT      NOT NULL,
  idem_key       TEXT        NOT NULL CHECK (length(idem_key) <= 255),
  endpoint       TEXT        NOT NULL,              -- 'POST /v1/payments'
  request_hash   BYTEA       NOT NULL,              -- fingerprint of the body
  status         TEXT        NOT NULL,              -- in_progress | completed
  locked_until   TIMESTAMPTZ,                       -- lease for the running attempt
  recovery_point TEXT        NOT NULL DEFAULT 'started',
  response_code  INT,
  response_body  JSONB,
  created_at     TIMESTAMPTZ NOT NULL DEFAULT now(),
  PRIMARY KEY (account_id, idem_key)
);

Size it before you promise a retention period. At 500 payments a second, 500 × 86,400 s ≈ 43 million keys a day; at about 1 KB per row, mostly the stored response, that is ≈ 43 GB per day of retention, so 72 hours is roughly 130 GB. Partition by day and drop old partitions instead of deleting rows (table partitioning).

Claim the key atomically

Never SELECT and then INSERT: two concurrent retries both see nothing and both proceed. Let the unique index decide:

INSERT INTO idempotency_keys
       (account_id, idem_key, endpoint, request_hash, status, locked_until)
VALUES ($1, $2, $3, $4, 'in_progress', now() + interval '30 seconds')
ON CONFLICT (account_id, idem_key) DO NOTHING
RETURNING status;                -- a row back means this request owns the key

In PostgreSQL, if the first claimant has not committed yet, the second insert waits for it and then sees the conflict, so duplicates queue up on the index without explicit locks. Redis can make the same claim with SET idem:7:8f14… in_progress NX EX 86400, but a key in Redis cannot commit atomically with a row in your database, as you’ll see in a moment.

Answer a key you have seen before

If the insert returned nothing, read the existing row and branch:

• Different fingerprint. A client bug. The draft says 422 Unprocessable Content; EC2 returns IdempotentParameterMismatch.
• Completed. Replay the stored status and body. Stripe adds Idempotent-Replayed: true so clients and logs can tell a replay from a fresh result.
• In progress, lease still valid. The first attempt is still running. The draft says 409 Conflict; Stripe tells clients to wait and retry with the same key, because a new key starts a second operation. Holding the duplicate until the stored result lands is friendlier but ties up a connection.
• In progress, lease expired. The worker that claimed the key died. Take over and resume from the stored recovery point.

Commit the result with the work

The key row and the business change must commit or roll back together. If the payment commits and the process dies before the response is saved, the retry pays again; if the key is marked completed and the payment rolls back, the retry replays a success that never happened. AWS’s rule: recording the token and every mutation it guards must be one ACID operation.

So the key table belongs in the same database as the data it protects, updated in the same transaction as the business write. With the key in Redis and the payment in PostgreSQL you have a dual write that no amount of care makes atomic. Redis can be a fast first check, but the database row is the truth for anything involving money.

Decide which outcomes to replay

Stripe’s v1 rule is a good default: save the result once the endpoint starts executing, errors included. Even a 500 is replayed, because it may have had side effects, which forces the client to reconcile rather than guess. Requests rejected before execution, by validation or by a clash with a concurrent request using the same key, are not saved and can be retried. Stripe’s newer v2 API goes further and re-runs failed or partially failed work to completion.

For your own API: store definitive outcomes, including business rejections like “card declined”, which no retry will change. When an attempt fails and its transaction rolled back cleanly, release the key so the next retry runs fresh.

Keep keys longer than any client retries

Retention must outlast the longest retry path, or a late retry becomes a new operation:

Multi-step operations: atomic phases and recovery points

A checkout that charges a card cannot be one database transaction, because you cannot roll back a charge at the payment gateway. Brandur Leach’s Postgres design splits such a request into pieces a retry can resume:

• A foreign state mutation is any call that changes another system: a gateway charge, an email, even a publish to Kafka.
• An atomic phase is the set of local writes between two foreign mutations, done in one transaction.
• A recovery point is a name stored on the key row and committed with each phase, so a retry jumps to where the last attempt stopped.

Three rules make it work. Commit each phase before the foreign call, so a crash always leaves a record of what was attempted. Give the foreign call its own key, derived from your record: Leach sends the gateway a key built from his own key row’s ID rather than the client’s string, so it is unique across all his customers. Push non-urgent side effects out of the request: the receipt email becomes a job row inserted in tx4, the transactional outbox pattern.

A completer process finishes keys stuck in progress after clients give up, and a reaper deletes expired keys. If a downstream service offers no idempotency key at all, never retry it blindly: look the operation up by your own reference first, or mark it “unknown” and reconcile.

Across services: every hop dedupes its own retries

Saltzer, Reed and Clark made the argument in 1984: a network can suppress its own duplicate packets, as TCP does, but an application’s retries look like brand-new messages to every layer below, so only the application can recognise them. Likewise, when Checkout calls Payments and retries after a timeout, the API’s dedupe on K doesn’t help; Payments needs its own. Kafka’s idempotent producer (on by default unless settings conflict) stops the producer’s own retries from duplicating records; it cannot stop your service from publishing the same event twice after a crash.

What holds a multi-hop flow together is deterministic derived keys. Each service computes downstream keys from the key it received, its caller and the step name, for example hash(account, K, "charge"). The account matters because K is unique only per caller, while downstream all your customers look like one caller; that is why Leach uses his own row ID. A retried checkout arrives with the same K, recreates the same downstream keys, and every hop recognises the duplicate. If a step can legitimately run twice, such as two partial refunds, add a counter: K:refund:1, K:refund:2.

Sagas lean on this hard. An orchestrator retries both forward steps and compensations, so “refund the charge” must itself be idempotent (keyed, say, K:refund-of-charge), or a retried compensation refunds twice. See distributed transactions: 2PC vs sagas.

Consumers face the asynchronous version. Brokers deliver at least once, so a consumer records each message ID in a processed-messages table in the same transaction as its effect, and skips IDs it has seen. That inbox pattern is covered in effectively-once: idempotent consumers and the inbox pattern.

Choosing an approach

A payments design typically layers four of the five: keys at the public API, client-chosen order IDs, version-checked ledger updates and an inbox on every consumer (see design a payment system). A profile service may need nothing beyond PUT and ETags.

Going deeper: how idempotency fails in production

A new key per retry, or one key for many operations

Suppose 1 million payments a day, 0.5% time out, and half of those had in fact committed: 1,000,000 × 0.005 × 0.5 = 2,500 double charges a day (illustrative numbers) from one misplaced line. The opposite bug is as quiet: deriving the key from something reused, like the user ID, so the second legitimate ₹100 recharge gets the first one’s stored response and never happens. The fingerprint catches a reused key with a different body, not an identical one.

Keys that are guessable or unscoped

If keys are short or sequential and the lookup is global, one client can guess another’s key and read its stored response. The draft’s fixes: publish a key format, validate every key against it, and scope the lookup to the authenticated account. Adyen asks for random v4 UUIDs for this reason.

Retention shorter than the retry window

An app with an offline queue that retries for three days, against a server that forgets keys after 24 hours, charges again on day two. Keep keys longer than the longest retry path, or make clients stop and reconcile when the window closes, as Stripe asks after 24 hours.

A key store that can lie

The claim must see the latest committed state. Read keys from an asynchronous replica with 200 ms of replication lag and a retry arriving 100 ms after the first commit finds “no such key”. Regions are worse: Adyen does not check keys across its regional endpoints, so a client that fails over from Mumbai to Singapore mid-retry can pay twice. Pin each account’s writes to a home region or use a globally consistent store (multi-region and active-active). And if the key store is down, fail closed for money: Adyen returns a 503 and lets the caller decide.

Side effects outside the transaction

Emails, SMS, webhooks and third-party calls cannot be rolled back with your database. Each needs its own key, a place in the outbox, or a lookup before any retry. A webhook sent twice is harmless only if receivers dedupe on the event ID.

Late retries after the resource changed

A retry can sit in a queue and arrive after the resource it created was changed or deleted. EC2 answers a late RunInstances retry with the semantically equivalent original response rather than launching a new instance. Whatever you return, never let a late retry redo the create.

Cheat sheet

How to say it in the interview

“Any write a client might retry has to be idempotent, because a timeout doesn’t tell the client whether we committed. Updates I make naturally idempotent: set absolute values, upsert, and check a version so a stale retry can’t overwrite newer data. For create-payment, the client sends an Idempotency-Key, a UUID generated once per checkout and reused on every retry. We claim it with a unique constraint on account and key, do the work and save the response in the same transaction, and a retry gets the saved response: 409 while the first attempt is running, 422 if the body changed. The gateway call uses a key derived from ours, consumers dedupe on message ID, and keys live longer than any client retries, say 72 hours.”

What if two copies of the same request arrive at the same moment? The unique index decides: one insert wins, the other waits for that transaction, then sees the row and gets a 409 or the stored result.

Why not dedupe on a hash of the request body? Identical requests can be intentional, such as two ₹100 recharges. Only the client knows intent; the hash is a fingerprint that catches a key reused with a different body.

The server crashes after charging the card but before saving. What happens? The key row is still in progress at recovery point payment_created. When the lease expires, the client’s retry (or a completer job) re-runs only the charge step with the same derived key, and the gateway returns the original charge instead of making a new one.

Can I keep the keys in Redis? As a fast first check, yes; as the source of truth, no. A Redis key can’t commit atomically with your database row, so a crash between the two loses the key or the payment.

Isn’t this just exactly-once delivery? No. Delivery stays at least once; idempotency makes the effect happen once, and every hop with side effects needs its own dedupe.

Where to go next

• Timeouts, retries and backoff with jitter: when to retry, how often, and how to avoid a retry storm.
• Effectively-once: idempotent consumers and the inbox pattern: the consumer-side dedupe table in depth.
• Delivery guarantees: at-least-once, exactly-once and DLQs: what brokers really promise.
• Transactional outbox: getting side effects out of the request without losing them.
• Distributed transactions: 2PC vs sagas: idempotent steps and compensations.
• Design a payment system: idempotency keys, ledgers and reconciliation in one design.
• API design for system design interviews: where idempotency keys fit in the endpoints you sketch.
• Contention and double booking: when the duplicate is a second user, not a second retry.

References and further reading

1. RFC 9110 — HTTP Semantics, §9.2.2 Idempotent Methodsrfc-editor.orgThe definition (same intended effect, possibly a different response), which methods are idempotent, and the rule that a proxy must not retry non-idempotent requests.
2. The Idempotency-Key HTTP Header Field (IETF httpapi Internet-Draft -08, 2026)datatracker.ietf.orgHeader syntax, fingerprints, 409 for a concurrent duplicate, 422 for a reused key with a different payload, and the composite-key security advice.
3. Stripe API reference: Idempotent requestsdocs.stripe.comWhat Stripe saves and when, 255-character keys, pruning after at least 24 hours, parameter comparison, and which requests are not saved.
4. Stripe docs: Advanced error handlingdocs.stripe.comReplayed 500s, wait-and-retry on 409, the 24-hour retry window, reconciling instead of minting a new key, and the Idempotent-Replayed header.
5. Implementing Stripe-like Idempotency Keys in Postgres (Brandur Leach)brandur.orgForeign state mutations, atomic phases, recovery points, a key schema, deriving the downstream key, and the completer and reaper processes.
6. Making retries safe with idempotent APIs (Malcolm Featonby, Amazon Builders’ Library)aws.amazon.comWhy hashing parameters is not enough, caller-provided request IDs recorded atomically, semantically equivalent responses, late retries and parameter mismatches.
7. Ensuring idempotency in Amazon EC2 API requestsdocs.aws.amazon.comClientToken (up to 64 ASCII characters), IdempotentParameterMismatch, and Regional versus zonal scope for the same token.
8. Adyen docs: API idempotencydocs.adyen.comKeys valid for 7 to 14 days per company account, no duplicate check across regional endpoints, and a 503 when the idempotency store is unavailable.
9. End-to-End Arguments in System Design (Saltzer, Reed and Clark, ACM TOCS, 1984)web.mit.eduDuplicate suppression belongs at the application level, because the application’s own retries look like new messages to every layer below it.
10. PostgreSQL documentation: Index uniqueness checkspostgresql.orgA second insert of the same unique key waits for the first, uncommitted transaction, then fails or succeeds, which serialises concurrent duplicates.