A ledger is a company’s memory of money. Every time money moves — a customer pays ₹1,249.50 through a payment provider, the marketplace keeps its 10% commission, a seller is paid a week later, a wallet user sends ₹500 to a friend — some service posts an entry to the ledger, and payouts, finance, reconciliation and the regulator read the result. It is a system of record, not a cache: if the ledger says a seller is owed ₹44,190.20, that is what the seller is paid. Three demands pull against each other. Every entry must balance and post exactly once, then never change. Balances must be exact on the write path, so a wallet never dips below zero. And a handful of accounts — the marketplace’s commission revenue, each payment provider’s receivable — take a posting for every single payment, 5,200 a second at a festive-sale peak, while one database row can absorb only a few hundred updates a second.
Understand the problem
Agree on the vocabulary first; interviewers test it. An account is a named bucket of value: a seller’s payable balance, a customer’s wallet, the money a payment service provider (PSP) owes you, your commission revenue. A journal entry records one business event as two or more postings, each moving an amount into or out of one account. The rule that gives double entry its power: within an entry, total debits equal total credits, per currency. Money is never created or destroyed; it only moves, and every movement says where it came from and where it went.
Debit and credit are directions, not good or bad news. Each account has a normal balance set by its type. Assets (cash in the bank, money a PSP owes you) and expenses (PSP fees) are debit-normal: a debit increases them. Liabilities (what you owe sellers and wallet holders), revenue and equity are credit-normal: a credit increases them. A customer’s wallet is money you owe the customer, so topping it up credits it. Many systems store debits as positive and credits as negative numbers — Square’s Books service does exactly this — so an entry is valid when its postings sum to zero.
Functional requirements
You are building the central ledger service for a marketplace with payments, customer wallets and seller payouts — the kind of ledger that sits behind Stripe, Square or Uber, or that Modern Treasury sells as a product. Every product team posts to it; finance, payouts and reconciliation read from it.
- Product services should be able to post a balanced journal entry with any number of legs, atomically and exactly once — payments, refunds, settlements, payouts, wallet transfers, fees.
- Clients should be able to read an account’s balance — posted, pending and available — in milliseconds.
- Clients should be able to read an account’s history: a statement for any period, and the balance as of any past moment.
- Operators should be able to correct mistakes without editing history, by reversing or adjusting entries.
- The ledger should enforce account rules when posting: a wallet may not go below zero, a closed account takes no postings, currencies must match.
Non-functional requirements
- Never wrong, never twice. Every entry balances per currency. An acknowledged entry is never lost (zero data loss within a region), and a retried or redelivered request never posts twice.
- Immutable and auditable. Append-only; every past balance reproducible; tampering detectable. Regulators and auditors typically ask for 7–10 years depending on the country — plan for 10.
- Strong consistency where money is decided. A wallet spend must see every earlier committed posting on that wallet. Statements, dashboards and the warehouse may trail by seconds.
- Throughput. About 100 million entries a day; 17,400 entries and 52,000 postings a second at a festive-sale peak (estimates below), including single accounts that receive 5,200 postings a second.
- Latency. Posting under 50 ms at p99, including synchronous replication; balance reads under 10 ms at p99.
- Availability. 99.99% for posting and balance reads. During a brief outage, recording producers queue and catch up; authorising callers fail closed.
In the interview
Ask one question before drawing: “Does the ledger record money that already moved somewhere else, or does it decide whether money may move?” A recording ledger can be asynchronous and eventually consistent; an authorising ledger needs strict ordering on each account. Modern Treasury’s write-ups split ledgers along exactly this line, and a real marketplace needs both — often inside the same entry.
Out of scope
- PSP, card and UPI flows. The payment system talks to providers and hands us events; we book them.
- Wallet product rules — KYC tiers, top-up limits, the transfer experience. That is the digital wallet; we enforce the balance rules it asks for.
- The general ledger and financial statements. Finance’s accounting system receives daily summaries from us; profit and loss, GST returns and closing the books happen there.
- Computing amounts. Callers send commission, tax and FX amounts already calculated; we check that they balance.
- Securities positions. A brokerage app could keep its customers’ cash here, but shares and orders are its own problem.
The set-up
Plan the approach
Say the shape early: “an append-only log of balanced entries, with balances as a cached view that can always be re-derived from the log.” Build the five requirements on one PostgreSQL primary, then spend the deep dives on what breaks: exactly-once posting, fast historical balances, hot accounts, entries that span shards, holds, and proving to an auditor that nothing was lost or altered.
Core entities
- Account. For example
seller_payable:4471: type (asset, liability, revenue, expense, equity), normal side, currency, owner, rule flags (no_overdraft,closed), and four counters — debits and credits, each posted and pending — plus a version that increases with every posting. - Journal entry. One business event: ID, idempotency key (usually the producer’s event ID), effective time (when it happened in the world), created time (when we recorded it), metadata, and an optional link to the entry it reverses.
- Posting. One leg: entry, account, currency, amount in paise as a signed 64-bit integer (positive for a debit, negative for a credit), and, for accounts that keep them, the account version and balance after this posting.
- Hold. A pending entry that reserves money until it is posted, voided or expires.
- Balance anchor. An account’s totals at the end of a day, for fast “as of” reads.
Now book the order from the payment-system design in full. The customer pays ₹1,249.50; the marketplace keeps 10% (₹124.95); the PSP takes a 2% fee (₹24.99) and settles the rest the next day; the seller is paid on T+7.
What remains is the economic truth of the order: ₹99.96 more in the bank, ₹24.99 spent on fees, ₹124.95 earned, and 124.95 − 24.99 = 99.96. In India the PSP also charges 18% GST on its fee — ₹24.99 × 0.18 ≈ ₹4.50 — which adds one leg debiting a GST input-credit account and makes the settlement ₹1,220.01; the figures leave tax out.
API
Post an entry. Recording producers and authorising services use the same call. The idempotency key is the producer’s event ID, so a redelivered event maps to the same entry.
POST /v1/entries
Idempotency-Key: evt_01JB9C2W4P
Content-Type: application/json
{
"effective_at": "2026-10-27T10:15:02+05:30",
"source": { "type": "payment.captured", "payment_id": "pay_7Hq2" },
"postings": [
{ "account": "psp_receivable:psp_a", "amount": 124950, "currency": "INR" },
{ "account": "seller_payable:4471", "amount": -112455, "currency": "INR" },
{ "account": "commission_revenue", "amount": -12495, "currency": "INR" }
]
}
{ "id": "je_8Kd2", "status": "posted", "created_at": "2026-10-27T10:15:02.418+05:30" }
A replay with the same key and body returns the same 200 and entry ID. The same key with a different body is 409 Conflict. An entry that does not sum to zero per currency is 422 unbalanced; a wallet spend that would go negative is 422 insufficient_funds.
Read a balance, now or as of a moment:
GET /v1/accounts/wallet:cus_42/balance
GET /v1/accounts/seller_payable:4471/balance?as_of=2026-03-31T23:59:59+05:30
{ "account": "wallet:cus_42", "currency": "INR", "normal": "credit",
"posted": 500000, "pending_debits": 200000, "available": 300000, "version": 912 }
Read a statement, keyset-paginated, with opening and closing balances:
GET /v1/accounts/seller_payable:4471/postings?from=2026-10-01&to=2026-10-31&after=58190
Correct and hold. POST /v1/entries/je_8Kd2/reverse (with its own key and a reason) writes a mirror-image entry. A hold is the same call with "mode": "pending" and an expiry, later resolved with POST /v1/entries/je_9Qx/post (full or partial amount) or /void.
Back-of-envelope estimates
These numbers decide the shard count and, more importantly, expose the hot-account problem (method: back-of-envelope estimation).
Entries. The payment platform handles 30M payments a day. Each produces about 2.5 entries over its life — capture, settlement, and on average half an entry more for refunds, fee adjustments and disputes: 30M × 2.5 = 75M. Wallet top-ups, transfers and cashback add about 25M. Total ≈ 100M entries a day, so 100,000,000 ÷ 86,400 ≈ 1,160 entries/s on average.
Peak. A festive-sale opening runs at 15× the average: 1,160 × 15 ≈ 17,400 entries/s. With about three postings per entry, that is 17,400 × 3 ≈ 52,000 postings/s. Morning settlement files add 30M entries in a burst, but a batch can be throttled — 30M ÷ 3,600 s ≈ 8,300 entries/s if loaded over an hour — and a sale cannot.
Shards. A ledger transaction writes about eight rows (key, entry, three postings, up to three balance updates) and waits for a synchronous replica. Budget 2,500 entries/s per primary: 17,400 ÷ 2,500 ≈ 7, so 8 shards, about 2,200 entries/s each at peak.
The hot account. Every capture credits commission revenue: at the payment peak, 5,200 postings/s on one account. If a running balance is updated in each posting’s transaction, the row stays locked from the UPDATE until commit — about 2.5 ms including the replica round trip. One row can then take at most 1,000 ÷ 2.5 = 400 updates/s, 13 times short.
Storage. An entry header is about 300 bytes and a posting about 120; with three postings and indexes, call it 1.5 KB an entry: 100M × 1.5 KB = 150 GB a day, × 365 ≈ 55 TB a year, about 550 TB over ten years before replication. Keep 90 days hot — 150 GB × 90 = 13.5 TB, about 1.7 TB per shard — and move older months to columnar files in object storage.
Accounts and reads. About 50M wallets, 500,000 sellers with three accounts each and a few hundred system accounts: roughly 52M rows at about 200 bytes, around 10 GB. Balance reads peak near 20,000/s — wallet screens, checkout checks and payout runs — or 2,500 point reads per shard. Easy.
Tip
When you size a ledger, the total write rate is rarely the problem; eight ordinary shards absorb it. The number that shapes the design is postings per second on the busiest single account.
High-level design
Build one requirement at a time on a single PostgreSQL primary with a synchronous replica. By the end, the design works end to end at moderate load.
Product services should be able to post a balanced journal entry
Two kinds of callers post. Recording producers report money that already moved: the payment service commits each state change together with an outbox row, a relay publishes it to Kafka, and ingest workers turn each event into an entry (the hand-off is described in the payment system design). Authorising callers, such as the wallet service, need a yes or no before they act, so they call the ledger synchronously. Both use the same API.
- The payment service commits
payment.capturedwith an outbox row; the relay publishes it topayment-events. - An ingest worker maps the event through a posting template —
payment.capturedbecomes three legs — and posts it with the event ID as the key. - The wallet service posts a transfer synchronously and waits for the answer.
- The ledger service validates the entry and writes it in one database transaction.
Validation is strict and cheap: at least two postings; every account exists, is open and is in the posting’s currency; amounts are non-zero integers; and postings sum to zero per currency. Templates live in the ledger and are versioned. Product teams say what happened; the ledger decides which accounts move. Stripe describes the same idea — its Ledger models each producer system as a fund flow between accounts — and it stops a hundred services from inventing a hundred private versions of accounting.
Why PostgreSQL? A ledger needs multi-row ACID transactions, unique constraints, checks that run at commit, and decades of tooling that auditors trust. Raw write throughput matters less, and the deep dives show how to get it anyway.
Clients should be able to read an account’s balance
Summing every posting on each read is correct but gets slower every day. Instead, keep four counters on each account row — debits and credits, each posted and pending, the shape TigerBeetle uses — and update them in the same transaction that inserts the postings. The balance can then never disagree with the postings, because both commit or neither does.
The visible balance comes from the counters and the normal side. For a credit-normal wallet, posted balance = credits posted − debits posted, and available = credits posted − debits posted − debits pending: money on hold is not spendable. Each update also bumps the account’s version, which identifies exactly which postings make up a given balance.
Clients should be able to read an account’s history
Every posting on an account that keeps history also stores the account version and the balance after it. A statement is then a range scan on an index over (account, version), paginated by version so it stays fast on page 500. The opening balance is the balance after the last posting before the period; the closing balance is the balance after the last one inside it. Statements, finance queries and month-end reports never run on the primary: they read replicas, or the warehouse fed by change data capture.
Operators should be able to correct mistakes without editing history
Suppose a bug booked this seller’s commission at 10% when their promotional rate was 6%: commission should have been ₹74.97 and the payable ₹1,174.53. The tempting fix is two UPDATEs. Never do it. The seller may already have been paid, statements have been sent, and an auditor will ask why the numbers changed.
Instead, correct with new entries. A reversal (R1) mirrors the wrong entry exactly — every debit becomes a credit — and links to it; a re-post (E1′) records what should have happened. There are two options: reverse fully and re-post, or post only the difference (here, debit commission ₹49.98 and credit the seller ₹49.98). Full reversal is easier to read; the difference is smaller. Either way, the original stays.
Make immutability a property of the database, not a promise: the application’s database role gets INSERT and SELECT on entries and postings, and no UPDATE, DELETE or TRUNCATE. Square’s Books service works this way — apart from the cached balance on each book, its tables only ever see inserts, and a mistake is fixed by a new entry that corrects the old one.
Keep corrections separate from new business events. A refund is not a reversal: if the customer returns a ₹500 item after the payout, that is a new entry — debit seller payable ₹450.00, debit commission ₹50.00, credit PSP receivable ₹500.00, which the PSP nets off its next settlement. The seller’s payable now has a debit balance of ₹450: the seller owes the marketplace, and the next payout is reduced. That is allowed for a seller account and forbidden for a wallet, which is the next requirement.
The ledger should enforce account rules when posting
Some accounts carry rules: a wallet may not go below zero, a closed account takes nothing, a seller under investigation may receive but not pay out. The rule must be checked against the balance at the moment of commit, or two concurrent spends both pass.
Put the rule inside the write. In PostgreSQL’s default Read Committed mode, a second UPDATE on a locked row waits for the first transaction to commit and then re-evaluates its WHERE clause against the new row version — so this one statement is both the check and the debit:
UPDATE accounts
SET debits_posted = debits_posted + 80000, -- ₹800 in paise
version = version + 1
WHERE id = 'wallet:cus_42'
AND NOT closed
AND credits_posted - debits_posted - debits_pending >= 80000;
-- 0 rows updated → roll back the whole entry, return 422 insufficient_funds
TigerBeetle expresses the same rule as an account flag, debits_must_not_exceed_credits, which rejects any transfer that would make pending plus posted debits exceed posted credits. Rules sit on accounts, not in callers, so a new product cannot forget them.
You now have a working ledger: balanced, exactly-once, immutable, rule-checked, with fast balances and statements. On one primary it handles about 2,500 entries a second, and its busiest row handles 400. The deep dives fix both numbers.
Deep dives
How do we guarantee every entry balances and posts exactly once?
Two failures matter here. A half-written or unbalanced entry comes from a template bug, a crash between legs or a hand-written fix. A duplicate entry comes from at-least-once delivery: Kafka redelivers after a worker crash, or a caller times out and retries while the first attempt is still committing — the scenario Modern Treasury walks through for its own API.
Bad: validate in code, write legs one at a time. Validation catches most bugs, but if each posting commits on its own, a crash after leg two of three leaves the ledger out of balance until someone notices. Without a unique key, every retry posts again.
Good: one transaction and a unique idempotency key. Insert the key, the entry, all postings and the counter updates in one transaction. The key’s primary-key constraint is the lock: of two concurrent duplicates, one insert wins; the other waits on the unique index until the first commits, then fails. The ledger then looks up the existing entry and compares a hash of the request: the same hash replays the original answer; a different hash is 409, because the caller reused a key for a different movement. Recording producers use their event ID as the key; synchronous callers generate one per logical operation, outside their retry loop.
Keep keys at least as long as anyone might retry — longer than Kafka’s retention and the payment system’s 72-hour key window. Thirty-five days is a comfortable margin: 100M × 35 = 3.5B keys at about 50 bytes of index each ≈ 175 GB, about 22 GB per shard, in a table partitioned by day so old partitions are dropped, not deleted row by row.
Great: make the database itself refuse an unbalanced entry. Validation in the service can have bugs, and someone will one day run a manual fix in psql. So add a backstop that runs at commit. PostgreSQL’s constraint triggers can be deferred to the end of the transaction, after every leg is in, and an exception there aborts the whole entry:
CREATE TABLE postings (
entry_id bigint NOT NULL REFERENCES journal_entries (id),
leg smallint NOT NULL,
account_id bigint NOT NULL REFERENCES accounts (id),
currency char(3) NOT NULL,
amount bigint NOT NULL CHECK (amount <> 0), -- paise: + debit, − credit
PRIMARY KEY (entry_id, leg)
);
CREATE FUNCTION assert_entry_balanced() RETURNS trigger AS $$
BEGIN
IF EXISTS (SELECT 1 FROM postings WHERE entry_id = NEW.entry_id
GROUP BY currency HAVING sum(amount) <> 0) THEN
RAISE EXCEPTION 'entry % does not balance', NEW.entry_id;
END IF;
RETURN NULL;
END $$ LANGUAGE plpgsql;
CREATE CONSTRAINT TRIGGER entry_must_balance
AFTER INSERT ON postings
DEFERRABLE INITIALLY DEFERRED -- checked at COMMIT, not per row
FOR EACH ROW EXECUTE FUNCTION assert_entry_balanced();
REVOKE UPDATE, DELETE, TRUNCATE ON journal_entries, postings FROM ledger_app;
What you pay: a few extra index lookups per entry at commit, cheap next to the commit itself. What you gain: an unbalanced entry is impossible through any path that uses the application role, and the nightly trial balance in the last deep dive checks the checker. Never use floating point anywhere near this table: ₹0.10 + ₹0.20 is not ₹0.30 in binary floating point, and a ledger that is off by a paisa does not balance.
How do we read balances fast, now and as of any past moment?
Finance asks “what was this seller owed at 23:59 on 31 March?”; support asks “why did my balance drop on Tuesday?”; the payout job asks “what is owed right now?”. Each wants a different read.
Bad: sum the postings on every read. Correct, and simple — but it is O(n) in the account’s history. A busy seller with 2,000 orders a day collects 2,000 × 365 = 730,000 postings a year; commission revenue collects 30M a day, about 11 billion a year. No read path survives that.
Good: counters on the account row, updated in the posting transaction, as in the high-level design. The current balance is one row read. But it answers only now, and it puts a lock on every posting — which the next deep dive has to deal with.
Great: three structures, one per question.
- Now: the counters, synchronous for accounts with rules, folded asynchronously for the rest (next deep dive).
- Statements: each posting on a strict account stores the account version and balance after it, written under the row lock you already hold. “Which postings make up the balance at version 912?” becomes one range read; Modern Treasury exposes exactly this
account_versionso a client can show a balance and its entries consistently. - As of a time: a daily anchor per account and effective date holds the end-of-day totals. A balance as of time T is the anchor for the day before T plus the postings from midnight to T. Anchors are built asynchronously from the change stream in batches by account and date, so they cost nothing on the write path. Modern Treasury describes the same anchoring, with finer anchors for accounts that take many postings a day — anchor hot accounts every minute rather than every day.
Two refinements make this audit-grade. First, keep two clocks on every entry: effective_at, when the money moved in the world, and created_at, when the ledger learned of it. A settlement file for 26 October that arrives on the 27th is posted with yesterday’s effective time; it marks the 26 October anchor dirty, and until the rebuild finishes, reads add the not-yet-folded postings on the fly. Second, check the caches: every night, recompute a sample of balances (all of them, weekly) from raw postings and compare. Modern Treasury runs the same drift check and automatically stops serving cached reads for an account that has drifted — a cache you never verify is a second, unaudited ledger.
What you pay: the anchor table (about 5M active accounts a day × 100 bytes = 500 MB a day) and a pipeline to keep it fresh. What you gain: “balance as of any moment” in two reads, at any account age.
How do we handle hot accounts?
Commission revenue takes a posting for every capture: 5,200 a second at the sale peak. Each PSP’s receivable, the GST payable account and the settlement bank account are almost as busy. With a running balance updated synchronously, every one of those transactions takes the account’s row lock and keeps it until commit. At 2.5 ms per hold, the row serialises at 400 updates a second. Demand is 13 times that, so the queue grows by about 4,800 transactions every second. Worse, each waiting transaction already holds locks on its other rows — a wallet, a seller — so the hot row’s queue spreads into accounts that were never busy. Modern Treasury reports authorising performance degrading at around 100 entries a second on one account.
Bad: one row, updated in every transaction. Correct and simple, and capped at 400 postings a second whatever hardware you buy, because the bottleneck is lock hold time, not CPU.
Good: split the account into buckets. The logical account commission_revenue becomes 32 physical sub-accounts; each posting picks one by hashing its entry ID; the balance is the sum of 32 rows. Each bucket takes 5,200 ÷ 32 ≈ 163 postings a second, about 41% of a row’s capacity. The cost: balance reads touch 32 rows, and a limit check across buckets is no longer atomic — fine for revenue, which has no limit, and wrong for a wallet.
Great: decide per account whether it authorises or records. Accounts with rules — wallets, anything that must not go negative — keep synchronous counters; their traffic is per person and low. Accounts that only record — revenue, receivables, fee expense, tax payable — skip the counter update entirely. Their posting is a plain insert, and inserts do not wait for each other. An aggregator reads the change stream and folds postings into each bucket’s counters in batches, once a second, so 5,200 postings become about 32 row updates a second. A balance read returns the counters plus postings not yet folded (exact), or the counters alone (up to a second stale). Modern Treasury’s card-authorisation example is exactly this mix inside one atomic entry: a balance-locked leg on the cardholder’s account, an unlocked, batch-applied leg on the processor’s account.
A third lever stacks with both: batch at the writer. Ingest workers can apply 50 entries in one database transaction and update each hot row once per batch, as the last statement before commit. If the lock is still held about 2.5 ms, that is 400 batches × 50 = 20,000 postings a second, even on one synchronous row. The costs are a few milliseconds of wait to fill each batch and care with failures — a savepoint per entry, so one rejected entry does not sink 49 good ones. TigerBeetle builds the whole database around this idea: every request is a batch of up to 8,190 transfers, and replication is paid once per batch.
Do not shard your way out of a hot account
Adding shards raises total throughput, but a single account still lives on one row on one shard. TigerBeetle’s documentation makes the point bluntly: a few hot accounts appear in a large share of transactions, so the shards that own them become the bottleneck. Fix hot accounts with buckets, folding and batching; fix total volume with shards.
How do we shard when one entry touches accounts on different shards?
Eight shards absorb the total volume. Shard by account owner — a seller, a customer — so one person’s wallet, pending and reward accounts live together. The trouble is entries. A capture touches commission revenue and PSP receivable (system accounts) and a seller’s payable; a wallet transfer touches two customers who, with eight shards, sit on different shards seven times in eight.
Bad: two-phase commit across shard databases. Prepare on both shards, then commit on both (two-phase commit versus sagas). It is correct, but every cross-shard entry holds its locks across extra round trips and a coordinator write. If the coordinator dies after the prepare, the prepared transactions keep their locks until someone recovers it — on a wallet that blocks one customer; on a system account it blocks everyone.
Good: never split an entry. System accounts already have buckets (previous deep dive), so place four on every shard: commission_revenue#s3 lives on shard 3 next to seller 4471, and the whole capture commits locally. The same trick makes settlement lines, refunds and payouts single-shard, and reports simply sum the buckets. Only owner-to-owner movements remain cross-shard.
Great: move those through clearing accounts, as a saga. Each shard keeps a clearing account per counterpart shard. Every step is an ordinary balanced, local entry, and the pair of clearing accounts nets to zero when both steps land.
- Shard 2 debits wallet A and credits its clearing account in one local transaction — the no-overdraft check runs here — and writes an outbox row.
- A relay carries transfer
tr_77to shard 5 (transactional outbox, at least once). - Shard 5 debits its clearing account and credits wallet B, idempotent on
tr_77. If it refuses — wallet B is closed or over its KYC limit — a compensating entry on shard 2 returns the money.
What you pay: wallet B sees the money a few hundred milliseconds later, transfers become two entries, and you run a relay. What you gain: no distributed locks, and a built-in alarm. A clearing pair that stays non-zero means a stuck transfer, and Stripe’s Ledger uses the same signal at scale — “find the clearing accounts with non-zero balance”. The product side of transfers is in the digital wallet design.
Two other architectures avoid application sharding altogether, at different prices:
| Sharded SQL + clearing sagas | Distributed SQL (Spanner-style) | Single-leader ledger engine (TigerBeetle-style) | |
|---|---|---|---|
| Atomic across any accounts | Within a shard; sagas across | Yes, the database runs the commit protocol | Yes, one state machine applies everything |
| Hot accounts | Buckets and folding | Still one hot row; same fixes needed | Batches of up to 8,190 transfers amortise each commit |
| Throughput ceiling | Add shards | Add nodes | One leader’s core; split by region or product |
| Cost per entry | One local commit | Higher when a commit spans ranges | A short batching wait, then one replicated commit |
| What you operate | Shard map, relay, clearing checks | A distributed database you must model carefully | A specialised database beside your SQL store |
| Public example | This article’s design | Square’s Books on Cloud Spanner | TigerBeetle |
Square chose Cloud Spanner so that its Books team — three engineers at the time of its write-up — would not run application-level sharding, and it still had to design the schema around hotspots. TigerBeetle chose one leader on one core, replicated by consensus, because, it argues, contention on hot accounts cancels most of the benefit of sharding. For an interview, the sharded design with clearing sagas is the one you can defend line by line; name the other two as the build-versus-buy alternatives.
How do we hold money before we move it?
Card payments authorise first and capture later; a ride or hotel reserves a deposit and charges the final amount; a marketplace may hold funds until delivery. The ledger must show the money as reserved — not spendable, not yet moved.
Bad: post at once and reverse if the plan changes. A ₹2,000 ride hold leaves the wallet immediately; when the fare is ₹1,640 you reverse and re-post. Statements fill with noise, and a missed reversal leaves a customer short.
Good: a holds table beside the ledger. Available balance = ledger balance − active holds. It works, but now two stores must change together, and expiry and partial capture live outside the accounting model, invisible to auditors.
Great: two-phase entries inside the ledger. A pending entry moves the amount into the pending counters — debits_pending on the wallet, credits_pending on the counterparty — and leaves posted counters alone. A later entry, linked by the pending entry’s ID, posts it (fully or partly, releasing the rest), voids it, or expires it after a timeout. Each is a new immutable entry, and a unique constraint on the pending ID makes it resolve at most once. This is TigerBeetle’s model, and because the pending debit already counts against the wallet, the later post can never break the no-overdraft rule.
Expiry needs one clock. A sweeper using the ledger’s own time voids pending entries past their deadline; callers’ clocks never decide. For cards the mapping is direct — authorisation is pending, capture is post (partial if the final amount is lower), an authorisation reversal is void, and an unclaimed authorisation expires. Most UPI payments are push-and-done, so they post in one phase. The price is two more counters per account and an API that must always say which balance it means: posted or available.
How do we prove the ledger is complete and untampered?
An auditor asks three questions. Does it balance? Does it match the outside world? Has anyone changed history? “The code is correct” answers none of them.
Bad: trust the ledger because the code validates. Validation stops the bugs you imagined. It cannot see a missing event, a stuck transfer, a cache that drifted or a privileged UPDATE.
Good: daily checks from three directions.
- Trial balance. The sum of all postings per currency must be zero, and each account’s counters must equal the sum of its postings. Either failing is a sev-1.
- Clearing accounts must clear. PSP receivable per settlement batch, inter-shard clearing, refunds in flight: each should return to zero within its expected time. Stripe’s Ledger, which ingests about five billion events a day, measures exactly this and calls it clearing; it reports 99.99% of its dollar volume fully ingested and verified within four days.
- Completeness and the outside world. Every producer event ID must have a ledger entry — another of Stripe’s checks. Matching PSP settlement files and bank statements against the ledger runs in the payment system’s reconciliation engine; the ledger supplies the expected side and receives adjusting entries with a reason code.
Great: add tamper evidence and sealed history. Grants stop the application from editing rows; they do not stop a superuser. So make edits detectable. Give each shard a hash chain: an entry’s hash covers the previous entry’s hash and its own canonical bytes, so changing any old entry changes every hash after it. Compute the chain from the commit-ordered change stream, not in the posting transaction, or the chain becomes a serialisation point on every shard. Every 30 minutes, seal the window: a manifest with the shard, time range, entry count, per-currency totals (zero) and last hash, signed with a key the database administrators cannot use, written to object storage with a retention lock.
Uber’s LedgerStore works this way: it seals 30-minute windows into signed manifests of records and checksums, and closes a window only when four checksums agree — computed from the table and from the change stream, in each of two regions. Corrections to a sealed window are new records, never edits. The same change stream feeds the warehouse, which doubles as the ten-year archive in columnar files, with per-window counts checked against each seal. At the average rate, a window holds 1,160 × 1,800 ≈ 2.1M entries.
In the interview
Say why the checks exist: “Constraints and idempotency stop the bugs I can imagine. Trial balances, clearing accounts and sealed hash chains catch the ones I can’t — and they are what lets the ledger prove to an auditor that it is complete and unaltered.”
The final design
Put the pieces together. Writes reach the shards through one API; everything that summarises, archives or checks the ledger reads its change stream.
- Record. The payment service commits each money event with an outbox row; ingest workers read
payment-events, map each event through a versioned template, and post in small batches with the event ID as the key. - Authorise. The wallet service calls the ledger API synchronously for spends, transfers and holds, and acts only on a committed entry ID.
- Post. The API validates the entry, routes it to the owner’s shard and runs one local transaction: key, entry, postings, conditional counter updates. The deferred trigger checks the balance at commit, and the synchronous replica confirms before the caller hears “posted”.
- Fold. Recording-only accounts — revenue, receivables, fees, tax — have four buckets per shard and skip the counter update; the aggregator folds their postings once a second.
- Cross shard. Owner-to-owner transfers post locally into a clearing account; the relay carries them to the receiving shard, which posts the second half or triggers a compensating entry.
- Read. Rule checks read inside the write. Balance screens, payouts and statements read replicas: counters for now, versions for statements, anchors for “as of”.
- Stream. Change data capture feeds
ledger-changes, which builds anchors and folds, loads the warehouse that doubles as the ten-year archive, and drives the sealer’s hash chains and signed 30-minute seals. - Prove. Nightly trial balances, drift checks, clearing-account ageing and completeness checks run on replicas and the warehouse; external matching runs in the payment system’s reconciliation engine.
Check it against the requirements. Balanced and exactly once: validation, a deferred trigger and a unique key in one transaction. Immutable and auditable: insert-only grants, reversals, hash chains and signed seals. Strong where it matters: rule checks inside the write on the primary, with eventual consistency only for recording accounts and reports. Throughput: 17,400 entries a second on eight shards at about 2,200 each, and hot accounts at 163 postings a second per bucket. Latency: one local commit with a synchronous replica, and O(1) balance reads.
What interviewers look for by level
| Area | Mid-level | Senior | Staff |
|---|---|---|---|
| Accounting model | Debits equal credits | Normal balances, integer paise, templates; recording versus authorising | Clearing accounts as invariants; two clocks per entry; a chart of accounts treated as a product |
| Exactly once | A unique key | Key per logical operation, request hash and 409; outbox plus idempotent consumers | A deferred balance trigger, insert-only grants, key retention sized to every retry horizon |
| Corrections | Updates or soft deletes | Reversals and adjusting entries; refunds as new events | Closed periods, prior-period adjustments, revisions to sealed windows |
| Balances | SUM on read |
Counters in the same transaction; statements by version | Anchors for “as of”, backdated entries, drift checks |
| Hot accounts | Not noticed | Does the row-lock arithmetic; proposes buckets | Authorise-or-record per account, asynchronous folding, writer batching |
| Sharding | “Use a distributed database” | Shards by owner; avoids hand-rolled 2PC | Per-shard system buckets, clearing sagas, weighs Spanner-style and TigerBeetle-style designs |
| Holds | Not discussed | Pending and available balances | Two-phase entries, expiry by the ledger’s clock, card auth and capture mapping |
| Audit | Application logs | Trial balance and reconciliation | Hash chains off the write path, signed seals, completeness metrics |
Follow-up questions
What if the marketplace starts selling in dollars?
Accounts stay single-currency, and an entry must sum to zero in each currency — never across them through an exchange rate, which moves and which nobody agrees on exactly. Conversion goes through FX clearing accounts, one per currency. At ₹84.00 to the dollar with a 1% spread, a $100.00 sale is one entry: debit PSP receivable (USD) $100.00, credit FX clearing (USD) $100.00; debit FX clearing (INR) ₹8,400.00, credit seller payable ₹8,316.00, credit FX revenue ₹84.00. Each currency balances, the rate and spread are recorded on the entry, and the FX clearing accounts hold the marketplace’s open dollar position until treasury sells it and books the gain or loss.
What if a correction belongs to a month that is already closed?
Once finance closes March, the ledger rejects any entry with an effective time in March (409 period_closed). The fix goes into the open period as an adjustment that references the original entry and carries a reason code. Because every entry has both clocks, you can still answer the auditor’s favourite question — “what was the balance on 31 March, as you knew it on 5 April?” — by filtering on effective time up to 31 March and created time up to 5 April.
What if the ledger is down for ten minutes at the sale peak?
Authorising callers fail closed: no entry, no spend. Recording producers keep committing to their outboxes, and Kafka holds the backlog: 17,400 × 600 ≈ 10.4M entries. Eight shards can post about 20,000 a second, so at peak the spare 2,600 a second clears the backlog in about 10,400,000 ÷ 2,600 ≈ 67 minutes; if traffic falls back to the average, with about 18,800 spare, in about 9 minutes. Within a region, failing over to the synchronous replica loses nothing acknowledged. Cross-region disaster recovery replicates asynchronously, and anything lost in the last seconds is replayed from producers’ outboxes and deduplicated by the same idempotency keys.
What if we must migrate from a single-entry system?
Open each account with one entry against a migration equity account at a cut-over moment, then dual-write for weeks and compare balances per account every day before moving reads, then writes. Backfill history window by window with checksums and checkpoints, so a failure restarts one window rather than the whole job — Uber moved more than 250 billion records, about 300 TB, into its new ledger backend this way.
What if a caller wants one entry with 10,000 legs?
A bulk payout to 10,000 sellers as one entry would lock 10,000 rows across every shard in one transaction. Cap entries at about 100 legs. Post one entry per seller — debit seller payable, credit a payout clearing account — tagged with a batch ID, and one entry that clears the batch against the bank when the bank confirms it. Atomicity stays where it matters, per seller, and the payout clearing account must return to zero when the batch completes.
How to say it in the interview
“I’ll design the central ledger for a marketplace with payments, wallets and seller payouts: about 100 million entries a day, 17,000 a second at peak, and a few system accounts that take a posting for every payment.
The model is double entry. Accounts have a normal side; a journal entry is a set of postings in integer paise that must sum to zero per currency. Each entry is written in one database transaction with a unique idempotency key — the producer’s event ID — so retries and redeliveries never double-post, and a deferred trigger refuses anything unbalanced at commit. Nothing is updated or deleted; mistakes get reversing entries.
Balances are counters on the account row, updated in the same transaction. Wallets use a conditional update, so a spend that would go negative matches zero rows and fails. Statements read postings by account version, and ‘balance as of’ reads a daily anchor plus that day’s postings.
The hard part is hot accounts. Commission revenue gets 5,200 postings a second, and a row lock held for 2.5 milliseconds caps one row at 400. So recording-only accounts get buckets — four per shard — and skip the synchronous update; an aggregator folds their postings once a second. Strict accounts stay synchronous.
I shard by account owner into eight PostgreSQL shards. Because system accounts have a bucket on every shard, payments post on one shard; owner-to-owner transfers run as a saga through clearing accounts that must net to zero. Holds are two-phase pending entries, posted, voided or expired.
Finally, proof: nightly trial balances, clearing-account and completeness checks, and hash chains sealed every 30 minutes into signed manifests, so we can show an auditor nothing was lost or changed.”
Concepts used in this design
- Idempotency — a unique key per entry turns retries into replays.
- ACID transactions — an entry, its postings and its balances commit together or not at all.
- Database constraints — unique keys and a deferred trigger that refuses unbalanced entries.
- Transaction isolation levels — why the conditional
UPDATEis safe under Read Committed. - Lost update problem — the read-check-write race behind a double-spent wallet.
- Optimistic vs pessimistic locking — version checks versus row locks on balances.
- Contention and double booking — the hot-account problem in general form.
- Scaling writes — buckets, batching and asynchronous folding.
- Database sharding — sharding by account owner.
- Two-phase commit vs sagas — clearing-account sagas instead of cross-shard locks.
- Transactional outbox — how producers and the transfer relay hand off events safely.
- Idempotent consumers and the inbox pattern — exactly-once effects from at-least-once delivery.
- Change data capture — feeds anchors, folding, the warehouse and the sealer.
- Event sourcing and CQRS — balances as projections of an immutable log.
- Strong vs eventual consistency — authorising accounts are strict, recording accounts may lag.
- Consensus and leader election — how single-leader ledger engines replicate each batch.
References and further reading
- Books: an immutable double-entry accounting database service (Square Engineering)developer.squareup.comJournal entries that must balance to zero, debits stored as positive and credits as negative, books / journal_entries / book_entries tables, a per-book version counter, insert-only tables with corrections as new entries, built on Cloud Spanner.
- Ledger: Stripe’s system for tracking and validating money movementstripe.devAn immutable log of about five billion events a day, fund flows modelled as balances moving between accounts, clearing accounts that must reach zero, and completeness checks against producer databases.
- How to Scale a Ledger, Part IV: recording and authorizing (Modern Treasury)moderntreasury.comRecording versus authorizing ledgers, effective_at versus created_at, account versions, version and balance locking, and authorizing performance degrading around 100 entries per second on one account.
- How to Scale a Ledger, Part VI: concurrency and performance (Modern Treasury)moderntreasury.comIdempotency keys kept for 24 hours, cached posted and pending debit and credit totals, end-of-day anchors and resulting-balance caches for effective-time reads, and drift checks that switch off a bad cache.
- How Uber Migrated Financial Data from DynamoDB to Docstore (LedgerStore)uber.comImmutable records with trackable corrections, 30-minute sealing windows with signed manifests, four matching checksums across table, stream and two regions, and a migration of more than 250 billion records (about 300 TB).
- TigerBeetle docs: Two-phase transfersdocs.tigerbeetle.comPending transfers that reserve debits_pending and credits_pending, then post (fully or partly), void or time out, each as a new immutable transfer that resolves the pending one at most once.
- TigerBeetle docs: Account referencedocs.tigerbeetle.comPosted and pending debit and credit counters, the debits_must_not_exceed_credits rule and its exact formula, and the invariant that total debits equal total credits.
- TigerBeetle docs: Performancedocs.tigerbeetle.comBatches of up to 8,190 transfers with one replication cost per batch, 128-byte transfers, and a single-leader, single-core design chosen because hot accounts make sharding a ledger hard.
- PostgreSQL documentation: CREATE TRIGGER (constraint triggers)postgresql.orgConstraint triggers are AFTER ROW triggers that can be deferred to the end of the transaction and raise an exception to abort it.
- PostgreSQL documentation: Transaction isolationpostgresql.orgUnder Read Committed, a second UPDATE on a locked row waits for the first to commit, then re-evaluates its WHERE clause against the new row version.