Design a brokerage app (Robinhood)

A brokerage app is the shop window of the stock market. Millions of people open it to watch prices tick, tap Buy, and expect “Filled” a moment later — shares in the portfolio, money gone from the balance. Behind the screen, the broker streams prices it receives once from each exchange to millions of phones, checks every order against the customer’s cash and shares, sends it to a venue over a session that must never lose or repeat a message, and keeps its own books until the trade settles a day later. All of it is hit at once when the market opens. Here you design the broker — Robinhood in the US, Zerodha or Groww in India. The venue that matches orders is covered in design a stock exchange.

Understand the problem

Separate the broker from the exchange first; candidates often design the wrong one. An exchange matches buyers with sellers. A broker acts for its customers: it holds their money, records their shares, decides whether each order is allowed, sends it to a venue and reports what happened. Its OMS (order management system) owns the life of every order, its RMS (risk management system) decides “may this customer place this order now?”, and its back office keeps the ledger and settles with the market.

An Indian broker is a member of NSE and BSE and sends orders straight to them. A US retail broker such as Robinhood usually sends them to a market maker — a wholesaler that fills them from its own inventory and may pay for the flow — under a duty of best execution. Either way, trades settle one business day later (T+1) through a clearing corporation (see the exchange design). Until then the broker carries the obligation, which is why it blocks your money before your order leaves.

Functional requirements

1. Users should be able to watch live prices for the stocks in their watchlists and holdings, and open a chart.
2. Users should be able to place, modify and cancel orders — market, limit and stop-loss, for delivery or intraday — and see each order’s status and fills in real time.
3. Users should be able to see their holdings, positions and profit and loss.
4. Users should be able to add money and withdraw it to their own bank account.

Non-functional requirements

• Money and shares are always right. An order leaves only after the cash or shares to cover it are blocked; a phone and a laptop racing can never spend the same rupee twice. Strong consistency on the order and funds path.
• No lost or duplicated orders. Every accepted order reaches exactly one final state — filled, cancelled or rejected — matching the exchange’s records, through crashes and dropped links.
• Fast order path. Request in to order out: p50 under 20 ms, p99 under 100 ms inside our systems, even at the open. Status changes reach the app within 1 s at p99.
• Fresh prices. Exchange tick to phone under 500 ms at p99 for symbols on screen.
• Available in market hours. Order entry and cancel at 99.99%: NSE’s normal market runs 6.25 hours (09:15 to 15:30), and 6.25 × 250 days = 1,562.5 hours a year, so about 9 minutes of downtime a year. Streaming at 99.9%. Cancel must work when everything else is degraded.
• Scale. 15 million funded accounts, 3 million active on a busy day, about 2 million connections just after the open, 10 million order requests a day with a busiest second near 25,000.
• Audit. Orders, fills and ledger entries are immutable and kept for years.

Out of scope

• The exchange — matching, market-data generation, circuit breakers. See design a stock exchange.
• Account opening, KYC and regulatory reporting.
• Derivatives margining, mutual funds and crypto (options return as a follow-up).
• Ledger and payment internals — see the ledger and payment system designs.
• Research, news and recommendations.

The set-up

Plan the approach

Say the shape in one breath: “two paths with opposite personalities — a huge, loss-tolerant market-data path and a small, exactly-right order-and-money path — that meet only in the app.” Meet each requirement simply, then spend the time in the deep dives.

Core entities

• Account. Client ID, its cell, linked bank and demat accounts, flags such as “trading blocked”.
• Instrument. Numeric token, symbol, exchange, tick size, lot size, today’s price band, the maximum quantity per order (freeze quantity), status. Reloaded every morning.
• Order. Order ID, the app’s idempotency key, our client order ID for the exchange, the exchange’s order ID, side, type, product (delivery or intraday), quantity, price, trigger price, validity, status, filled quantity, average price, version.
• Order event. Every state change with its cause and timestamps; the order row is a projection of these.
• Fill. Exchange trade ID, order ID, quantity, price, time. Immutable.
• Funds and blocks. Settled cash, one block per open order, amounts owed at settlement, payouts in flight — all double-entry postings.
• Holding and position. Per instrument: settled shares, bought-but-undelivered shares, shares blocked by open sells, quantity authorised for sale, average cost; plus today’s intraday positions.

API

REST for actions, one WebSocket for everything pushed; money as strings or integer paise, never floats.

Place an order. The phone generates the idempotency key when the user taps Buy and reuses it on every retry of that tap (idempotency). The answer is 202 Accepted: a broker can promise the order was recorded and sent, never that it will execute — Kite’s API documentation makes the same point.

POST /v1/orders
Idempotency-Key: 6f1c2d9e-0b7a-4c55-9a1e-3d2f8b7c4e10

{
  "exchange": "NSE",
  "symbol": "ACME",
  "side": "BUY",
  "type": "LIMIT",
  "product": "DELIVERY",
  "qty": 20,
  "price": "1525.00",
  "validity": "DAY"
}

{
  "order_id": "ord_8KQ2",
  "status": "SENT",
  "blocked": "30500.00",
  "available_after": "19500.00"
}

A market order carries a protection percentage instead of a price, and a stop-loss a trigger_price. PUT and DELETE /v1/orders/{id} modify and cancel; both answer 202 with a pending status, because nothing is cancelled until the exchange says so.

Read state and move money.

GET  /v1/orders?date=today         status, fills, average price
GET  /v1/portfolio/holdings        settled and unsettled quantity, average cost
GET  /v1/funds                     cash, blocked, available, withdrawable
POST /v1/funds/deposits            returns a UPI intent or bank redirect
POST /v1/funds/withdrawals         { "amount": "10000.00", "bank_account_id": "ba_1" }

Stream. The client subscribes by instrument token and detail level; prices arrive as binary frames, order updates as JSON with a per-account sequence number so the app can spot a gap and refetch.

{ "op": "subscribe", "tokens": [408065, 884737], "mode": "quote" }
{ "type": "order", "seq": 41, "order_id": "ord_8KQ2", "status": "PARTLY_FILLED", "filled": 12, "avg_price": "1525.00" }

This mirrors Kite’s WebSocket: subscriptions per instrument token, three modes — last price (8 bytes), quote without depth (44 bytes), full with five levels of depth (184 bytes) — several packets per binary frame, and order updates as text on the same socket.

Back-of-envelope estimates

These decide the gateway fleet, how to partition the order path, and what “peak” means (back-of-envelope estimation).

Connections at the open. 15M accounts × 20% active on a busy day = 3M; half open the app just after 09:15 = 1.5M people; at 1.3 connections each (app plus web), 1.5M × 1.3 ≈ 2 million WebSocket connections.

Price fan-out. Each connection follows about 40 symbols, 10 on screen. Popular symbols dominate watchlists, so assume followed symbols tick 10 times a second at the open.

• Every tick of every followed symbol: 2M × 40 × 10 = 800 million messages/s.
• Only symbols on screen: 2M × 10 × 10 = 200 million/s.
• On screen, conflated to at most 4 updates per symbol per second: 2M × 10 × 4 = 80 million messages/s.

The feed handler ingests perhaps 200,000 updates a second, so the edge multiplies traffic by 80M ÷ 200k = 400×. That is real-world sized: Zerodha reported its ticker serving about 175 million ticks a second at peak on 4 June 2024, India’s election-results day.

Bandwidth. A quote packet is 44 bytes plus a 2-byte length: 40 packets a second × 46 B = 1,840 B, plus about 40 B of WebSocket and TLS framing on each of 4 frames = 160 B — about 2 KB/s per phone. The fleet sends 2M × 2 KB/s = 4 GB/s, about 32 Gbit/s. JSON with named fields would be about five times bigger.

Gateway nodes. Zerodha reports one ticker process benchmarked at about 500,000 connections. Budget 100,000 per node for headroom: 2M ÷ 100k = 20 nodes, and to survive losing one of three zones, 20 × 3 ÷ 2 = 30 nodes, each sending about 4 GB/s ÷ 30 ≈ 135 MB/s, or 200 MB/s with a zone down.

Orders. 10M requests a day: 7M new, 2M modifications, 1M cancels. The session is 6.25 × 3,600 = 22,500 s, so 10M ÷ 22,500 ≈ 444/s on average. If 5% arrive in the first minute, 500,000 ÷ 60 ≈ 8,300/s, and the busiest second runs at three times that, about 25,000/s — 56 times the average. Design for 30,000/s.

After-market orders (AMOs). If 500,000 orders were placed overnight, sending them over NSE’s eight-minute pre-open order-entry window is 500,000 ÷ 480 s ≈ 1,040/s; sending them at 09:15 would add 500,000 in one second.

Fills. 80% of new orders fill, 1.25 fills each: 7M × 0.8 × 1.25 = 7M fills a day, 7M × 0.05 ÷ 60 ≈ 5,800/s in the first minute.

Reads at the open. 1.5M people refreshing something every 5 s: 1.5M ÷ 5 = 300,000 requests/s. For scale, Robinhood’s brokerage system peaked at 750,000 requests a second in June 2020.

Storage. Order events: 7M × 5 states + 3M × 3 = 44M × 250 B = 11 GB a day. Fills: 7M × 300 B = 2.1 GB. Ledger postings, four per order (block, then release or convert) and four per fill: 56M × 150 B = 8.4 GB. Total about 21.5 GB a day, ≈ 5.4 TB a year (× 250 days). Holdings: 15M × 8 = 120M rows × 200 B = 24 GB.

Order-path databases. One order is one transaction of about four statements. At a comfortable 3,000 orders a second per primary, 30,000 ÷ 3,000 = 10 independent partitions of 15M ÷ 10 = 1.5 million accounts.

High-level design

Build one requirement at a time. By the end the broker works for one market at modest load, in one database.

Users should be able to watch live prices

The broker receives each exchange’s market data once; phones never see the exchange. Zerodha’s 2024 infrastructure note lists 44 exchange leased lines terminating in its own racks, while its user-facing apps run in the cloud — a common split, because exchange connectivity is physical and fan-out is elastic.

• Feed handler. Decodes each feed into one internal tick format keyed by instrument token: last price, volume, best bid and ask, open, high, low, previous close.
• Tick bus. In-memory publish-subscribe, one topic per instrument. Deliberately not durable: a second-old price is worthless, so there is nothing to replay.
• Quote snapshot. The latest packet per instrument, so a new subscriber gets the price at once.
• Stream gateways. Stateless WebSocket servers mapping each instrument to the connections that want it, subscribed to the bus only for instruments their own clients need.

A candle builder also writes one-minute bars to a columnar store, so a chart is one range read.

The app subscribes to the tokens on its screen; the gateway sends each snapshot, then every new packet. If a gateway dies, its clients reconnect elsewhere and resubscribe — no state lived there.

Users should be able to place, modify and cancel orders

The API gateway authenticates the user, applies per-user rate limits and forwards to the order service, which:

1. Deduplicates by idempotency key, so a retried tap returns the first answer.
2. Checks the order: static rules (tradable, tick and lot size, price band, freeze limit), then whether the account has the cash — or, for a sell, the shares.
3. Records and blocks in one transaction: the order, the block on money or shares, and an outbox row. Then it replies 202 SENT.
4. Sends via an exchange connector that reads the outbox and sends the order over the broker’s FIX or exchange session with a client order ID the exchange will echo back.

Acknowledgements, fills and rejections arrive asynchronously. Each becomes an event that moves the order’s state machine, turns the block into a debit or releases it, and is pushed to the app. Cancel is a request too: the order is CANCEL_PENDING until the exchange confirms, and a fill can win that race. Kite’s documentation shows the same steps as interim statuses — received, pending RMS validation, pending at the exchange, open — with separate pending states for modify, cancel and stop-loss triggers.

Users should be able to see their holdings, positions and P&L

Holdings are shares you own; at an Indian broker they sit in your demat account once the trade settles, and until then they are unsettled. Positions are today’s intraday trades and open derivatives; an equity position carried overnight becomes a holding.

A portfolio service folds fill events into both views — quantity, average cost, today’s buys and sells, unsettled quantity — as a projection you can rebuild by replaying fills (event sourcing and CQRS). P&L is arithmetic on top: quantity × (last price − average cost), and quantity × (last price − previous close) for today. The server sends quantities and costs; the phone multiplies by prices it is already streaming. On the server that would be 120 million revaluations per market move; each phone does about eight.

Users should be able to add money and withdraw it

Deposits arrive through a payment gateway — UPI or net banking in India, bank transfers in the US — into a client money account kept separate from the firm’s own money. A funds service credits the ledger only when the payment is confirmed, keyed by the payment reference so a duplicate callback is harmless, exactly like a wallet top-up.

Withdrawals differ from a wallet’s: money blocked by open orders, owed for today’s buys, or from a sale that has not settled is not withdrawable.

withdrawable = settled cash − blocked for open orders − owed at settlement − payouts in flight

A request moves the amount to “payout pending” at once, so it cannot also be traded, and batched payouts go only to the customer’s own linked bank account. Indian brokers also return idle client money periodically; Zerodha reports about 2.5 million payouts on a quarterly settlement day.

You now have a working broker that would survive an ordinary afternoon. The deep dives are about the morning — two million connections, 25,000 orders in one second — and the ways money and orders go wrong.

Deep dives

How do we stream live prices to two million phones?

At the open, 2 million connections follow about 40 symbols each, and popular ones tick many times a second. How much should we send, and to whom?

Bad: the app polls a quote API. Each app asks for its prices every second: 2M requests a second through authentication and load balancers, mostly returning unchanged numbers, each up to a second stale. You pay headers and lookups on every poll and the phone’s radio never sleeps. Fine for chart history or as a fallback; wrong for a live watchlist.

Good: a WebSocket per app, every tick to every subscriber. Fresh, and pushed only on change — but up to 800 million messages a second. A stock trading 100 times a second sends 100 frames a second to a phone that cannot draw that fast. Worse, a slow client makes the gateway buffer: queue every tick for a phone in a tunnel and memory grows until one bad network takes down a gateway with 100,000 healthy clients.

Great: subscribe to what is on screen, conflate per symbol per connection, and make every packet a full state.

• Interest management. The app subscribes only to the rows it shows — about 10 — and swaps subscriptions as the user scrolls; the holdings screen subscribes to all holdings so total P&L stays live. 800M becomes 200M a second.
• Conflation. Per connection, the gateway keeps a slot per instrument holding the latest packet and a dirty flag. A tick overwrites its slot; every 250 ms a timer writes all dirty slots as one frame. Twenty-four ticks still cost the phone four packets, each the newest price. 200M becomes at most 80M.
• Packets are states, not diffs. Each carries the full current value for its mode, so a skipped packet corrupts nothing — the next replaces it. That makes conflation and lossy reconnects safe. Order updates on the same socket are the opposite: sequenced, and refetched over REST on a gap.
• Slow clients cost nothing extra. The slots are a fixed-size buffer; a slow phone just gets fewer, newer prices. A socket unwritable for 10 seconds is closed.
• Binary, tiered, snapshot first. Small binary packets, with full depth only for the instrument whose order ticket is open; the snapshot goes out before any live tick on every subscribe, so a reconnecting app is right after one frame.
• Backgrounding and storms. Mobile operating systems suspend background sockets; missed heartbeats drop the subscriptions, and fills reach the user by push notification. When a gateway dies, its clients reconnect with jittered backoff and the load balancer caps new connections per node per second.
• Two feeds in. The feed handler takes more than one source per exchange and switches when one lags, as Zerodha describes doing after derivatives feeds slowed on volatile days.

You gain an edge that scales with connections, not ticks, and fixed memory per client. You pay up to 250 ms of extra delay, a smarter gateway, and a rule: nothing that acts on prices — stop-loss triggers, margin monitors — reads conflated data; those read the full tick bus.

How do we check and route an order in a few milliseconds?

Between “tap” and “sent” sit a dozen checks, run 25,000 times a second at the open. Group them by the state they need:

• Static checks use reference data loaded each morning: tradable, tick size, lot size, a limit price inside today’s band (the exchange would reject it anyway — see its price bands), quantity under the freeze limit, value under a fat-finger cap.
• Stateful checks need the account: cash or margin for a buy, shares and depository authorisation for a delivery sell, position limits, a per-user order rate.
• Shaping changes the order. A market order becomes a limit with a protection band — Kite’s market_protection percentage, which must sit inside the circuit limits — so a thin book cannot fill someone 20% from the last price. An order above the freeze quantity is split into child orders (Kite’s autoslice). An after-hours order is parked as an AMO.

Bad: one network call per check. Instrument, risk, funds and holdings services called in turn: four hops, four dependencies that must all be up at 09:15, and a race between the funds check and the eventual debit.

Good: one risk engine with reference data in memory. Static checks become map lookups in microseconds, and funds live in the same database as the order, so the stateful check and the block are one statement. But rules hard-coded in the engine make every new regulatory rule a deploy.

Great: rules as data, cheapest first, and the stateful check is the block itself. Zerodha describes this for its RMS: a rules engine that can block orders instantly, scores rules by complexity to keep latency down, and handles CDSL TPIN authorisation. Cheap checks run first, so most rejections never touch the database; the funds check is the conditional write that blocks the money (see buying power, below). Then a router picks the venue.

The router is where India and the US part ways. Investor.gov describes the US choices: an exchange, a market maker that may pay “a penny or more per share” for the order (payment for order flow), an electronic communications network, or the broker’s own inventory. Whatever it picks, the broker owes best execution: evaluate customers’ orders in aggregate and periodically assess which venues give the most favourable terms, including the chance of beating the quote. In software: a routing table per symbol and order type, venue health checks with fallbacks, and a log of every routing decision.

You gain a pre-trade path of a few milliseconds and one database round trip. You pay for keeping reference data fresh in every process — a stale band rejects good orders — and for routing decisions you must defend months later.

How do we make sure no order is lost or sent twice?

An order can be lost or doubled between the phone and us, inside us, and between us and the exchange.

Bad: call the exchange from the request handler, then save. If the process dies between the two, an order is live at the exchange with no record, no block and no way to cancel it from the app. If the phone times out and retries, it is sent twice. Both are routine at the open.

Good: idempotency key plus a state machine in the database. A unique index on account and key turns a retried tap into a lookup, and the order is written before it is sent. But “write, then send” is still two systems: crash in between and the order sits in RECEIVED for ever; resend after a restart and the exchange holds two.

Great: outbox, client order IDs, sequenced sessions, and the exchange’s file as the final word.

• Transactional outbox. The order, its block and an outbox row commit together (transactional outbox). Only the connector sends, in outbox order, recording what it sent; a crash on either side leaves a resumable state.
• One client order ID per order, for life. It is unique for the day and echoed by the exchange on every ack and fill. A connector unsure whether an order went out waits for the session replay or asks about that ID; it never mints a new ID for the same intent.
• Sequenced sessions. The FIX Trading Community defines a session as one ordered stream in each direction, numbered from 1, that outlives individual TCP connections and resynchronises on reconnect. So when a line flaps, the connector logs back on stating the next number it expects, and the exchange resends what was missed. Persist the last number processed with the event it produced, and dedupe fills by trade ID (idempotent consumers).
• A strict state machine. Transitions are validated and versioned — FILLED never returns to OPEN; a fill before its ack implies the ack — and day orders still open at the close expire into CANCELLED.
• End-of-day reconciliation. Every trade in the exchange’s trade file must match exactly one of our fills, and vice versa; anything else is a break worked before settlement.

You gain orders that survive any single crash or dropped line, and a daily proof that our books match the exchange’s. You pay for an outbox relay, persisted sequence numbers, a reconciliation job with people to work its breaks, and orders that sit in “unknown” for seconds during an incident.

How do we keep buying power right when orders race?

Asha has ₹50,000. In the same second she buys 20 ACME at ₹1,525 (₹30,500) on her phone and 20 BETA at ₹1,500 (₹30,000) on her laptop. Only one can go through.

Bad: check the balance, send, debit on fill. Both requests read ₹50,000 and pass; when both fill, the broker has bought ₹60,500 of shares for a customer with ₹50,000 and owes the gap at settlement. In India it is also a penalty: as Zerodha explains the peak-margin rules, the clearing corporation checks margins at five random moments in the day as well as at the close, and an upfront-margin shortfall is charged to the broker, not the client. Hence the rule: no order leaves without its money blocked.

Good: one conditional update. Put the check inside the write — UPDATE … SET available = available − 30500 WHERE account = 'acc_41' AND available >= 30500 — and treat zero rows as “insufficient funds”. The second request waits for the row lock, re-checks against the new balance and fails (lost update problem; the wallet design walks through this race). What it lacks is a record per order: after a half-fill and a cancel, how much do you release?

Great: a block per order, in the same transaction as the order, in a double-entry ledger.

BEGIN;
-- 1. A retried tap hits the unique (account_id, client_key) index and returns the first result.
INSERT INTO orders (order_id, account_id, client_key, symbol, side, qty, limit_paise, status)
VALUES ('ord_8KQ2', 'acc_41', '6f1c2d9e', 'ACME', 'BUY', 20, 152500, 'RECEIVED');

-- 2. Reserve the money only if it is there. Zero rows -> ROLLBACK and reject.
UPDATE funds
   SET available_paise = available_paise - 3050000,
       blocked_paise   = blocked_paise   + 3050000
 WHERE account_id = 'acc_41'
   AND available_paise >= 3050000;

-- 3. One block per order, so a later release or conversion is exact.
INSERT INTO blocks (order_id, account_id, amount_paise, state)
VALUES ('ord_8KQ2', 'acc_41', 3050000, 'ACTIVE');

-- 4. Only rows in the outbox are ever sent.
UPDATE orders SET status = 'SENT' WHERE order_id = 'ord_8KQ2';
INSERT INTO outbox (topic, msg_key, payload)
VALUES ('exchange-orders', 'ord_8KQ2', '{"client_order_id": "c-77"}');
COMMIT;

Every later event moves money out of that block and nowhere else:

• Fill. 12 at ₹1,525 = ₹18,300 moves from blocked to “owed at settlement”; the shares appear as unsettled holdings.
• Cancel or reject. The rest of the block, ₹12,200 for 8 unfilled shares, returns to available.
• Market orders block quantity × last price × (1 + protection): 20 × ₹1,525 × 1.02 = ₹31,110, which always covers a fill inside the protection band; the unused part is released.
• Charges are estimated into the block and trued up on the contract note.

Underneath, these are pending and posted postings in a double-entry ledger; the tables above show the shape, not the whole.

Sells block shares, and need permission. A delivery sell runs the same conditional update on the holdings row (sellable_qty >= 20). In India it also needs the client’s authorisation to debit their demat account, and CDSL’s eDIS flow is neat design: the depository issues the client a TPIN directly; the broker’s app opens the depository’s own page, where the client confirms and enters it, so the broker never sees it; and the depository later lets the delivery through only if it matches the validated instruction. The broker records the authorised quantity, and sells block against it.

Margin is just a different block amount. For leveraged products the risk engine blocks the required margin instead of the full value. The rules move: FINRA is replacing the US pattern-day-trader rules and their $25,000 minimum with intraday margin requirements from June 4, 2026 (transition allowed until October 20, 2027) — 25% maintenance margin on long margin securities throughout the day, which firms may enforce by blocking trades in real time or by computing deficits after the close. Per-order blocks plus a streaming risk view make the real-time option possible.

You gain a balance two devices can never overspend, exact releases, and upfront-margin compliance by construction. You pay a write to the account’s row on every order — fine for people, but an algorithm firing hundreds of orders a second makes it hot, one reason for per-user rate limits.

How do we survive the opening bell?

In March 2020 Robinhood suffered outages on two consecutive trading days; its founders wrote that unprecedented load — volatile markets, record volume, record sign-ups — produced a “thundering herd” that brought down its DNS system. Its brokerage system went from 100,000 requests a second at peak in December 2019 to 750,000 in June 2020. The lesson: the open is a scheduled spike that arrives faster than any reactive system can respond.

Bad: autoscale on CPU and hope. Metrics lag by tens of seconds and new instances take minutes to boot and warm. By then the busiest seconds are over, the customers who tried to sell at 09:15 have seen errors, and their retries make the next minute worse (retry storm).

Good: scale by the calendar and smooth what you can. Scale the API, order and gateway tiers to the design peak by 08:45 (autoscaling done right) and warm what the first request touches: instrument master, balances and holdings, connection pools, exchange sessions. Then flatten: send overnight AMOs at a controlled rate in the pre-open window (09:00 to 09:08 at NSE), accept and hold orders placed between 09:08 and 09:15 and release them in arrival order, and jitter the app’s own morning refreshes. You pay for a few idle minutes of capacity daily — cheap next to an outage on a record day.

Great: admission control that never drops an accepted order, plus cells.

• Priorities at the door. Cancels and modifications first (they reduce risk), then new orders, then portfolio reads, then charts. Under overload the gateway sheds from the bottom — 503 with Retry-After for charts, a cached portfolio snapshot — while cancels still get through (backpressure and load shedding).
• Accepted means durable. A 202 means the order, block and outbox row are committed. If an exchange session is saturated, orders wait in the outbox in arrival order; nothing accepted is dropped.
• Per-user limits stop one script taking a cell’s capacity (rate limiting).
• Kill the herd at its source. Long-lived connections, cached service discovery and jittered retries keep a hiccup from becoming a synchronised reconnect.
• Cells. Split the whole account stack — services, ledger, database, exchange connectors, deploy pipeline — into independent cells, each owning a slice of customers.

Both brokers in the references arrived here. Robinhood’s brokerage system began 2020 on a single PostgreSQL database; rather than rewrite for a distributed database, it sharded the whole system at the application level, each shard with its own servers, database and deployment pipeline. A routing layer maps request to user to shard with durable, cacheable records; an aggregation layer fans out the rare cross-shard queries with a per-shard cursor; every shard consumes Kafka topics but processes only its own users. It went from one shard to three in 2020, then ten. Zerodha, growing from about 2 million clients in March 2020 to about 14 million by July 2024, went from one OMS installation to eight independent “silos” on separate racks across Mumbai and a Chennai disaster-recovery site.

Our estimate gave 10 cells of 1.5 million accounts at about 3,000 orders a second each (cell-based architecture, database sharding). You gain a blast radius of one tenth, risky changes shipped to one cell first, and growth by adding cells. You pay for a router on every request, an aggregation layer for cross-cell views, careful customer moves, and ten of everything to operate.

How do we keep portfolios, P&L and the books correct?

The portfolio is where the broker’s books meet the market’s, and they must agree to the share and the paisa.

Bad: a mutable holdings table that fills overwrite. A handler adds to quantity and recomputes avg_cost — until a replayed fill is counted twice or a crash splits the holdings and cash updates. There is no history to explain it, and the statement disagrees with the depository.

Good: positions as a projection of fills. A buy raises quantity and moves the average cost; a sell lowers quantity and books realised P&L. The fold is idempotent per trade ID, and any portfolio can be rebuilt by replaying fills. Only leveraged accounts need server-side revaluation, for margin: 300,000 accounts × 5 open positions = 1.5 million positions revalued each second from the full tick bus — easy for a stream processor. What the projection cannot see is the outside world: splits, dividends and the depository’s own records change holdings behind its back.

Great: the projection, plus corporate actions as events, plus four-way reconciliation.

• Corporate actions. A 1-for-5 split multiplies quantity by 5 and divides average cost by 5 on the ex-date, leaving value unchanged; a bonus adds shares at zero cost; a dividend credits cash to holders on the record date. Apply each as an event keyed by the action’s ID, and adjust historical candles too, so charts show no fake 80% crash. Zerodha reports that rebuilding its corporate-action processing cut discrepancies from up to a lakh items to about a thousand.
• Beginning of day. Rebuild sellable holdings and opening balances from what actually settled — Zerodha generates about 90 million holding records each morning — so the day starts from the depository’s truth.
• End of day, four ways. Our fills against the exchange’s trade files, our obligations against the clearing corporation’s, our holdings against the depository, our client cash against the bank. Each mismatch is a break with an owner. Then contract notes go out — Zerodha reports about 1.5 million signed PDFs in roughly 25 minutes.
• History elsewhere. Years of fills and P&L for statements live in a columnar store, not the cell databases (OLTP vs OLAP).

You gain a portfolio you can rebuild and defend, with breaks caught the same evening. You pay for a nightly pipeline that must finish before the next open, a team that owns breaks, and storage for every fill.

The final design

Put the pieces together. The money path lives inside a customer’s cell; market data is shared by everyone; reconciliation closes the day.

Follow Asha’s buy order, and her prices, through the system:

1. Admit and route. The API gateway authenticates her, applies her rate limit, ranks the request (below a cancel, above a chart) and sends it to her cell.
2. Record and block. The order service returns the stored answer for a repeated idempotency key; otherwise it runs in-memory checks, then in one transaction records the order, blocks ₹30,500 and writes an outbox row, and replies 202 SENT.
3. Send. The exchange link drains the outbox in order over a sequenced session to NSE (or the venue a US router picks), with the order’s client order ID. After any reconnect, the session’s numbering replays missed acks and fills.
4. Apply events. Each ack, fill, cancel or reject moves the state machine, turns the block into a debit or releases it, and goes onto the bus for the portfolio service.
5. Push. Stream gateways push the update to her WebSocket within a second, or a push notification if the app is in the background.
6. Prices. The feed handler takes two feeds per exchange and publishes ticks on the in-memory bus; gateways send each connection its on-screen symbols, snapshot first, conflated to four frames a second, and her phone computes live P&L.
7. Money. Deposits are credited on confirmed payment; withdrawals reserve the withdrawable amount at once and pay out in batches to her own bank.
8. Close the day. Reconciliation matches fills, obligations, holdings and client cash against the exchange, clearing corporation, depository and bank; contract notes go out.

Check it against the requirements. No overspending: the block and the order commit together. No lost or doubled orders: idempotency key, outbox, one client order ID, sequenced sessions and the nightly trade file. Speed: one transaction plus in-memory checks, well inside 20 ms at p50, and ten cells at about 3,000 orders a second give the 30,000/s design peak, with priorities keeping cancels alive. Prices: 2 million connections on 30 gateways at about 80 million conflated messages a second, at most 250 ms behind the exchange plus network. A bad cell touches 1.5 million customers, not 15 million.

What interviewers look for by level

Follow-up questions

What if our link to the exchange drops at 09:16?

Each cell holds sessions on at least two lines, so the connector fails over and logs on again; sequence numbers replay missed acks and fills, and orders without a confirmed outcome stay “unknown”, funds blocked, until the replay or a status query settles them. New orders wait in the outbox. If both lines stay down, stop accepting new orders for that exchange with a clear message rather than queueing indefinitely — but keep accepting cancels, and send them first when the session returns.

What if we must trigger stop-loss orders ourselves?

Where the exchange holds stop orders, send them with a trigger price. Otherwise — or for trailing stops — run a trigger engine: per instrument, sell stops sorted by descending trigger and buy stops ascending, fed from the full tick bus, never a conflated stream. Each tick pops every trigger it crosses and submits the child order through the normal order path, so funds checks and idempotency still apply. Partition by instrument and rebuild from the orders table on restart; a burst of triggers in a crash is exactly what admission control is for.

What if we add futures and options?

The skeleton holds, but three things grow. Every strike and expiry is an instrument, so the master and subscriptions balloon. Margin becomes a portfolio calculation where hedges reduce requirements, so the block comes from a risk engine and can rise after a trade when a hedge is closed. And margin must be monitored continuously from the full tick stream, with automatic square-off of positions that breach it.

What if we open an API for algorithmic traders?

Give API clients their own rate limits, and put heavy ones in dedicated cells so a busy strategy cannot slow the app (noisy neighbours and fairness). Offer the same WebSocket with larger limits — Kite allows up to 3,000 instruments per connection and three connections per API key — order updates by webhook too, and a per-account kill switch that cancels all open orders and blocks new ones.

What if a customer says they were filled at a terrible price?

Rebuild the moment: the order event log shows when we received, checked and sent the order and on which session; the tick archive shows the market at each instant; in the US, the routing log shows the venue and why. An unprotected market order in a thin book is a product fix — protection by default. If our own latency was to blame, the per-order timestamps show which hop.

How to say it in the interview

“I’m designing the broker, not the exchange: about 15 million accounts, 2 million app connections at the open, 10 million order requests a day with a busiest second around 25,000, and a hard rule that no order is lost, doubled or sent without its money.

I split it into two paths. Market data comes in once from each exchange through a feed handler, onto an in-memory bus with one topic per instrument. Stateless WebSocket gateways keep per-connection subscriptions to just the symbols on screen, conflate them to four updates a second, and send a snapshot first on every subscribe. That turns 800 million potential messages a second into about 80 million, and a slow phone simply gets fewer, newer prices.

The order path is small but exact. The phone sends an idempotency key. The order service runs static checks from memory, then in one database transaction records the order, blocks the cash or shares, and writes an outbox row, and replies 202. An exchange connector sends from the outbox with a client order ID over a sequenced FIX or exchange session, so after any reconnect missed acks and fills are replayed rather than guessed. Fills drive a strict state machine, turn blocks into debits, update the portfolio projection and get pushed to the app.

For the open, I pre-scale by the clock, drain overnight orders during pre-open, hold orders placed before 09:15, and use priority admission control so cancels always get through and nothing accepted is ever dropped. The account stack is split into ten cells of 1.5 million customers, each with its own database and exchange sessions — the same move Robinhood and Zerodha describe — so one bad cell is one tenth of the customers.

Every night I reconcile fills, obligations, holdings and client cash against the exchange, clearing corporation, depository and bank, and send contract notes. In the US the router also logs venue choices for best-execution review; in India, sells use the depository’s TPIN authorisation and upfront margin is blocked before every order.”

Concepts used in this design

• Real-time connections: long polling, SSE and WebSockets — the streaming channel for prices and order updates.
• Real-time updates — fan-out and reconnects at scale.
• Publish-subscribe — one topic per instrument on the tick bus.
• Backpressure and load shedding — conflation for slow clients and priority shedding at the open.
• Idempotency — the per-tap key that turns retries into lookups.
• Transactional outbox — recording an order and sending it without a dual write.
• Effectively-once with idempotent consumers — deduplicating replayed fills by trade ID.
• Message ordering and per-key queues — keeping each order’s events in sequence.
• Event sourcing and CQRS — holdings and positions as projections of fills.
• Lost update problem — why buying power needs a conditional write.
• Cell-based architecture — limiting the blast radius to one slice of customers.
• Database sharding — splitting accounts across cell databases.
• Autoscaling done right — scaling by the calendar for a predictable spike.
• Rate limiting — per-user limits that keep one script from filling a cell.
• Kafka — the durable event bus for orders and fills.

References and further reading

1. How we scaled Robinhood’s brokerage system for greater reliabilityrobinhood.comLoad growing from 100k to 750k requests a second in six months, and sharding the whole brokerage stack by customer with routing and aggregation layers.
2. An update from Robinhood’s foundersrobinhood.comThe March 2020 outages: record load, a thundering herd and a failed DNS system.
3. Zerodha’s infrastructure upgrades (Z-Connect)zerodha.comIndependent OMS “silos”, ~14 million clients, a ticker serving ~500k connections per process and ~175 million ticks a second at peak, feed failover, Kafka for margin data.
4. Kite Connect API: orderskite.tradeA real broker’s order varieties, types and validities, market protection, freeze-quantity slicing and interim order statuses.
5. Kite Connect API: WebSocket streamingkite.tradePer-instrument subscriptions, ltp/quote/full modes of 8, 44 and 184 bytes, batched binary packets, heartbeats and order updates on the same socket.
6. FIX session protocol (FIX Trading Community)fixtrading.orgA session as one ordered, numbered message stream that outlives individual connections and resynchronises after a reconnect.
7. Executing an order (Investor.gov, U.S. SEC)investor.govWhere US brokers send orders, payment for order flow, internalisation, and the duty of best execution and price improvement.
8. Intraday margin requirements (FINRA)finra.orgReplacing the pattern day trader rules with risk-based intraday margin from June 4, 2026, and how firms may monitor or block trades in real time.
9. Margins and margin penalties when trading with leverage (Zerodha Z-Connect)zerodha.comIndian peak-margin rules: random intraday snapshots by the clearing corporation and upfront-margin penalties that fall on the broker.
10. eDIS and TPIN (CDSL)cdslindia.comHow a client authorises a debit of demat shares for a sale with a depository-issued TPIN the broker never sees.