Design a stock exchange

A stock exchange is a machine for agreeing on prices. Brokers send it orders — buy 1,000 ACME at ₹1,000.15, sell 300 at ₹1,000.10, cancel that one — and it must decide, one message at a time, who trades with whom and at what price, tell both sides within microseconds, and show everyone what happened. The hard part is not data size: every order book of a large market fits in a few gigabytes of memory. It is four demands that pull against each other — one exact order of events, microsecond latency, zero lost trades, and fairness a regulator can check by replaying the day. Here you design the exchange itself, the way NSE or Nasdaq runs one: gateways, matching engine, market data and the hand-off to clearing. The app an investor taps is a separate system, covered in design a brokerage app.

Understand the problem

Agree on the vocabulary first. Only members — brokers and trading firms — connect to an exchange; investors reach it through them. An order names side, symbol, quantity and a limit price (or none, for a market order). Each symbol has an order book of resting buys (bids) and sells (asks). An order that reaches the best opposite price trades; otherwise it rests. Price-time priority decides who trades first: better price first, then earlier arrival — the rule Nasdaq’s order entry specification states for its own book.

Each trade sends a private execution report to both members and a public trade message to everyone. A clearing corporation then nets what each member owes and settles it — shares between depository accounts, money between banks — one business day later, T+1. NSE Clearing runs all equity settlement on T+1, and the US moved from T+2 to T+1 for trades on or after May 28, 2024.

Functional requirements

1. Members should be able to place, modify and cancel orders — limit and market orders, valid for the day or immediate-or-cancel — and get an acknowledgement or a rejection for every message.
2. The exchange should match orders by price-time priority and report every execution to both sides.
3. Everyone should be able to receive market data: every order event and trade, and the best bid and ask, in exactly the order it happened.
4. Every trade should reach the clearing corporation for netting and T+1 settlement.
5. Operators should be able to run the trading day: a pre-open call auction, continuous trading and a close, plus halts for one symbol or the whole market.

Non-functional requirements

• Latency. Gateway in to acknowledgement out: p50 under 50 µs and p99 under 250 µs for a co-located member. Public data leaves with the private reports.
• Throughput. 5,000 symbols, about 2 billion order messages a day (new, modify, cancel), peaks near 1 million a second after the open. Design for 2 million.
• Deterministic order. One sequence of inputs per symbol, and the same sequence always produces the same trades, so any day can be replayed exactly.
• Zero loss. No acknowledged order or executed trade is lost or changed by a crash or failover inside the primary site.
• Availability. 99.99% of trading hours: 6.25 hours × 250 days = 1,562.5 hours, so about 9 minutes of disruption a year. Failover must take well under a second.
• Fairness. Time priority is decided at one point; no gateway or clock gives anyone a better place in the queue.
• Data size. Books fit in a few gigabytes; the input journal is about 200 GB a day and is kept for years.

Out of scope

• The broker’s side — client accounts, KYC, margin, the app. That is the brokerage app.
• Clearing internals — netting, margin, collateral, defaults. The accounting is double entry; see design a double-entry ledger.
• Futures and options — see the follow-up questions.
• Surveillance, listings and index calculation. They read our journal and feeds or supply reference data; none sits on the order path.

The set-up

Plan the approach

Say the shape early: “a deterministic state machine behind a sequencer, with fan-out on the way out.” Meet each requirement simply, then spend the time in the deep dives: the book, threading, durability, latency and fairness, market data, risk and partitioning.

Core entities

• Member and session. A firm, its ports and credentials, with message-rate and order-size limits.
• Instrument. Symbol, a small integer locate code used as an array index (Nasdaq’s feed assigns these daily), tick size, lot size, price band and state: pre-open, open, halted, closed.
• Order. Exchange order ID, member and client reference, side, type, limit price in integer ticks, original and remaining (“leaves”) quantity, time in force, and the sequence number that fixes its time priority.
• Price level. One price on one side: total quantity and a FIFO queue of orders.
• Order book. Bids and asks for one instrument, with the best of each at hand.
• Sequenced event. Sequence number, timestamp and payload — new order, cancel, replace, phase change, halt. The only input the engine accepts.
• Execution. Match ID, buy and sell orders, price, quantity, aggressor side, and the sequence number of the input that caused it.

Prices never touch floating point: both Nasdaq protocols carry them as integers with four implied decimals, so ₹1,000.15 travels as 10001500.

API

Three surfaces: private order entry (a persistent session per member port), public market data, and a post-trade feed to clearing. Order entry is really a binary protocol over TCP — Nasdaq’s OUCH Enter Order message is 47 bytes before optional fields — or FIX; it is shown as JSON for readability.

Enter an order. client_ref is the member’s own number for the order. OUCH requires it to be strictly increasing per port and ignores a new order whose number is not above the last one processed, so a member can safely resend anything it is unsure about after a disconnect: idempotency at microsecond cost.

{
  "type": "enter_order",
  "client_ref": 900417,
  "symbol": "ACME",
  "side": "buy",
  "order_type": "limit",
  "price": 10001500,
  "qty": 1000,
  "time_in_force": "day"
}

Replies arrive asynchronously on the same session: an acceptance, then one message per fill. Timestamps are nanoseconds since midnight, as in both Nasdaq protocols.

{ "type": "order_accepted", "client_ref": 900417, "order_id": "P3-1042", "seq": 1042, "ts_ns": 33300000123456 }
{ "type": "executed", "client_ref": 900417, "match_id": 77001, "qty": 200, "price": 10001000, "liquidity": "taker" }
{ "type": "executed", "client_ref": 900417, "match_id": 77002, "qty": 100, "price": 10001000, "liquidity": "taker" }
{ "type": "executed", "client_ref": 900417, "match_id": 77003, "qty": 500, "price": 10001500, "liquidity": "taker" }

Modify and cancel. The two kinds of change treat time priority differently — interviewers like this detail:

cancel_order   client_ref=900417  new_qty=0         remove what is left
cancel_order   client_ref=900417  new_qty=100       shrink to 100 shares; keeps its place in the queue
replace_order  client_ref=900417  new_ref=900422    new price or larger size; goes to the back of the queue
               price=10002000 qty=200
mass_cancel    symbol=ACME (optional)               cancel every resting order on this port

OUCH spells it out: a replace always gets a new timestamp for time priority, while a partial cancel keeps it. Shrinking harms nobody behind you; raising price or size would jump the queue.

Market data is a stream of small binary events, modelled on Nasdaq’s TotalView-ITCH 5.0, from which any subscriber can rebuild every book.

Recovery is request-response, used only when a subscriber misses data:

REQUEST   session=20261027P3  seq=5001  count=3   →  unicast resend of messages 5,001–5,003
SNAPSHOT  channel=P3                              →  every book as of sequence N; then replay from N+1

Post-trade, each execution goes to the clearing corporation as a record:

{
  "match_id": 77001,
  "trade_date": "2026-10-27",
  "symbol": "ACME",
  "qty": 200,
  "price": 10001000,
  "buy_member": "M041",
  "sell_member": "M112",
  "seq": 1042
}

Back-of-envelope estimates

These decide how many engines you need, how fast the journal writes and whether market data can be unicast (method: back-of-envelope estimation).

Message rate. NSE’s continuous session runs 09:15 to 15:30: 6.25 × 3,600 = 22,500 seconds. 2,000,000,000 ÷ 22,500 ≈ 88,900 messages/s on average. The open, the close and news bunch traffic, so take 10× for the peak — about 890,000/s, call it 1 million — and design for 2 million/s.

Trades. Assume 2% of messages produce a trade: 2,000,000,000 × 0.02 = 40 million trades a day, about 40,000,000 ÷ 22,500 ≈ 1,780/s on average and 17,800/s at the 10× peak.

Input journal. A sequenced input — order fields plus sequence number, timestamp and member — is about 100 bytes. 2,000,000,000 × 100 B = 200 GB a day; at the 2 million/s design point, 200 MB/s of sequential appends. 250 trading days × 200 GB = 50 TB a year before compression.

Market data. Most inputs produce one public event; a trade produces several. Assume 1.2 events per input at 32 bytes plus a 2-byte length = 34 bytes. At 1 million inputs/s: 1,200,000 × 34 B ≈ 41 MB/s, plus about 5% for packet headers (a 1,500-byte packet holds around 42 messages) — ≈ 43 MB/s, about 345 Mbit/s at peak. Over TCP to 1,000 subscribers that would be 1,000 × 345 Mbit/s = 345 Gbit/s.

Book memory. Say 10 million orders rest at once, 2,000 per symbol. At 64 bytes each (one cache line): 10,000,000 × 64 B = 640 MB. An order-ID hash map at 16 bytes per entry, half full: 10,000,000 × 16 B ÷ 0.5 = 320 MB. The price-level arrays from the deep dives take 256 KB per symbol: 5,000 × 256 KB ≈ 1.28 GB. Total about 2.2 GB, roughly 280 MB per engine across eight.

Engine capacity. Martin Fowler reports LMAX’s single business-logic thread processing 6 million orders a second. Budget far less: 500,000 messages/s per thread, 2 µs each. 2,000,000 ÷ 500,000 = 4 engines minimum; use 8, so a hot partition can run at twice the average within budget.

High-level design

Build one requirement at a time; by the end the exchange works for one partition of symbols, without failover.

Members should be able to place, modify and cancel orders

Three components carry an order. An order gateway terminates member sessions: it authenticates, decodes, checks (symbol, tick, quantity) and rejects junk at once. A sequencer puts valid messages from every gateway into one line, stamping each with the next sequence number and a timestamp and appending it to a journal. The matching engine applies that line, in order, to the in-memory books.

1. The broker sends enter_order over its session to a gateway.
2. The gateway checks it. A price of ₹1,000.12 when the tick is ₹0.05 is rejected here and never reaches the core.
3. The sequencer stamps it #1042 and journals it. From now on #1042 is the order’s place in time.
4. The engine applies #1042 to ACME’s book: it trades, rests, or both.
5. Results flow back through the gateways: acceptance and fills to the buyer, fills to every seller it met.

Cancels and replaces take the same path. They must: a cancel and a trade on the same order race, and only one ordering can be true. If the cancel was sequenced first, the order is gone; if the incoming sell was, the order traded and the cancel is rejected as too late.

The exchange should match orders by price-time priority

The rule fits in three lines. An incoming buy at limit P trades against the lowest ask while that ask is at or below P. Within a price, the oldest resting order fills first. Each trade happens at the resting order’s price, so the newcomer gets any improvement. Sells mirror this. A day order’s remainder rests at its limit; an immediate-or-cancel remainder is cancelled. A market order is treated as a limit at the edge of today’s price band, so it cannot walk an empty book to an absurd price. Interviewers often ask for a worked example:

Check the numbers. B9 takes 300 shares at ₹1,000.10 (S1, then S2) and 500 at ₹1,000.15 (S3): 300 × 1,000.10 + 500 × 1,000.15 = 300,030 + 500,075 = ₹800,105 for 800 shares, an average of 800,105 ÷ 800 ≈ ₹1,000.13. The next ask is above the limit, so 200 shares rest as a bid at ₹1,000.15, and the spread shrinks from ₹0.10 to one tick.

One input made a lot of output: an acceptance and three fills to B9’s member, a fill each to the members behind S1, S2 and S3, four public events, and three trade records for clearing. That fan-out is why the output side needs as much design as matching. The loop itself is short; prices are integer ticks and nothing allocates, blocks or reads a clock:

def match(book, order):
    """Apply one sequenced order to its book. Returns nothing; emits events."""
    opposite = book.asks if order.side == BUY else book.bids
    while order.leaves > 0:
        level = opposite.best()                  # lowest ask or highest bid
        if level is None or not crosses(order, level.price):
            break
        resting = level.head()                   # oldest order at this price
        qty = min(order.leaves, resting.leaves)
        emit_trade(level.price, qty, aggressor=order, passive=resting)
        order.leaves -= qty
        resting.leaves -= qty
        level.total -= qty
        if resting.leaves == 0:
            level.pop_head()
            book.by_id.pop(resting.id)
            if level.is_empty():
                opposite.remove(level)           # best() now finds the next price
    if order.leaves > 0 and order.tif == DAY:
        book.rest(order)                         # joins the tail of its price level
        emit_add(order)
    elif order.leaves > 0:
        emit_cancelled(order)                    # IOC remainder never rests

Everyone should be able to receive market data

Give the engine one output stream, in input order, and let separate consumers read it. A market data publisher turns each book change into a small event — add, executed, cancel, delete, replace — with its own feed sequence number. A subscriber that applies them in order holds an exact copy of every book; the ITCH specification tells subscribers to subtract each cancel and execution from the original add until the order reaches zero. A top-of-book feed is a smaller view of the same stream.

Two rules keep the feed honest: it is built from the engine’s output, never a separate copy of the books, and it leaves with the private execution reports, not after them. How to deliver 345 Mbit/s to a thousand subscribers is a deep dive.

Every trade should reach the clearing corporation

A trade capture service reads the same stream, keeps only executions, writes them to a trade store and forwards them to the clearing corporation, plus an end-of-day file both sides reconcile. It is off the hot path: if it falls behind, matching carries on and it catches up by sequence number.

The clearing corporation does the money work. NSE Clearing uses multilateral netting: each member ends the day with one net position per security and one net cash amount — a broker that bought 50,000 ACME and sold 48,000 receives 2,000. Obligations are published provisionally by 9 PM on trade day and finally by 9 AM next day, and settle on T+1. The postings behind that are double-entry bookkeeping; see design a double-entry ledger.

Trades are never edited. To break an erroneous trade the exchange sends a new event — OUCH has a Broken Trade message, always after the original execution — so history stays append-only.

Operators should be able to run the trading day

A trading day is a sequence of phases. NSE’s equity timetable is a good template: pre-open 09:00 to 09:08, continuous trading 09:15 to 15:30, and a closing session 15:40 to 16:00. Nasdaq has an Opening and a Closing Cross, publishing imbalance data in the minutes before each.

In the pre-open the engine collects orders without matching. Then it runs a call auction: it finds the one price at which the most shares can trade, executes them all at that price, and carries the rest into continuous trading. ACME’s pre-open book, as cumulative quantities:

At ₹1,000, 2,800 shares can trade — more than at any other price — so that is the opening price. Every trade prints at ₹1,000, whatever each buyer bid, and the 3,500 − 2,800 = 700 unfilled buy shares carry over as resting bids. The auction turns the opening rush into one batch at one price — a good way to re-open after a halt, too. (Unlike the ascending-bid online auction, it has many sellers, many buyers and one clearing price.)

The key choice is in the notice: a session scheduler does not flip a flag inside the engine; it submits a PHASE event through the sequencer like any order, and so do symbol and market-wide halts. The engine never reads a clock, and a replay reproduces exactly where each phase began among the orders.

You now have a working exchange for one partition. The deep dives make it fast, unbreakable, fair and big.

Deep dives

How do we store the order book and match in microseconds?

A book needs four operations: add at a price, find the best opposite price, take the oldest order there, and cancel by ID. Cancels matter more than people expect: we assumed only 2% of messages trade, so most traffic is orders being added, cancelled and replaced. Removing an order must be as cheap as adding one.

Bad: rows in a database. An orders table indexed on symbol, side, price and time is easy, and fine at a few bids a second. Here every order becomes an index scan, row updates per fill and a commit — milliseconds, not microseconds — with every order on a busy symbol fighting for the same rows’ locks: the busy database anti-pattern.

Good: a sorted map of price levels plus a hash map of orders, in memory. Each side is a balanced tree keyed by price, each node a FIFO queue. A hash map from order ID to queue node makes cancels constant-time: look up, unlink, subtract from the level. A new price level costs O(log L); the best price is the first node. You gain fast, simple code. You pay in pointer chasing and allocation: every level and order is a scattered heap object, and with a garbage collector each allocation is a future pause.

Great: an array indexed by tick, intrusive lists and preallocated pools. Price bands allow a striking simplification. If ACME closed at ₹1,000 with a 20% band, every legal price lies between ₹800 and ₹1,200: with a ₹0.05 tick, (1,200 − 800) ÷ 0.05 + 1 = 8,001 prices. Each side becomes an array of 8,001 slots, and a price’s slot is arithmetic. Each slot holds the level’s total and the head and tail of a doubly linked list threaded through the order records themselves (“intrusive”). Order records come from a pool allocated before the open, so nothing allocates while trading. When the best level empties, a bitmap of occupied slots finds the next one in a few instructions.

What you pay: empty slots — 8,001 × 16 bytes ≈ 128 KB per side, the 1.28 GB in the estimates — which is cheap. Pools must be sized up front, with a clear rejection when one runs out. And the array needs a bounded range: NSE sets no fixed daily band on stocks with derivatives, only a 10% operating range, so for those re-centre the array as the price drifts or fall back to the sorted map.

Why is matching single-threaded, and how do we still use many cores?

Matching is a connected domain: every order changes the book the next one sees. Fowler names exactly this as the sign that the LMAX architecture fits.

Bad: a thread pool on shared books, guarded by locks. Two threads handling ACME take the same lock, so they run one at a time anyway, plus the lock’s cost. Worse, the operating system decides who wins the lock, so the order of events — and who trades — is no longer reproducible. The LMAX team measured what coordination costs:

Good: one thread per book, fed by a queue. Each book has a single writer, so no locks, and the queue fixes the order. But 5,000 threads on a few dozen cores means constant context switches, and conventional queues are slow: the Disruptor paper measured a mean of 32,757 ns per hop through a blocking queue in a three-stage pipeline against 52 ns for its ring buffer, mostly due to locks and condition-variable signalling.

Great: the LMAX pattern — one business-logic thread per partition, fed by a ring buffer, with all I/O elsewhere. One thread owns many books, all in memory, and processes its sequenced input one message at a time. It never does I/O, takes a lock or allocates. Decoding, journaling, replicating and encoding run on other cores in parallel, linked by a pre-allocated ring buffer in which each stage writes only its own sequence counter. Fowler describes LMAX doing all its business logic on one thread at 6 million orders a second: sequential in-memory code alone gave about 10,000 transactions a second, well-factored code with small methods about 100,000, and cache-friendly custom collections the last order of magnitude.

The price is the programming model. The engine can never call out — one slow call stalls every book on the thread — so anything external becomes an output event whose answer returns later as an input. And with nothing to roll back, every input is fully validated before it touches the book.

How do we survive a crash without losing or changing a trade?

A crash wipes the in-memory books, yet no acknowledged order or trade may be lost, and the market must never see two versions of events.

Bad: snapshot the books every few seconds. After a crash, everything since the last snapshot is gone — though members already hold acknowledgements and fills for it, and the feed printed it.

Good: stream state changes to a standby asynchronously. Fast, but it lags: if the primary dies just after sending a fill the standby never received, the standby takes over without that trade. And shipping state means trusting whatever the primary computed, bugs included.

Great: replicate the inputs, not the state, and process only what two machines hold. This is event sourcing with discipline: the sequenced input log is the source of truth, the books are derived. Three rules:

1. Durable before acted on. The sequencer journals each input and sends it to the standby; the primary processes input N only after the standby acknowledges it. LMAX likewise journals and replicates every input before its business logic sees it, with the leader fixing one order because messages can reach nodes in different orders.
2. Same input, same output. The standby applies every input in the same order with outputs switched off, so its books match the primary’s at every sequence number.
3. Every output names its input. Reports and feed events carry the causing input’s sequence number, and members’ client references only increase, so duplicates after a failover are dropped.

Two more pieces finish the design.

Who decides the primary is dead? If the standby promotes itself whenever heartbeats stop, a network blip yields two primaries matching the same symbols — split brain. Put leadership through a small consensus group: the two engines plus a lightweight witness, so two of three agree who leads and a cut-off node knows to stop. The general form, “an input is committed once a majority holds it”, is the replicated log behind Raft.

How does a crashed node come back? Replaying a whole day is slow: one partition’s 250 million daily messages at a replay speed of 2 million/s take 250,000,000 ÷ 2,000,000 = 125 seconds. So a non-voting follower snapshots its books every minute without pausing anyone. A restarted node loads a snapshot (about 280 MB) and replays at most 60 s × 250,000 messages/s = 15 million messages: 15,000,000 ÷ 2,000,000 = 7.5 seconds. LMAX reports the same shape — snapshot plus a day of journal restarts in under a minute, while switching to a running replica takes microseconds.

What you pay: an extra in-rack round trip before every match (about 10 µs in the budget below), a standby doing full work for nothing, and a strictly deterministic engine.

How do we keep latency in microseconds and arrival order fair?

Start with a budget, because “fast” is not a design:

Matching is 2 µs because that is our per-message budget; the rest is moving bytes. The usual techniques, at interview depth: members co-locate servers in the exchange’s data centre; network cards use kernel bypass, so packets reach user space without system calls; hot threads are pinned to cores and busy-spin; messages are fixed-length binary; nothing allocates on the hot path. The p99 target of 250 µs is the queueing budget — 1,000 orders arriving at once behind a 2 µs-per-message thread means the last waits 2 ms, which is why auctions, throttles and partitions follow.

The other half is fairness, which is about where time is defined.

Bad: trust the member’s timestamp. Clocks differ, and anyone can lie.

Good: timestamp at each gateway with synchronised clocks. Gateways still disagree by their clock error, so priority depends on which gateway an order used.

Great: the partition’s sequencer is the only clock that matters. Priority is the sequence number, assigned at one point. Then remove physical advantages: the same gateway hardware and the same cable length for every co-located member; per-session rate limits so one member’s flood cannot queue ahead of everyone (mechanics in the rate limiter design); public data released with private reports. Inside the engine, “now” is the sequencer’s stamp on the current message, so replays reproduce every timestamp — the clocks and ordering problem solved by having one clock.

How do we publish market data to thousands of subscribers?

The feed peaks near 345 Mbit/s, and brokers, vendors, surveillance and the index calculator all want every byte within microseconds.

Bad: a TCP connection per subscriber. Ordering and retransmission come free, but every byte is sent once per subscriber — 1,000 × 345 Mbit/s = 345 Gbit/s — and a slow reader’s full buffer stalls its stream, leaving the publisher juggling a thousand queues.

Good: UDP multicast with sequence numbers. The network does the copying. In Nasdaq’s MoldUDP64, each packet is sent once regardless of listeners and starts with a 20-byte header: a 10-byte session, the 8-byte sequence number of its first message, and a 2-byte count. A receiver expecting 5,001 that sees 5,004 knows exactly what it missed. Heartbeats — a count of zero, typically once a second — carry the next sequence number so losses show up even when quiet; a count of 65,535 ends the session.

Great: multicast plus recovery, redundancy and tiers. Detecting a gap is not repairing it, so add:

• Re-request servers. The receiver sends session, first sequence and count, and gets the messages back by unicast as a normal packet on its multicast socket — as many as fit in one packet, asking again for more. The spec suggests caching re-request servers near distant receivers.
• Snapshots for late joiners. A subscriber starting at 11:00 gets every book as of sequence N, then applies the live feed from N + 1. The ITCH specification points to Nasdaq’s GLIMPSE service for this.
• A and B feeds. The same stream on two separate networks; receivers take each sequence number from whichever copy arrives first.
• Channels per partition, so subscribers join only what they need. ITCH puts the stock locate code at the same offset in every message for cheap filtering.
• A conflated tier for people. A retail app needs top of book a few times a second, not 345 Mbit/s. A downstream service samples it and brokers fan it out over WebSockets — the real-time connections problem of the brokerage app.

What you pay: multicast needs a network you control end to end — a co-location hall or dedicated lines, not the internet — and receivers must handle gaps, A/B duplicates and recovery themselves. Bandwidth binds subscribers too: Nasdaq requires a 10 or 40 Gb connection for the unthrottled hardware version of its feed.

How do we stop bad orders and wild prices without slowing the hot path?

Someone types 1,000,000 shares instead of 1,000, or ₹10,000 instead of ₹1,000. The exchange must stop such orders, and prices collapsing in seconds, without a round trip per order.

Bad: call a central risk service per order. A synchronous round trip adds at least two of the budget’s 5 µs hops plus the check itself, and a hot-path dependency: if it slows, the market slows.

Good: all checks in the gateway, with cached reference data. Fast and parallel, but some checks need state only the engine has: a halt a microsecond ago, the last trade price, the member’s resting orders.

Great: split checks by the state they need — stateless in the gateway, in parallel; stateful inside the engine, in log order, as part of the deterministic state machine.

The checks come in three layers:

• Per-order limits. A quantity cap — OUCH requires fewer than 1,000,000 shares — and a notional cap per member, so 10,000 shares at ₹10,000, a ₹10 crore order, is stopped by configuration, not luck.
• Price bands. NSE sets daily bands of 2%, 5%, 10% or 20% either way; with a 10% band on a ₹1,000 close, an ACME buy at ₹1,150 is rejected at the gateway. A static band cannot stop a slide inside it, so the engine also runs a dynamic band. The US Limit Up-Limit Down rule is a documented example: 5%, 10% or 20% bands around the previous five minutes’ average price; if the price sits at a band for 15 seconds, trading pauses for five minutes; bands double in the last 25 minutes for many stocks.
• Market-wide circuit breakers. In the US, an S&P 500 fall of 7% or 13% before 3:25 pm halts all stocks for 15 minutes; 20% halts the rest of the day. That crosses partitions: an index calculator reads trades from every cell, and the control plane writes HALT_ALL into every cell’s sequencer. Each halts at its own sequence number and re-opens with a call auction.

Add a kill switch — disable a port and mass-cancel its orders in one message (OUCH has Mass Cancel and Disable Order Entry requests) — and offer cancel-on-disconnect so a crashed algorithm leaves no stale orders. What you pay: limits configured in two places, and extra work on the engine thread, so keep stateful checks to a few comparisons on data it already holds.

How do we scale to thousands of symbols and survive the opening bell?

No trade in ACME depends on any other book, so matching partitions naturally by symbol.

Bad: one engine for the market. Simple, until the peak exceeds one thread and everything slows together.

Good: hash symbols across 8 engines, about 625 each. But load is uneven: index heavyweights and the day’s meme stock carry outsized shares, and two hashed together blow one engine’s budget while others idle — a noisy neighbour in the core.

Great: a load-aware map with hot symbols isolated. Each partition is a matching cell — sequencer, primary, standby, journal and its own multicast channel. Before each day, assign symbols from yesterday’s message counts so every cell’s expected peak sits well under 500,000 messages/s, and give the busiest symbols their own cells. Gateways route with the same map, which changes only between sessions or during a halt: moving a symbol means snapshotting its book and replaying it into the new cell.

Check the arithmetic: 5,000 − 1 − 3 = 4,996 symbols for six ordinary cells, 4,996 ÷ 6 ≈ 833 each. At the 2 million/s design peak cells average 250,000 messages/s; P3 runs at 380,000 ÷ 500,000 = 76% of budget.

Use a sequencer per cell. A global one would impose a cross-symbol order matching never needs, and become everyone’s bottleneck and single point of failure. Global actions — a market halt, a member’s mass cancel — fan out to every cell as separate events.

The opening bell is the day’s sharpest burst, like a flash sale’s first second. The call auction absorbs most of it — pre-open only inserts, and one uncross replaces thousands of matches. What remains at 09:15 meets the headroom (8 cells for a 4-cell load), gateway throttles and the p99 queueing budget.

The final design

Each partition of symbols is a matching cell; everything else either feeds it in sequence or reads its ordered output.

Follow one order through the system:

1. Enter. A broker sends an order to a gateway, which runs the stateless checks — tick, lot, quantity cap, band, fat-finger notional, message rate — and rejects bad orders before they cost the core anything.
2. Sequence. The gateway routes by today’s symbol map to the cell’s sequencer, which stamps a sequence number and time, journals the order and replicates it to the standby. It is now durable on two machines and has its place in time.
3. Match. The primary engine’s single thread runs the stateful checks and price-time matching on tick-indexed arrays. The standby applies the same input, muted; a follower snapshots for restarts.
4. Report. Acknowledgements and fills, tagged with the input’s sequence number, go back through the gateways to every member involved.
5. Publish. The market data publisher multicasts order-by-order events on A and B feeds; recovery servers answer re-requests and snapshot requests. A conflated retail feed is derived downstream.
6. Clear. Trade capture stores executions and sends them to the clearing corporation, which nets each member’s obligations and settles on T+1.
7. Control. Phases, halts, circuit breakers and kill switches enter through an administrative session as sequenced events, ordered against orders and replayable.

Check it against the requirements: about 35 µs typical latency against a 50 µs p50 target; 8 cells of 500,000 messages/s against a 2 million/s design peak; nothing lost, because nothing is processed until two machines hold it; failover to a running standby, with consensus against split brain; one replayable order per symbol; fairness fixed by one sequencer per cell; about 345 Mbit/s of market data sent once per network; and every trade delivered for T+1 settlement.

What interviewers look for by level

Follow-up questions

What if a member’s algorithm goes rogue?

Gateway rate limits and per-order caps bound how fast and how big it can go. A kill switch — disable the port and mass-cancel in one message — stops it in milliseconds, and cancel-on-disconnect clears its orders if the session drops. The journal lets surveillance reconstruct exactly what it did.

What if the whole primary data centre is lost?

Within a site, replication is synchronous. Across sites, a round trip of a millisecond or more breaks the budget, so the disaster-recovery site receives the log asynchronously and may be slightly behind. After a site loss, trading halts, members reconcile their orders against the last sequence numbers the DR site holds — client references make this mechanical — and trading resumes with a call auction. You trade a non-zero recovery point for microseconds every normal day; see disaster recovery.

What if a regulator asks what happened at 10:31:07?

Load the snapshot before that time into an offline engine and replay the journal to that sequence number. Being deterministic, it reproduces every book, trade and rejection exactly, timestamps included. LMAX debugs the same way, replaying production events on a development machine.

What if we add futures and options?

Each contract is another book, so the core holds, but three things change. Every strike and expiry is a book, so instrument counts and memory jump. Multi-leg orders (“buy the spread”) touch several books, so related contracts must share a cell to keep matching single-threaded. And members trade on margin, so the clearing corporation’s real-time margin checks join pre-trade risk.

What if one stock suddenly trades 50 times its usual volume?

A typical symbol in an ordinary cell carries about 250,000 ÷ 833 ≈ 300 messages/s at the design peak, so even 50× — 15,000/s — fits in the 250,000/s of spare budget. Price bands and dynamic-band pauses slow the frenzy, and throttles stop any one member flooding the cell. Overnight, the map gives the stock a cell of its own. What you never do is move a book while it trades.

How to say it in the interview

“I’m designing the exchange itself — the venue that matches orders — for about 5,000 symbols, 2 billion messages a day and peaks around a million a second, with acknowledgements in tens of microseconds and no lost trades.

Members connect over a binary protocol to order gateways, which run stateless checks like tick size, quantity caps and price bands, and route each order by symbol to a matching cell. In each cell, a sequencer gives every input a sequence number and timestamp, journals it and replicates it to a standby. Only when two machines hold the input does the primary engine process it. That sequence number is the order’s place in time, so fairness is decided in exactly one place.

The engine is single-threaded and keeps all its books in memory, LMAX-style: no locks, no I/O, no allocation on the hot path. Each book is an array of price levels indexed by tick within the day’s band, with a FIFO queue per level and a hash map from order ID for constant-time cancels. Matching is price-time priority, trading at the resting order’s price.

Because the engine is deterministic, the input log is the source of truth. The standby applies the same inputs with outputs muted, so failover is just unmuting it, with a small consensus group to avoid split brain. Outputs carry the input’s sequence number, and members resend with increasing client references, so nothing is duplicated.

Outputs fan out from one ordered stream: execution reports back to members, an order-by-order feed over UDP multicast with sequence numbers, re-request servers and snapshots for gaps, and trade capture sending executions to the clearing corporation for netting and T+1 settlement.

To scale, I partition symbols into eight cells using a load-aware map, with the hottest symbols in cells of their own, each budgeted at half a million messages a second. The opening call auction turns the morning rush into one uncross, and circuit breakers and halts are just events in the log, so any day can be replayed exactly.”

Concepts used in this design

• Event sourcing and CQRS — the sequenced input log is the source of truth; the books are derived state.
• Write-ahead logging — journal every input before acting on it.
• Data replication — synchronous replication of inputs to a hot standby.
• Consensus and leader election — deciding which engine leads without a split brain.
• Failover — switching to a standby that already holds identical books.
• Single point of failure — why each cell has its own sequencer and none of them stands alone.
• Clocks, ordering and conflict resolution — one sequencer as the only clock that decides priority.
• Message ordering and per-key queues — strict order per symbol, parallelism across symbols.
• Concurrency vs parallelism — single-threaded engines, many partitions.
• TCP vs UDP — TCP sessions for orders, UDP multicast for the feed.
• Publish-subscribe — one ordered output stream, many independent readers.
• Idempotency — safe resends using strictly increasing client references.
• Rate limiting — per-session message throttles at the gateway.
• Database sharding — partitioning symbols across matching cells.
• Noisy neighbour — why hot symbols get cells of their own.
• Disaster recovery — asynchronous replication to a second site and its recovery point.

References and further reading

1. The LMAX Architecture (Martin Fowler)martinfowler.comSingle-threaded, in-memory business logic at 6 million orders a second; input journaled and replicated before processing; snapshots, sub-minute restarts and replica failover.
2. LMAX Disruptor: high performance alternative to bounded queues for exchanging data between concurrent threadslmax-exchange.github.ioMeasured cost of locks and CAS against a single thread, and 52 ns versus 32,757 ns mean latency per hop against a blocking queue.
3. Nasdaq TotalView-ITCH 5.0 specificationnasdaqtrader.comOrder-by-order market data messages and their byte layouts, daily stock locate codes, nanosecond timestamps and delivery over MoldUDP64.
4. MoldUDP64 protocol specification (Nasdaq)nasdaqtrader.comSequenced UDP multicast: the 20-byte packet header, heartbeats, end of session, and re-request servers for gap recovery.
5. Nasdaq OUCH 5.0 order entry specificationnasdaqtrader.comBinary order entry, price-time priority, strictly increasing UserRefNum for safe resends, replace versus cancel and time priority, mass cancel.
6. Stock market circuit breakers (Investor.gov, U.S. SEC)investor.govUS market-wide circuit breaker levels and the Limit Up-Limit Down single-stock price bands.
7. New T+1 settlement cycle: what investors need to know (Investor.gov, U.S. SEC)investor.govThe US move from T+2 to T+1 for trades on or after May 28, 2024.
8. NSE: equity market timingsnseindia.comPre-open session 09:00 to 09:08, normal market 09:15 to 15:30, closing session 15:40 to 16:00.
9. NSE: price bandsnseindia.comDaily price bands of 2, 5, 10 and 20 percent, and the 10 percent operating range for stocks with derivatives.
10. NSE Clearing: equities clearing and settlementnseindia.comMultilateral netting, obligations by 9 PM on trade day and 9 AM on T+1, and T+1 settlement for all equity trades.