Senior engineers rarely draw different boxes from everyone else. They make different decisions about the same boxes, and they can say why. This page distils that judgement into ten principles: one-line rules you can defend with a number or a public example, each with how to apply it, what it prevents, what it costs and when to break it.
Picture it first: the head chef on a Saturday night
Watch an experienced head chef run a 120-cover restaurant on a Saturday. Nothing she does is clever; all of it is deliberate.
- She preps for tonight’s bookings plus a margin. A 2,000-guest wedding would need a different kitchen.
- The menu is dishes her line can cook half asleep, plus one new special at a time.
- Every order lives on a ticket, not in a cook’s head, so any cook can pick it up.
- The last four portions of biryani are counted in one place; the printed menu can be a day out of date.
- Tickets carry a table number, so an order called twice is cooked once.
- Guests don’t wait while plates are washed; washing happens in the background.
- If the tandoor dies, the menu loses naan, not the evening.
- She times ticket to table and watches the slowest table, not the average one.
- A new dish runs as a special that can be pulled mid-service. Moving the restaurant is a different kind of decision.
- She knows the food cost and staff hours behind every plate.
Map it back: covers are requests, tickets are state in a durable store, the biryani count is strongly consistent inventory, the table number is an idempotency key, the dish pile is a queue, the special is a feature flag and the slowest table is your p99.
The ten at a glance
The principles follow a system through its life: decide, build, run, evolve. Each one is really a question you ask of every box.
| Principle | The question it makes you ask | Anti-pattern it prevents | Cost of following it |
|---|---|---|---|
| Start from requirements and numbers | What is the real load, now and in two years? | Building for 100× on day one | A planned rebuild at around 10× |
| Prefer boring technology | Can we do this with what we already run? | A new database per access pattern | Clumsy fits for specialised jobs |
| Keep compute stateless, state explicit | Where does this state live? | Sticky sessions, carts in server memory | A network hop for every state read |
| Choose consistency per operation | What does a stale answer cost, compared with no answer? | All-strong (slow) or all-eventual (oversold) | Two models to explain and test |
| Make operations safe to retry | What happens if this runs twice? | Double charges after a timeout | Key storage, plus checks on every write |
| Decouple in time | Must the user wait for this result? | Checkout failing because e-mail is slow | Queues to run, harder tracing |
| Design for failure | What happens when this box dies? | A single database with no standby | Spare capacity and failover that can misfire |
| Measure what users feel | Which percentile, against which SLO? | Green dashboards while users see spinners | Instrumentation, storage and people |
| Make change safe and reversible | Can we undo this in minutes? | Big-bang migrations, global config pushes | Slower rollouts, flag and version debt |
| Optimise for total cost and operability | What does it cost per request, and who is paged? | Self-hosting everything at small scale | Managed-service premiums and lock-in |
Decide: scope and toolset
Start from requirements and numbers
The rule: design for the load you have plus the growth you can see, roughly 10×, and no further.
Why it matters. Jeff Dean’s 2009 LADIS keynote put it in two lines: make sure the design works if scale changes by 10× or 20×, “but the right solution for X often not optimal for 100X”. The same deck warns against imagining “unlikely potential needs that aren’t really there”. Numbers also pick components. Three million bookings a day is 3,000,000 ÷ 86,400 ≈ 35 writes/s, perhaps 350/s at a 10× peak: well inside the roughly 10,000 small writes a second that numbers to know plans for one relational primary.
In the round. Turn adjectives into numbers (users, peak factor, read:write ratio, growth, latency), then say which number picks which box (back-of-envelope estimation).
It prevents “Netflix-scale on day one”: five microservices and a global database for 10,000 users.
What it costs, and when to break it. You will rebuild parts near 10×, with revenue and real traffic data by then. Break the rule for choices that are expensive to change later, such as an ID format or partition key: give those 100×.
Say it like this: “At 350 writes a second at peak, one primary with a standby is plenty. I’ll pick the shard key now so splitting is cheap if we pass 10×.”
Prefer simple, boring technology
The rule: use technology whose failure modes are well known, and spend novelty only where it is the product.
Why it matters. Google’s SRE book calls “boring” “actually a positive attribute” in software, and says that for a 24/7 service, “to some extent, every new line of code written is a liability”. Dan McKinley’s 2015 essay “Choose Boring Technology” gives each company about three “innovation tokens”; a new database or language spends one. His argument is about failure: the failure modes of Postgres or cron are well understood, while a young tool’s unknown unknowns are far larger.
In the round. Default to one relational database, a cache, a queue and object storage, and justify each extra store with a requirement they can’t meet. McKinley’s test works well out loud: how would we solve this without adding anything new?
It prevents résumé-driven design: four databases for four access patterns, each needing its own backups, upgrades and on-call skills.
What it costs, and when to break it. Boring tools handle specialised jobs (full-text search, vector similarity, huge time series) clumsily. Spend a token when a requirement truly needs one, and say what it replaces.
Say it like this: “Bookings stay in Postgres. Search is the one place I’d add a specialised engine, because fuzzy matching and ranking are core to the product.”
Saying no is a design skill
Dean’s deck notes that when clients ask for 8 things, doing 6 is easy, 7 “requires real thought” and all 8 “usually results in a worse system”. Naming what you won’t build is a senior signal.
Build: state, consistency, retries and time
Keep compute stateless and make state explicit
The rule: servers keep nothing a later request depends on, and every piece of state lives in a store chosen for its job.
Why it matters. The Twelve-Factor App is blunt: processes are “stateless and share-nothing”, persistent data lives in a backing service, and sticky sessions “should never be used or relied upon”. Then any request can go to any server, autoscaling is arithmetic, and a crash or deploy loses only requests in flight (scalability).
In the round. Say where each kind of state lives: sessions in signed tokens or Redis, records in the database, files in object storage, work in progress in a queue, WebSockets on a thin gateway tier.
It prevents sticky sessions, carts in server memory and uploads on local disk.
What it costs, and when to break it. Every read of moved state is a network hop. Dean’s 2009 table puts a round trip inside one data centre at 500,000 ns against 100 ns for main memory: about 5,000× slower. Keep small in-process caches where a miss is cheap, and accept stateful tiers when the state is the product (a game match, chat connections), kept small and drainable (stateful vs stateless design).
Say it like this: “App servers are stateless behind the load balancer. Sessions are signed tokens and carts live in Redis, so any server can die mid-sale.”
Choose consistency per operation
The rule: for each operation, ask what a stale or wrong answer costs compared with no answer.
Why it matters. Every replica you wait for adds a round trip. Light in fibre covers about 200,000 km a second, so a quorum spanning Mumbai and Singapore, about 3,900 km apart, costs at least 2 × 3,900 ÷ 200,000 ≈ 39 ms per write. Eventual consistency is fast but can sell one seat twice. Neither suits a whole system (CAP and PACELC).
In the round. Sort operations into two lists. Strong: moving money, scarce inventory, uniqueness, locks. Eventual: counts, feeds, search, recommendations, analytics. Then say how each strong one is enforced: a conditional update on one leader, a unique constraint or a consensus store.
It prevents both “everything strongly consistent” (slow and fragile across regions) and an eventually consistent wallet balance.
What it costs, and when to break it. A cached count can disagree with the booking result, so re-check at the strong step and design the message (“those seats just went”). The line moves with the stakes (strong vs eventual consistency).
Say it like this: “Availability counts come from cache and are eventually consistent. The seat hold is a conditional update on the primary, so one berth can’t be sold twice.”
Make every operation safe to retry
The rule: assume any request may arrive twice or not at all, and make repeating it harmless.
Why it matters. A timeout doesn’t tell you whether the other side acted, and clients, proxies and queues all retry. Amazon’s Builders’ Library describes its preferred fix: a “unique caller-provided client request identifier” in the API contract, recorded in the same atomic operation as the change, so the token and the resource can never disagree.
In the round. Put an idempotency key on every write that moves money or creates something, store it under a unique constraint beside the result, and deduplicate consumers by message ID. Pair it with timeouts and capped, jittered retries (idempotency, retries and backoff).
It prevents double charges after a timeout, two orders from a double tap and duplicate SMS from a redelivered message.
What it costs, and when to break it. A dedupe table, a retention window that outlasts every retry, and rejecting the same key with a different body. Reads and naturally idempotent writes (“set status to SHIPPED”) need none of it.
Say it like this: “The pay call carries an idempotency key stored in the same transaction as the payment row, so a retry returns the first result instead of charging twice.”
Decouple in time where the user doesn’t need to wait
The rule: do synchronously only what the user must see before moving on, and queue the rest.
Why it matters. A synchronous chain adds its latencies and multiplies its availabilities: five dependencies at 99.9% give 0.999⁵ ≈ 99.5%, five times the failure rate of any one. A queue cuts both links. The producer finishes in milliseconds, consumers drain at their own pace, and a spike like the 10 am tatkal rush becomes a backlog, not an outage.
In the round. Ask of each step: does the user need this result on this screen? A booking needs the seat; the ticket PDF, SMS, e-mail and loyalty points don’t. Publish one event through a transactional outbox and let workers fan out.
It prevents checkouts that fail because the e-mail provider is slow.
What it costs, and when to break it. You need dead-letter queues, alarms on queue age, ordering rules and a status users can poll, and one request becomes a trail across several logs. Stay synchronous when the user waits for the answer anyway, as with search or a balance check (synchronous vs asynchronous).
Say it like this: “The user waits only for the booking write. Ticket, SMS and e-mail hang off a booking-confirmed event, so an SMS outage delays texts, not bookings.”
Run: survive and see
Design for failure
The rule: assume every component fails, let no single one take down a critical path, and keep failures small.
Why it matters. Dean’s 2009 deck made the arithmetic vivid: build 10,000 servers that each fail once in 30 years and you will “watch one fail per day” (10,000 ÷ (30 × 365) ≈ 0.9 a day). A new cluster’s typical first year brought about 1,000 machine failures, 20 rack failures and thousands of disk failures. At scale, failure is the weather.
In the round. Ask what happens when each box on the critical path dies. Answer with N+1 across zones (single point of failure), failover for stores, timeouts and circuit breakers between services, and degraded modes. Dean’s advice: limited functionality beats an error page, so dead recommendations hide a carousel instead of breaking the home page.
It prevents the database with no standby, and the retry storm that turns one slow dependency into an outage (reliability patterns).
What it costs, and when to break it. Redundancy multiplies spend and adds failure modes of its own: misfiring failovers, split brain. Price it per component: an admin tool can be one instance, a payment path cannot, and multi-region is a business decision (fault tolerance).
Say it like this: “Every tier is N+1 across three zones. If recommendations are down, the home page renders without them; if the payment provider is down, orders wait as ‘pending’.”
Principles without numbers become cargo cults
“Design for failure” doesn’t mean five regions for a college-fest app, nor “decouple in time” a queue between every pair of services. If you can’t say what a principle protects and costs here, you are reciting it, not applying it.
Measure what users feel
The rule: define success in the user’s terms, with percentiles and SLOs, and build the instruments in from the first diagram.
Why it matters. Averages hide suffering users. Google’s SRE book describes a service averaging 100 ms at 1,000 requests a second where 1% of requests “might easily take 5 seconds”, and notes that one backend’s 99th percentile can become the median of a page that calls several of them. With a fan-out of 100, the chance a page hits at least one p99-slow call is 1 − 0.99¹⁰⁰ ≈ 63% (latency vs throughput).
In the round. State SLOs as numbers (“99% of seat holds under 500 ms”), name the four golden signals (latency, traffic, errors, saturation), trace across services and alert on error-budget burn, not CPU.
It prevents green dashboards while users stare at spinners.
What it costs, and when to break it. Instruments cost money and people: the SRE book notes that a Google SRE team of 10–12 typically has one or two members mainly building monitoring. Sample traces and high-cardinality metrics (observability); a nightly report needs no tight SLO.
Say it like this: “The SLO is 99% of holds under 500 ms in the booking window, with burn-rate alerts and sampled end-to-end traces.”
Evolve: change and cost over the years
Make change safe and reversible
The rule: make most changes small and undoable, and save the slow, careful process for the few you can’t undo.
Why it matters. Many outages start with a change. In his 2015 shareholder letter, Jeff Bezos split decisions into Type 1, “consequential and irreversible or nearly irreversible – one-way doors”, to be made “methodically, carefully, slowly”, and Type 2, two-way doors that “can and should be made quickly”. AWS’s operational excellence pillar turns this into a design principle: “make frequent, small, reversible changes”.
In the round. Name your one-way doors (public API, primary data model, partition key, data residency) and slow down there. Everywhere else, describe the escape hatch: canaries with automatic rollback, feature flags, and schema changes as expand-and-contract steps (also called parallel change), each safe to release alone (online migrations).
It prevents big-bang migrations, global config pushes and deploys that can’t roll back because the schema already moved.
What it costs, and when to break it. Two versions coexist for a while and flags pile up as debt; Fowler’s write-up warns that never finishing the contract phase can leave you worse off than you started. The SRE book cites Knight Capital as the cautionary tale for dead code behind an always-off flag. And a timeout value doesn’t need a design review.
Say it like this: “The partition key and public API are one-way doors, so I’ve spent time there. Everything else ships behind flags with a canary, and the column rename goes expand, backfill, switch, contract.”
Optimise for total cost and operability
The rule: judge a design by what it costs to run, change and staff for years, not just to build.
Why it matters. AWS’s cost optimisation pillar says to “measure overall efficiency” (business output against the cost of delivering it) and to “stop spending money on undifferentiated heavy lifting”. Its example: switching off dev and test environments outside 40 working hours of a 168-hour week saves a potential 75% (1 − 40 ÷ 168 ≈ 76%). People are often the biggest line: every component needs someone who understands it at 3 am.
In the round. Quote cost per request next to p99, say what you’d buy rather than build, note where data gravity and egress shape the design, and say who gets paged.
It prevents cheap-to-build, expensive-to-run designs: a self-hosted streaming cluster for 50 messages a second, or peak-sized fleets all day.
What it costs, and when to break it. Managed services cost more per unit and bring lock-in. Build when the component differentiates you, when scale makes the managed premium dominate the bill, or when latency or residency rules managed options out (cost-aware design).
Say it like this: “At 50 messages a second, a managed queue is a rounding error; our own cluster means an on-call rotation. I’d revisit if it became a top-three cost.”
Tensions between principles
Principles pull against each other. A senior answer picks a side for this context, says what it gives up, and names the trigger that would flip the choice.
| Tension | Why they pull apart | Usual resolution | Revisit when |
|---|---|---|---|
| Simplicity vs design for failure | Every replica, failover and fallback is more code and a new way to fail | Redundant stateless tiers; managed failover for stores; no home-grown consensus | An hour of downtime costs more than the next nine (the nines) |
| Stateless vs latency | Moving state off the box adds a network hop to every read | Stateless by default; small in-process caches where a miss is cheap | State fetches dominate the p99 budget, or the state is the product (games, chat) |
| Consistency vs availability and latency | Strong reads wait for replicas and fail on a partition’s minority side | Strong only for money, scarcity and uniqueness; eventual elsewhere | Complaints about stale data, or a move to multi-region |
| Async vs debuggability | Queues hide failures in backlogs and spread one request over many logs | Async only off the user’s path; trace IDs on messages; alarms on queue age | Queue age breaks its SLO, or diagnosis gets slower |
| Reversibility vs speed | Flags, canaries and expand-and-contract slow every change | Match ceremony to the door: fast for two-way, deliberate for one-way | A “small” change causes an incident, or rollouts block urgent fixes |
In the interview
The trigger is what makes a trade-off sound senior. “I’m keeping one region. If the business sets a recovery time under an hour, or a regulator requires a second region, I’d add a warm standby” beats both “we should be multi-region” and silence.
Worked example: a tatkal-style booking rush
Prompt: design train booking for a tatkal-style window opening at 10:00. Assume 1.5 million people try to book in the first 10 minutes, each with about 4 searches and 1 booking attempt, for about 60,000 berths.
- Searches: 1.5M × 4 = 6M in 600 s = 10,000/s on average; if the first minute runs at 3×, 30,000/s.
- Booking attempts: 1.5M in 600 s = 2,500/s, about 7,500/s in the first minute.
- Contention: 1.5M ÷ 60,000 = 25 people per berth, a scarcity problem first (contention and double booking).
| # | Principle | Decision for this prompt | The number |
|---|---|---|---|
| 1 | Numbers first | Size for the first-minute peak | 30,000 searches/s; 7,500 bookings/s |
| 2 | Boring tech | Postgres, Redis for cached counts, a managed queue, object storage | Zero innovation tokens |
| 3 | Explicit state | Stateless API fleet; signed-token sessions; seats as rows per train, date and coach | 30,000 ÷ 2,000 per server (assumed) ÷ 0.6 utilisation ≈ 25 servers |
| 4 | Consistency per op | Counts eventual from cache; the seat hold is a conditional update on the primary | Counts ≤ 5 s stale; holds never oversell |
| 5 | Safe retries | Idempotency key per attempt; payment callbacks deduplicated by payment ID | One booking per key, however many retries |
| 6 | Decouple in time | User waits for the hold and booking write; ticket, SMS and e-mail are queued | 1 synchronous dependency instead of 5 |
| 7 | Design for failure | Three zones; a virtual waiting room before booking; holds expire after 10 minutes | Lose a zone and 2 must carry 25: 13 per zone, 39 in total |
| 8 | Measure users | SLOs on hold latency and booking success, with burn-rate alerts | 99% of holds under 500 ms |
| 9 | Safe change | Change freeze around the window; flags; migrations the night before | Zero deploys from 09:30 to 11:30 |
| 10 | Total cost | Pre-scale at 09:45, scale down at 11:00, not peak capacity all day | 1.25 h of 24 ≈ 5%: about 95% saved on the extra fleet |
The waiting room serves three principles at once: admitting about 2,500 attempts a second protects the store, everyone else waits somewhere cheap, and a visible queue position beats a spinner.
How principles show up by level
How a system design interview works has the full level table. For principles, the difference is when they appear and whether they come with a price.
| Mid-level | Senior | Staff and above | |
|---|---|---|---|
| When principles appear | When asked (“what if this fails?”) | Unprompted, as each box is drawn | Before drawing, to decide which few matter most for this problem |
| How they are stated | As good practice: “add retries” | With the number and cost: “three jittered retries, so the pay call needs an idempotency key” | With the revisit trigger and who pays: budget, on-call, teams |
| Breaking one | Rarely | When a number justifies it, out loud | Picks which principle loses, and sequences the change over quarters |
Cheat sheet
| Principle | Remember it as | Number to quote |
|---|---|---|
| Numbers first | Today plus ~10×; expect a new design near 100× | Works at 10–20×, rethink at 100× (Dean) |
| Boring tech | Known failure modes beat new features | About three innovation tokens per company |
| Explicit state | Stateless servers; state in purpose-built stores | Data-centre round trip 0.5 ms vs RAM 100 ns |
| Consistency per op | Strong for money, scarcity and uniqueness | ~1 ms of round trip per 100 km of fibre |
| Safe retries | Caller’s key, stored atomically with the write | One effect per key, any number of tries |
| Decouple in time | Sync only what the user must see | 0.999⁵ ≈ 99.5% for a five-call chain |
| Design for failure | No SPOF on critical paths; break small | 10,000 servers at a 30-year MTBF ≈ one failure a day |
| Measure users | SLOs on percentiles; page on symptoms | Fan-out of 100 → 63% of pages hit a p99 |
| Safe change | Small and reversible; slow down for one-way doors | Expand → migrate → contract |
| Total cost | Cost per request; buy the undifferentiated | 40 of 168 hours → ~75% saved |
How to say it in the interview
“First the numbers: I’ll design for today’s peak plus about 10×, with boring parts: Postgres, Redis, a managed queue. App servers are stateless. Seat holds are strongly consistent; browsing is eventual. Every write carries an idempotency key, and whatever the user needn’t wait for is queued. Each box can fail, so N+1 across zones, with SLOs on p99. The shard key and public API are one-way doors; everything else ships behind flags. Where these conflict, I’ll say what I’m giving up and what would change my mind.”
When would you pick a non-boring technology? When a requirement is central to the product and the boring stack demonstrably can’t meet it, such as vector search for a retrieval product. Spend one token and say what it replaces.
Isn’t designing for 10× premature? No. If traffic doubles yearly, 10× is about three and a third years away (2³·³ ≈ 10), roughly a design’s lifetime. 100× is premature: the requirements will have changed by then.
How do you decide strong vs eventual consistency? Per operation: compare the cost of a wrong answer with the cost of no answer. If staleness can create or destroy money, or sell something scarce twice, it is strong; otherwise eventual, re-checked at the strong step.
Which principle would you relax first under pressure? Ceremony, never correctness. At small scale I’d drop async fan-out and heavy rollout machinery long before idempotency on money or a database standby: those failures are permanent, the others are only slow.
Aren’t these just generic best practices? The rules are; the judgement isn’t. The senior part is saying what each costs here, when you’d break it, and what would make you revisit it.
Where to go next
- Back-of-envelope estimation: turn requirements into servers, storage and shards.
- Scalability: stateless tiers and what breaks first as you grow.
- Single point of failure: find SPOFs and shrink the blast radius.
- CAP theorem and PACELC: the mechanics behind choosing consistency per operation.
- Idempotency: the mechanics of safe retries.
- Reliability: SLIs, SLOs and error budgets: targets and alerts for what users feel.
- Deployment strategies: canaries, blue-green and flags for reversible change.
- Cloud design patterns cheat sheet: the named patterns that put these principles into practice.
References and further reading
- Designs, Lessons and Advice from Building Large Distributed Systems (Jeff Dean, LADIS 2009 keynote)cs.cornell.eduDesign for 10× to 20× growth but expect a different solution at 100×; a 10,000-server fleet sees about one failure a day; limited functionality beats an error page.
- Simplicity (Google SRE book, chapter 9)sre.google“Boring” as a virtue, essential versus accidental complexity, every new line of code as a liability, and small release batches.
- Choose Boring Technology (Dan McKinley, 2015)mcfunley.comInnovation tokens, known versus unknown failure modes, and why running a system costs more than building it.
- The Twelve-Factor App: VI. Processes12factor.netProcesses are stateless and share-nothing; persistent data lives in backing services; sticky sessions are ruled out.
- Monitoring Distributed Systems (Google SRE book, chapter 6)sre.googleThe four golden signals, symptoms versus causes, why averages hide the tail, and the people cost of monitoring.
- Making retries safe with idempotent APIs (Amazon Builders’ Library)aws.amazon.comCaller-provided request identifiers, recorded atomically with the change, so retries return the original result.
- Amazon 2015 Letter to Shareholders (Jeff Bezos)s2.q4cdn.comType 1 (one-way door) and Type 2 (two-way door) decisions, and the cost of treating every decision as Type 1.
- Parallel Change, also called expand and contract (martinfowler.com bliki)martinfowler.comSplitting a breaking change into expand, migrate and contract phases, and the risk of never contracting.
- AWS Well-Architected Framework: operational excellence design principlesdocs.aws.amazon.comMake frequent, small, reversible changes; anticipate failure; implement observability; use managed services.
- AWS Well-Architected Framework: cost optimization design principlesdocs.aws.amazon.comMeasure overall efficiency, adopt a consumption model (the 40-of-168-hours example) and stop paying for undifferentiated heavy lifting.