Every principle is a question you ask of every box.
Ten questions across a system’s life: decide, build, run, evolve. A senior answer adds the number, the cost and when you’d break it.
25
people chasing every berth
Numbers first. The first minute may bring 30,000 searches a second, but 1.5 million people for 60,000 berths make this a scarcity problem first.
One consistency model for everything?
Bad: All strong, or all eventual
Slow across regions, or one seat sold twice.
Good: Choose per operation
Counts: cached, ≤ 5 s stale. Seat hold: a conditional update on the primary.
The pay call times out. Did it charge? The timeout can’t say, and clients retry.
The trick
Send an idempotency key, stored in the same transaction as the payment. A retry gets the first result.
Chain five calls at 99.9% and the booking gets 99.5%.
That’s five times the failure rate of any one. Wait only for the booking write; queue the ticket, SMS and e-mail.
Assume every part fails. Keep each failure small.
- N+1 across three zones for every tier
- A waiting room admits ≈ 2,500 attempts a second
- Payment provider down? Orders wait as “pending”
Five regions for a college-fest app. A queue between every pair of services.
The catch
Without a number, it’s a cargo cult. Say what it protects and costs here, and the trigger to revisit it.
Remember
Design from numbers, plus about 10×. Pick consistency per operation, make writes safe to retry, queue what users needn’t wait for, and expect failure. Name each trade-off and the trigger to revisit it.