2.3M/s
internal calls at a sale peak
230,000 product pages a second, ten times the daily average, each fanning out to about 10 internal calls and waiting for its slowest.
One call in 100 is slow. How many pages are?
The page waits for its slowest call. At 20 calls, 18% of pages wait on a slow one. At 150, 78%.
A call is late. Does the page wait?
Bad: Wait for every call
Slow recommendations set the page’s time.
Good: Deadlines and cut-offs
Cut optional rows at 100 ms, critical at 150. Never drop or invent the price.
Around 2001, Amazon’s caches failed together. Then the database.
Every web server fell back to the database. It locked up; the website and fulfilment centres stopped. Never fall back to the source.
3.6M badge lookups a second. Ask inventory, and you repeat 2001.
Push, don’t pull
Inventory pushes changes into an in-memory stock view; pages read only that. Inventory down? Badges freeze; pages stay up.
430
stuck orders an hour from 0.01% failed writes
At 1,200 orders a second, write then publish: crash in between and the order never ships. Commit it with an outbox row instead.
Midnight: 10× traffic in one step
- Pre-scale to 15×; losing a zone leaves 10×
- Warm caches, freeze deploys that week
- Shed crawlers and recommendations first, never checkout
- Cells: a bad deploy hurts a quarter, not all
Remember
The page is a read model: pushed by events, served under deadlines, seconds stale. Checkout is the truth; orders leave through an outbox.