16
shards, set by storage, not writes
Writes alone need 5 primaries. But 400 GB of messages a day is 146 TB a year, and Notion capped each database at 10 TB.
Every request asks one question. Which shard owns this key?
A router hashes the key to a logical shard, checks a cached, versioned shard map, and sends the request to exactly one shard.
Shard by created_at?
Bad: Unique values, so yes
Every new row is the newest yet, so all inserts hit one shard.
Good: Spread matters, not count
Hash monotonic keys, or pick one most queries and transactions share.
Add a fifth shard to hash mod 4. What moves?
16 of 20 keys change shard. In general N/(N+1) of keys move, so you copy almost everything to add one machine.
Add one machine without copying almost everything?
The trick
Hash keys into a fixed set of logical shards, far more than machines. A new host takes a few whole ones; no key changes logical shard.
63%
of 100-shard queries wait on a slow shard
Each shard is slow on just 1% of requests. Scatter-gather waits for the slowest, so a 100-way fan-out’s p99 is each shard’s p99.99.
A celebrity posts. One shard melts. Now what?
- Hot reads: cache the item and add replicas
- Hot writes: salt the key across k suffixes
- Hot tenants: give the whale its own shard
Remember
Shard when writes or data outgrow one primary. Pick a key that keeps queries and transactions on one shard; hash into logical shards so growth moves whole ones.