20
shards, set by bandwidth, not memory or ops
A 300 GB working set needs 13 and 1.2M reads/s needs 12, but 10 KB menus push 96 Gbit/s: 20. Compress to 3 KB and ops bind.
Three ways to find the node that owns a key.
The client hashes the key itself, a proxy routes it, or the cluster redirects you. Each puts the key-to-node map somewhere else.
In Redis Cluster every key maps to one of 16,384 slots. Ask the wrong node and it replies MOVED.
Two keys, one slot
Hash tags. {user42}:cart and {user42}:orders hash only user42, so they share a slot.
Adding a 21st node?
Bad: hash(key) mod N
95% of keys move. The database sees about 19× its normal reads.
Good: Consistent hashing
About 4.8% move, so the database sees 1.9×, falling as the node fills.
A replica can lose a write that was already acknowledged.
Redis replies OK before any replica has the write. A lost SET is only a miss; a lost DEL stays stale until its TTL.
117k/s
database reads when one of 20 nodes dies
Normal is 60,000, and the database handles 100,000. Facebook keeps about 1% of servers idle as a gutter pool, which cut client-visible failures by 99%.
Invalidating caches across regions
- Writes embed the keys to delete in their SQL
- Each region deletes after its replica applies
- Writers also delete their own cluster’s key
- Other regions’ writes leave a remote marker
Remember
Distribute when one node runs out of memory, ops, bandwidth or availability. Partition with consistent hashing or slots, add async replicas, keep TTLs for lost deletes, and plan for dead nodes.