Signals that you’re scaling writes
- “Ingest N events a second” from a fleet
- Append-mostly data, rarely read row by row
- Bursts tied to a moment: a midnight sale
- Counters on popular things: likes and views
Climb the ladder. Stop at the first rung that fits.
Cheaper writes, batches, a log for bursts, a key that spreads, LSM storage, hot-key tricks, writing less. Each costs latency, durability or read cost.
125,000 writes a second. Add read replicas?
Bad: More replicas
Every replica replays every write. They add no write capacity.
Good: Batch, log, partition
Cheaper writes, bigger transactions, a log for bursts, then shards.
10×
throughput from 100-row batches
Every transaction pays a fixed cost. Assume 50 µs a commit plus 5 µs a row, and one node goes from 18,000 rows a second to about 182,000.
A burst becomes backlog, not an outage
A city reconnects, 275,000 pings a second for 20 s. Kafka holds a 2.3-million backlog, and writers drain it with up to 14 s of lag.
You shard pings by time bucket. Eight shards, and seven sit idle.
The trick
Hash a high-cardinality key like partner ID. Each shard gets an even share, and each partner’s pings stay in order.
One hot row is a queue of lock waits
8,000 partners in one zone send 2,000 pings a second, but one locked row manages about 200 updates. Sum in memory and flush each second instead.
Say it like this
Make each write cheap, batch, put a log in front for bursts, partition by a key that spreads, then defuse hot keys and write less. Name what each rung can lose.