20×
the cache misses, from 99.9% to 98% hits
Telemetry’s 150 GB index competes with checkout’s 30 GB hot set for about 50 GB of memory. Order pages get evicted, so checkout p99 follows telemetry.
One database is simpler, right?
Bad: Everything in Postgres
Telemetry grows with users × features × screens; orders grow only with orders.
Good: Split by access pattern
Orders keep the relational store; events, files and sessions go where they fit.
Unrelated data fights over everything the box has one of
CPU, cache, WAL, vacuum, connections and backup. Every PDF byte flows through the WAL to each replica and backup.
Detect it: attribute every resource to a workload
Sum writes, bytes and WAL per kind of data. Just 72,000 invoice inserts write a tenth of all WAL; the order status update comes fourth.
Give each kind of data a store that fits
- Events and audit → Kafka, then a lake
- PDFs → object storage, key in the row
- Sessions → Redis with a TTL
- What stays: time partitions, retention by DROP
Orders in Postgres, events in Kafka. Write both from the request?
The catch
No transaction spans two stores. Commit an outbox row with the order, let a relay publish it, and make consumers idempotent.
Restoring checkout: 3.1 hours, or 38 minutes
2 TB at 200 MB/s copies in 2.8 hours before WAL replay starts. A 400 GB orders database copies in 33 minutes and fits a one-hour recovery objective.
Say it like this
Group data by access pattern and value, not convenience. Orders keep the relational database; events, files, sessions and search go to stores built for them, joined by an outbox and CDC.