Monolithic persistence means keeping every kind of data your application produces in one data store, usually one relational database, whatever its access pattern: orders next to clickstream, audit logs, sessions, invoice PDFs, a jobs table, search text and analytics rollups. Each workload wants something different, yet they all share one CPU, one buffer cache, one write-ahead log (WAL), one connection limit and one backup. So the data that earns the money ends up slowed, bloated and put at risk by data that doesn’t.
Picture it first: the cloud kitchen’s cold room
A busy cloud kitchen has one walk-in cold room, built for the paneer, dough and onions that cooks grab every minute of the dinner rush.
Over a year, other things move in, each for a sensible reason. Supplier invoices go there because it’s the only room with a lock. The CCTV recorder sits in a corner because there’s a socket. Catering trays for next month’s wedding order are stacked by the door. Staff lunch boxes go on the top shelf.
Now it’s 8:30 pm. Cooks queue at the door while someone hunts for a February invoice. The shelves nearest the door are full of trays, so paneer goes to the back. The door opens so often that the room warms up. The weekly deep clean takes all night because everything has to be moved. And when the compressor fails, the kitchen loses its ingredients and its records in one go.
Map it back:
- The cold room is your primary database, and the door is its connections, locks and IOPS.
- Shelves near the door are the buffer cache: whatever sits there pushes hot rows to slow storage.
- Ingredients are business data; invoices and CCTV are audit logs and telemetry; trays are images and PDFs; lunch boxes are sessions.
- The deep clean is vacuum, backups and restores; the compressor is a single point of failure.
The fix isn’t a bigger cold room. Invoices go to a filing cabinet, the CCTV gets its own recorder, and lunch boxes go to a staff fridge.
What gets mixed, and what it fights over
Take a Swiggy-style food-delivery app with one PostgreSQL primary (16 vCPUs, 64 GB of RAM) doing 250,000 orders a day. At the dinner peak it handles about 20 orders a second. Each order writes around 15 rows over its life (the order, its items, a payment, five status changes and so on), so the business workload is 300 row writes a second. Alongside it, the same database takes:
- Telemetry: client taps, scrolls and rider GPS pings, 6,000 rows a second at peak and 1,500 on average.
- An audit log: one row per business write, 300 a second, kept for two years.
- Sessions and a jobs table: 900 session updates and 500 queue writes a second.
- Invoice PDFs and menu photos in
byteacolumns: only about 20 rows a second, but 15 KB or more each.
Add it up: 300 + 300 + 6,000 + 900 + 500 = 8,000 row writes a second, of which the business is 300 ÷ 8,000 = 3.75%.
None of these workloads is a problem alone. Together they compete for everything the database has only one of:
- CPU and IOPS. Every insert parses, plans, locks, writes heap and index pages and logs the change. 6,000 tiny inserts a second cost real CPU even when each is cheap.
- The buffer cache. Say 50 GB of the 64 GB box ends up caching pages, in
shared_buffersplus the OS page cache. Checkout’s hot set (recent orders, live menus, active users and their indexes) is about 30 GB, so alone it fits. But 30 days of events at 1,500 a second is 30 × 1,500 × 86,400 ≈ 3.9 billion rows, and their index on(user_id, at)is roughly 3.9 billion × 40 bytes (entry plus partly empty pages) ≈ 150 GB. Each insert lands on a leaf picked byuser_id, effectively at random, so 30 GB + 150 GB of pages fight over 50 GB of memory, and order pages keep getting evicted. - The WAL. Every change, including every byte of every PDF, goes through one log, which is shipped to every replica and archived for point-in-time recovery. A PostgreSQL standby replays it largely in a single process, so non-business WAL becomes replica lag for everyone. See write-ahead logging.
- Vacuum and transaction IDs. Session updates and queue deletes leave dead rows that autovacuum must clean, and a nightly
DELETEof 130 million expired events leaves a 600 GB table to vacuum. If the event endpoint commits each event on its own, 1,500 a second burns 130 million transaction IDs a day, so every table in the database, orders included, ages towards the defaultautovacuum_freeze_max_ageof 200 million in about 1.5 days and needs a freezing vacuum that often, forced as an anti-wraparound vacuum if routine ones haven’t done it. See vacuum and bloat. - Connections. One
max_connections, shared by checkout, the event ingester, session middleware, job workers and the BI tool. - Backups and the failure domain. One backup to take, one restore to wait for, and one machine whose failure takes down everything from checkout to the logs you would use to debug it.
Azure’s guidance names two harms: contention between unrelated data, and a store that isn’t a good fit for some of it. The list above is the contention. The poor fit shows up as a 600 GB table you can only shrink with DELETE, PDFs that inflate every backup, and search done with ILIKE '%paneer%'.
Symptoms you’d see
- Business latency moves with non-business volume. Checkout p99 rises when an app release adds events, when the nightly retention
DELETEruns, or when a report scans the event table, while orders per second stay flat. - The cache hit ratio for hot tables sags. The orders table drops from 99.9% to 98% hits. That looks small, but misses go from 1 in 1,000 block reads to 20 in 1,000: twenty times the reads from outside the cache.
- Replica lag and WAL volume track the wrong thing. Lag spikes at the dinner peak because of GPS pings, or every afternoon when menu photos are uploaded.
- Storage grows from data nobody pays for. Most of the 2 TB is events, logs and files, and every byte is paid for again on each replica and in each backup.
- Backups and restores take hours. Restoring 2 TB at 200 MB/s takes 2,000,000 MB ÷ 200 MB/s = 10,000 s ≈ 2.8 hours before WAL replay even starts, even if checkout needs only 400 GB of it.
- Vacuum never finishes. Autovacuum is always busy on
django_session, the jobs table or a 600 GB event table, and a long report on the primary holds back cleanup everywhere, because PostgreSQL can’t remove dead rows that an open snapshot might still see. - Connections run out. Azure warns that such a system can slow dramatically and then fail when it runs out of resources such as database connections. When the event ingester holds 180 of 400, checkout waits.
- One outage takes out unrelated features. If the database fails, so do checkout, login, invoice downloads, the admin search box and the job queue, all at once.
Don’t keep the outage logs in the database that’s down
If application logs and audit rows live in the primary, the incident that kills the primary also hides the evidence. Telemetry should fail independently of the thing it observes.
Why it happens
Nobody designs a monolithic persistence layer. It accretes, one reasonable table at a time.
- One database is the right way to start. On day one the event table is 10 MB, there is one team, and one backup is simpler than four. The trouble is that telemetry grows with users × features × screens, while orders grow only with orders.
- “We already have Postgres.” The team knows it, it is approved and on-call already covers it. AWS’s Well-Architected guidance lists exactly this as an anti-pattern: sticking to one store because of internal experience, and assuming all workloads have similar storage and access needs.
- Frameworks default to the database. Django stores sessions in the database unless told otherwise, and never purges expired ones unless you run
clearsessions. Job libraries and audit plugins often ship as “add these tables”, and an ORM makes every new table one migration away. - Queues and caches built as tables. A
jobstable polled withSELECT … FOR UPDATE SKIP LOCKEDworks well at 50 jobs a second. At 500, it is a hot table of inserts, updates and deletes that never stops needing vacuum. - Logging the easy way.
INSERT INTO audit_loginside the request is one line, transactional and queryable with SQL. Azure’s example is exactly this: a purchase-order insert followed by a log insert into the same database. - Ownership. One team owns one database. Adding Kafka or an object store means a design review, new dashboards and a new pager rotation, so the next table goes where the last one went.
The root cause is that different data has different needs. Azure notes that log records are inherently sequential, while business data is more likely to need random access and is often relational. Its data-model guide adds that a single data store rarely satisfies all access patterns efficiently.
How to detect it
Azure’s method: instrument where the app reads and writes, find the slow periods, see which stores were busy then, and look for contended resources. The key move is to attribute every resource to a workload, not to a query.
Break the database down by workload
PostgreSQL keeps per-table counters in pg_stat_user_tables (n_tup_ins, n_tup_upd, n_tup_del) and per-table block I/O in pg_statio_user_tables (heap_blks_read and heap_blks_hit, plus index and TOAST columns). pg_total_relation_size adds indexes and TOAST to a table’s size. Map tables to workloads and sum:
WITH t AS (
SELECT CASE
WHEN s.relname IN ('orders', 'order_items', 'payments', 'users') THEN 'business'
WHEN s.relname IN ('app_events', 'rider_pings') THEN 'telemetry'
WHEN s.relname = 'audit_log' THEN 'audit'
WHEN s.relname IN ('django_session', 'jobs') THEN 'sessions+jobs'
WHEN s.relname IN ('invoices', 'menu_photos') THEN 'blobs'
ELSE 'other'
END AS workload,
s.n_tup_ins + s.n_tup_upd + s.n_tup_del AS rows_written,
pg_total_relation_size(s.relid) AS bytes,
io.heap_blks_read + coalesce(io.idx_blks_read, 0)
+ coalesce(io.toast_blks_read, 0) AS blks_read
FROM pg_stat_user_tables s
JOIN pg_statio_user_tables io USING (relid)
)
SELECT workload,
round(100.0 * sum(rows_written) / sum(sum(rows_written)) OVER (), 1) AS pct_writes,
pg_size_pretty(sum(bytes)) AS size,
round(100.0 * sum(blks_read) / sum(sum(blks_read)) OVER (), 1) AS pct_reads
FROM t
GROUP BY workload
ORDER BY sum(bytes) DESC;
workload | pct_writes | size | pct_reads
blobs | 0.2 | 702 GB | 11.2
telemetry | 74.9 | 598 GB | 54.7
business | 3.7 | 401 GB | 21.4
audit | 3.7 | 251 GB | 2.1
sessions+jobs | 17.5 | 48 GB | 10.6
(illustrative)
Telemetry does three-quarters of the writes and over half the block reads from outside the buffer cache, and many of the business reads are misses on order pages it evicted. PostgreSQL’s docs warn that these counters can’t tell a real disk read from one the OS page cache served, so pair them with OS disk metrics, and with pg_stat_io, whose evictions column counts buffers pushed out to make room.
Snapshot, then diff
The counters are cumulative since the last statistics reset, so a single read mixes last month with today. Save the query’s output at the start of the peak hour and again at the end, and compare the differences.
Find out who writes the WAL
pg_stat_statements records wal_records, wal_fpi and wal_bytes per normalised statement, so you can rank statements by the log they generate:
SELECT left(query, 44) AS statement,
calls,
pg_size_pretty(wal_bytes) AS wal,
round(100 * wal_bytes / sum(wal_bytes) OVER (), 1) AS pct_wal
FROM pg_stat_statements
ORDER BY wal_bytes DESC
LIMIT 5;
statement | calls | wal | pct_wal
INSERT INTO app_events (user_id, kind, paylo | 21,600,000 | 4635 MB | 45.0
UPDATE django_session SET session_data = $1, | 3,240,000 | 1339 MB | 13.0
INSERT INTO invoices (order_id, pdf, created | 72,000 | 1030 MB | 10.0
UPDATE orders SET status = $1, updated_at = | 360,000 | 824 MB | 8.0
UPDATE jobs SET locked_by = $1, locked_at = | 600,000 | 721 MB | 7.0
(illustrative, one dinner-peak hour)
Just 72,000 invoice inserts write a tenth of all WAL, about 15 KB each, because the PDF bytes themselves go through the log. The order status update, which checkout depends on, comes fourth.
Count connections by application
Give every client its own application_name in the connection string, then group pg_stat_activity by it:
SELECT application_name,
count(*) AS connections,
count(*) FILTER (WHERE state = 'active') AS active,
count(*) FILTER (WHERE state = 'idle in transaction') AS idle_in_tx
FROM pg_stat_activity
WHERE backend_type = 'client backend'
GROUP BY application_name
ORDER BY connections DESC;
application_name | connections | active | idle_in_tx
event-ingest | 180 | 64 | 0
checkout-api | 90 | 41 | 12
session-mw | 60 | 9 | 0
job-worker | 40 | 22 | 3
bi-dashboards | 12 | 6 | 0
(illustrative; max_connections = 400)
382 of 400 connections are in use, and the event ingester holds twice as many as checkout. See connection pooling.
Correlate, then test the correlation
Plot business p99 against non-business rates (event inserts, retention jobs, report runs) on one time axis, as in Figure 2. Slowdowns at the same time every day usually point to a batch job. Then prove cause: turn the suspect down with a feature flag (sample events, pause the retention job) and watch checkout p99 and the orders hit ratio follow.
On SQL Server and Azure SQL the same breakdown comes from sys.dm_db_partition_stats (row and page counts per table), sys.dm_db_index_operational_stats (leaf_insert_count, page_io_latch_wait_count) and program_name in sys.dm_exec_sessions. Azure’s own diagnosis found DTU use at 100% and heavy inserts into both the purchase-order table and the log table.
Telling it apart from look-alikes
| What dominates writes, bytes and I/O | Business p99 tracks | First fix | |
|---|---|---|---|
| Monolithic persistence | Non-business tables: events, logs, files, sessions, queues | Telemetry, logging and batch volume, not orders | Move each kind of data to a store that fits; partition what stays |
| Busy database | A few statements doing app work on CPU, touching few blocks | Calls to those statements | Move computation out of the database |
| Noisy neighbour | Another tenant, database or VM on the same host, disk or pool | Their traffic; your own stats look normal | Isolate, throttle or move them |
| Chatty I/O | Huge call counts, one row each, from your own requests | Items per request | Batch the calls |
| Under-provisioned | Business tables, in proportion to business traffic | Orders per second | Scale up, tune, add replicas |
They overlap. Monolithic persistence is a noisy neighbour you invited into your own database, and a report scanning months of orders on the primary is both a busy database and a workload in the wrong store. Extraneous fetching also inflates I/O, so check rows and bytes per call before blaming the mix.
How to fix it
Azure’s rule is short: separate data according to its use, and pick a store per data set that matches how you use it. Fowler called the result polyglot persistence: decide how you need to manipulate the data first, then choose the technology. Start with the biggest, least valuable workload, which here is telemetry.
Take events and logs out of the request transaction
Before: the order, its audit row and every buffered client event commit together, so a slow 600 GB event table slows the order and a failed event insert fails it.
# BEFORE: business rows, an audit row and client events share one transaction
def place_order(req):
with db.transaction():
order_id = db.insert("orders", user_id=req.user_id,
total_paise=req.total_paise, status="PLACED")
db.insert_many("order_items", order_items(order_id, req.items))
db.insert("audit_log", entity="order", entity_id=order_id,
action="PLACED", actor_id=req.user_id)
for e in req.client_events: # taps and scrolls since the last call
db.insert("app_events", user_id=req.user_id, kind=e.kind,
payload=e.payload, at=e.at)
return {"order_id": order_id}
After: the transaction holds only what must commit together, plus one outbox row that a relay publishes to Kafka. Client events go straight to a buffered producer and never touch the database.
# AFTER: the order commits alone; audit and telemetry flow through Kafka
def place_order(req):
with db.transaction():
order_id = db.insert("orders", user_id=req.user_id,
total_paise=req.total_paise, status="PLACED")
db.insert_many("order_items", order_items(order_id, req.items))
db.insert("outbox", topic="order-events", key=str(order_id),
payload={"type": "OrderPlaced", "order_id": order_id,
"actor_id": req.user_id}) # relayed, then deleted
for e in req.client_events: # best effort; never blocks the order
events.produce("app-events", key=str(req.user_id), value=e.to_json())
return {"order_id": order_id}
An audit consumer writes order-events to an append-only archive in object storage, and the event stream lands in a data lake or warehouse for analysis. 75% of row writes and almost half the WAL leave the primary.
You pay with a Kafka cluster and two pipelines to run, events that can be lost if a pod dies before its producer flushes (fine for scroll events, not for payments), an audit trail that is now seconds behind, and consumers that must be idempotent. If a regulator needs the audit row in the same transaction as the order, keep it in the database, partitioned, as shown below.
Put files in object storage and keep a pointer
-- BEFORE: PDF bytes live in the row, the WAL, every replica and every backup
CREATE TABLE invoices (
order_id bigint PRIMARY KEY REFERENCES orders (id),
pdf bytea NOT NULL,
created_at timestamptz NOT NULL DEFAULT now()
);
-- AFTER: the row keeps a key and enough metadata to check the object
CREATE TABLE invoices (
order_id bigint PRIMARY KEY REFERENCES orders (id),
object_key text NOT NULL, -- 'invoices/2026/10/27/9876543.pdf'
size_bytes int NOT NULL,
sha256 text NOT NULL,
created_at timestamptz NOT NULL DEFAULT now()
);
import hashlib
from datetime import date
def store_invoice(order_id, pdf):
key = f"invoices/{date.today():%Y/%m/%d}/{order_id}.pdf"
s3.put_object(Bucket=BUCKET, Key=key, Body=pdf, ContentType="application/pdf")
db.execute("INSERT INTO invoices (order_id, object_key, size_bytes, sha256) "
"VALUES (%s, %s, %s, %s) ON CONFLICT (order_id) DO NOTHING",
(order_id, key, len(pdf), hashlib.sha256(pdf).hexdigest()))
def invoice_link(order_id):
key = db.fetch_value("SELECT object_key FROM invoices WHERE order_id = %s",
(order_id,))
return s3.generate_presigned_url("get_object", ExpiresIn=300,
Params={"Bucket": BUCKET, "Key": key})
The object is written first and the row second, so a crash leaves an orphan object that a nightly sweep deletes, never a row pointing at nothing. A presigned URL gives the app time-limited access to the object, so the bytes skip both your servers and your database. The classic “To BLOB or Not To BLOB” study found objects under 256 KB did better in a database and over 1 MB in a file system, but it measured fragmentation, not what every byte costs in WAL, replicas and backups. See blob storage.
You pay with no atomic commit across the two stores (hence the write order and the sweep), a second place to secure and back up, and lifecycle rules to write.
Give sessions a store with a TTL
# BEFORE (Django's default): sessions are rows in django_session,
# and expired rows stay until someone runs clearsessions
SESSION_ENGINE = "django.contrib.sessions.backends.db"
# AFTER: sessions live in Redis and expire on their own
CACHES = {
"default": {
"BACKEND": "django.core.cache.backends.redis.RedisCache",
"LOCATION": "redis://sessions.internal:6379/0",
},
}
SESSION_ENGINE = "django.contrib.sessions.backends.cache"
SESSION_COOKIE_AGE = 60 * 60 * 24 * 14 # two weeks, also the key's timeout
900 updates a second and a vacuum-hungry table leave the primary. You pay with a Redis to run, and users logged out if it loses data. Django’s cached_db backend keeps a database copy, but it writes to both, so it saves reads, not writes.
Queues, search and analytics: use their engines
- Jobs: a real message queue instead of a polled table. You gain no polling and no bloat; you pay with delivery semantics and dead-letter queues to handle.
- Search: a search index fed by change data capture, instead of
ILIKEscans on the primary. You pay with seconds of index lag and reindexing. - Analytics: a columnar warehouse fed by CDC and the event stream; see OLTP vs OLAP. You pay with minutes of lag and a second copy of the truth.
The smaller step: separate databases and partitions
You don’t have to adopt five technologies at once. Azure’s worked example changed one thing, writing log rows to a separate database: peak throughput rose by about 500 requests per second and the business database’s DTU use fell from 100% to about 75%. A separate instance of the same engine gives each workload its own cache, WAL, connections and backup.
For append-only tables that stay, partition by time and make retention a metadata change:
CREATE TABLE audit_log (
id bigserial,
at timestamptz NOT NULL,
entity text NOT NULL,
entity_id bigint NOT NULL,
action text NOT NULL,
actor_id bigint
) PARTITION BY RANGE (at);
CREATE TABLE audit_log_2026_10 PARTITION OF audit_log
FOR VALUES FROM ('2026-10-01') TO ('2026-11-01');
-- Retention: drop a month instead of deleting ~110 million rows
ALTER TABLE audit_log DETACH PARTITION audit_log_2024_10 CONCURRENTLY;
DROP TABLE audit_log_2024_10;
With one audit row per business write, a month is roughly 250,000 orders × 15 × 30 ≈ 110 million rows. PostgreSQL’s partitioning docs say dropping or detaching a partition is far faster than a bulk delete and avoids its vacuum overhead, and that old partitions can move to cheaper storage. Their rule of thumb: partitioning pays once a table would exceed the server’s physical memory. See table partitioning.
What each move costs
| Data | Move it to | You gain | You pay |
|---|---|---|---|
| Telemetry, clickstream | Kafka, then a lake, warehouse or time-series store | 75% of writes and 45% of WAL leave the primary | Pipelines to run; events can be late or lost |
| Audit log | Outbox, stream, append-only archive; or partitions | Retention by DROP; a smaller hot set |
Seconds of lag, or the WAL stays if it stays |
| Images, PDFs | Object storage, key in the row | Bytes leave WAL, replicas and backups; CDN-ready | No atomic commit; orphans to sweep |
| Sessions | Redis or another KV store with a TTL | Expiry for free; sub-millisecond reads | Logouts if it loses data; another cluster |
| Jobs | A real queue | No polling, no bloat | Delivery semantics, dead letters |
| Search, analytics | Index and warehouse fed by CDC | Scans and fuzzy matching off the primary | Lag; another copy of the data |
| Any of the above, cheaply | Separate database or partitions, same engine | Isolation with skills you have | Still one engine’s limits |
Fowler named the overall price: every store is a new interface to learn and tune, and the single transactional source of truth goes away. Add more backups, more on-call, duplicated data and more spend. Split too far, a database per tiny service, and you trade contention for distributed joins and sagas; see monolith vs microservices.
No transaction spans two stores
Once orders and events live apart, “write the order and publish the event” can half-succeed. Use a transactional outbox or CDC so the event comes from the committed row, and make consumers idempotent. Never dual-write from the request and hope.
Verify the fix
Rerun the per-workload query and the evening dashboard: business share of WAL and bytes should rise, the orders hit ratio should hold near 99.9% through a telemetry burst, and checkout p99 should stop moving with event volume. Then time a restore, because disaster recovery is where the split pays most.
The arithmetic: 2,000 GB ÷ 200 MB/s = 10,000 s ≈ 2.8 h. Nineteen hours of WAL at an average 1.5 MB/s is about 103 GB, or 17 minutes at 100 MB/s. After the split, 400 GB copies in 2,000 s ≈ 33 minutes, and the orders database writes about a quarter of the old WAL, so replay drops to about 4 minutes. The throughputs are assumptions; measure yours.
When it’s fine
One database is often the right answer:
- Early products and small data. At 2 orders a second and 20 GB, one PostgreSQL with good backups beats five systems nobody has time to run. Fowler suspected that many projects, probably most, are better off staying relational.
- Data that must commit together. A ledger and its audit trail, or an order and its outbox row, belong in one transaction.
- Postgres doing several jobs at moderate volume.
jsonbdocuments, full-text search withtsvector, aSKIP LOCKEDqueue at tens of jobs a second, and time-partitioned event tables are all reasonable. Azure’s data-model guide advises combining models only where access patterns or life cycles clearly diverge.
You’ve crossed the line when one of these is true: non-business data is most of the writes, WAL or bytes; business p99 moves with non-business volume; one append-only table outgrows the server’s RAM; a full restore can’t meet your recovery time objective; or data with very different retention or privacy rules (30-day logs, 8-year invoices) shares one backup.
In the interview
The sentence to say: “I group data by access pattern and value, not by convenience. Orders get the relational database to themselves; events, files, sessions and search each go to a store built for that pattern, and I connect them with an outbox and CDC because nothing spans two stores in one transaction.”
Cheat sheet
| Idea | Remember it as |
|---|---|
| Monolithic persistence | Every kind of data in one store, whatever its access pattern |
| What it shares | CPU, IOPS, buffer cache, WAL, vacuum, connections, backup, failure domain |
| Telltale numbers | Business 4% of writes and 20% of bytes; hit ratio 99.9% → 98% = 20× the misses |
| Restore maths | 2 TB at 200 MB/s ≈ 2.8 h; 400 GB ≈ 33 min |
| Detect | Writes, bytes and reads per workload; WAL per statement; connections per application_name; p99 vs non-business volume |
| Fix | Events to Kafka and a lake; files to object storage; sessions to a TTL store; jobs to a queue; search and analytics via CDC |
| Smaller step | Separate database on the same engine; time partitions with DROP for retention |
| What you pay | More systems, no cross-store transaction (outbox, CDC), more backups, duplicated data |
| When it’s fine | Small data, one team, data that must commit together, moderate volume |
How it shows up in interviews
It hides inside storage choices: rider location pings in a food-delivery design, click analytics in a URL shortener, photos in an Instagram-style feed, product images and search in an online store, message history versus presence in a chat app. The probes: “Where do the GPS pings go? Why not the orders database?”, “How long does a restore take?” and “What happens to checkout when you double logging?” Answer by naming each kind of data, its access pattern, and the store and cost that follow.
How to say it in the interview
“Monolithic persistence is keeping every kind of data in one database because it’s convenient: orders next to clickstream, audit logs, sessions, PDFs and a jobs table. Their access patterns and retention differ, but they share one buffer cache, one WAL, one connection limit and one backup. If orders are 4% of writes and 20% of bytes, a telemetry spike evicts order pages and checkout p99 rises with no change in orders. I’d measure writes, bytes and WAL per workload, then move events to Kafka and a lake, files to object storage with a key in the row, sessions to Redis with a TTL, and search and analytics behind CDC. The orders database keeps orders and an outbox. The price is more systems and no cross-store transaction.”
Isn’t running five stores worse than one? Often, at small scale. Each store costs skills, on-call and backups. I split only where access patterns or life cycles clearly diverge and the numbers show contention, and a separate PostgreSQL instance is a fine first step.
How do you keep an order and its event consistent across stores? With a transactional outbox: the event row commits with the order, and a relay publishes it afterwards. Or with CDC from the orders table. Consumers dedupe by event ID, because delivery is at least once.
Why not move the logs to a read replica, or buy a bigger instance? A replica receives the whole WAL, so it replays every log row too, and it can’t take writes. A bigger instance buys time, but the backup is still 2 TB and the failure domain is still shared.
When would you keep files in the database? When they are small and few, and must commit atomically with the row, such as a 20 KB signature image for a KYC record at low volume. The 2006 study puts the crossover between 256 KB and 1 MB, but at volume the WAL and backup cost decides first.
How do you know you’ve outgrown one database? Non-business data dominates writes, WAL or bytes, business p99 tracks it, or a full restore misses the RTO.
Where to go next
- Busy database and noisy neighbour: the look-alikes.
- SQL vs NoSQL and database types and databases for system design interviews: picking a store per pattern.
- Table partitioning and vacuum and bloat: living with the big tables that stay.
- Transactional outbox, change data capture and message queues: moving data between stores safely.
- OLTP vs OLAP and design an object store like S3: where analytics and files go.
- Disaster recovery and single point of failure: one database for everything is one failure for everything.
References and further reading
- Monolithic Persistence antipattern (Azure Architecture Center)learn.microsoft.comContention and poor fit as the two harms, the detection steps, and a load test where moving log writes to a separate database raised peak throughput by about 500 requests per second and cut business-database DTU use from 100% to about 75%.
- PolyglotPersistence (Martin Fowler’s bliki)martinfowler.comChoosing storage by how data is manipulated, the complexity it costs, and the view that many projects are still better off relational.
- Understand data models (Azure Architecture Center)learn.microsoft.comRelational, key-value, time-series, object, search and analytical models with typical workloads, and the advice to combine models only where access patterns or life cycles clearly diverge.
- AWS Well-Architected, PERF03-BP01: Use a purpose-built data storedocs.aws.amazon.comThe data characteristics to inventory (durability, access pattern, latency, IOPS, retention) and the anti-pattern of sticking to one store because the team knows it.
- PostgreSQL documentation: The Cumulative Statistics Systempostgresql.orgpg_stat_user_tables write counters, pg_statio_user_tables block reads and hits, pg_stat_activity.application_name, and the caveat that reads may come from the OS cache.
- PostgreSQL documentation: pg_stat_statementspostgresql.orgPer-statement wal_records, wal_fpi and wal_bytes, used here to attribute WAL volume to workloads.
- PostgreSQL documentation: Table Partitioningpostgresql.orgDropping or detaching a partition is far faster than a bulk DELETE and avoids its VACUUM overhead; the rule of thumb that partitioning pays once a table exceeds the server’s memory.
- To BLOB or Not To BLOB: Large Object Storage in a Database or a Filesystem? (Sears, van Ingen and Gray, 2006)arxiv.orgFragmentation measurements behind the rule that objects under 256 KB suit a database and objects over 1 MB suit a file system.
- Django documentation: How to use sessionsdocs.djangoproject.comSessions live in the database by default, expired rows are never purged automatically, and cache-backed sessions expire on their own.
- Amazon S3 User Guide: Download and upload objects with presigned URLsdocs.aws.amazon.comTime-limited GET and PUT URLs, valid for up to 7 days when created with the SDK or CLI, so file bytes bypass your servers.