100k
row changes a second at sale peak
From 200 PostgreSQL and MySQL databases taking real payments. Every committed change must get out, in order per row, and the source must barely notice.
How do you see every change?
Bad: Poll an updated_at column
Deletes vanish; a slow commit is missed forever.
Good: Read the database’s log
Written anyway for crash recovery: deletes included, in commit order, no extra writes.
Copy 2 billion rows without pausing the stream
Write a marker into the log, read 1,024 rows by key, write another. Drop copied rows that changed in between: the log has them already.
Duplicates are normal: crashes and snapshots re-send events, and Kafka can’t tell they’re repeats.
The trick
Every event carries its log position. The sink applies it only if it’s newer than what it holds.
108 GB
of log kept per hour after a stall
The busiest primary writes 30 MB/s of log, and a slot keeps every byte its reader hasn’t confirmed. By default, until the disk fills.
Bound what a stalled slot costs
- Alert when a slot holds 50 GB
- Cap at 200 GB: invalidate the slot, save the disk
- Re-snapshot automatically, never drop slots by hand
- Heartbeats keep quiet databases confirming
Whoever runs ALTER TABLE changes every consumer’s contract
So check it before it runs. A gate in the migration pipeline asks the schema registry; a rename is blocked and becomes add, backfill, drop.
Remember
Read the log, not the tables. Key by primary key, apply only newer positions, and cap what a stalled slot can pin.