Three places to tap a write: table, transaction or log
Polling asks the table later, a trigger adds work to every write, and a log reader follows what the database writes anyway.
Poll by updated_at?
Bad: Newer than the watermark
A slow transaction stamps 10:00:05, commits at :12, and falls behind the watermark forever.
Good: Read the log
Every change after commit, in commit order, deletes and old values included.
WAL → replication slot → Debezium → a topic per table
Each message is keyed by primary key, so one row’s changes stay in one partition, in order. Consumers read at their own pace.
Inside a Debezium change event
- before and after hold the row images
- op is c, u, d, or r for snapshot reads
- source carries table, transaction and log position
- A delete is followed by a tombstone
A snapshot copies a chunk while live changes stream. Which copy of a row wins?
The trick
Bracket the read with low and high watermarks. Any key the log touches in between drops its snapshot row.
Consume it how?
Bad: Insert every event
Redelivery after a crash duplicates rows.
Good: Upsert by primary key
Drop stale versions, delete on d, and treat r as an upsert.
6.9 h
until a stalled slot fills 500 GB
At 20 MB of WAL a second, a slot whose connector stopped holds 72 GB an hour. Cap it with max_slot_wal_keep_size and alert.
Remember
Follow the log, not the tables. Every committed change, in order per row, at least once, so consumers upsert by key.