Layer 3 · Go deeper · 12 min
Go deeper
Size it, build it, then make the calls an interviewer will push you on.
Back of the envelope
Size it up
Tap through each step of the arithmetic.
How much storage do a week of replay and the latest rows need?
Can one log reader per database keep up with the busiest one?
How much WAL does a stalled slot pin on the busiest primary?
How long does copying the largest table take?
Build-up · 5 steps
Build the high-level design
One functional requirement at a time, ending with a design that works for a handful of databases.
Step 1 of 5
Capture every committed change
The database already logs each change before commit returns. A slot decodes committed transactions in commit order, and the connector confirms a position only after Kafka holds the events, so a crash is survivable.
6 interview moments
You’re the candidate
The interviewer pushes on the hard parts. Pick an approach and see how it holds up.
Deep dive 1 of 6
Interviewer
“200 databases, 2,000 tables, and the sources take real payments. How do you learn that a row changed?”
Your move. Pick an approach:
Deep dive 2 of 6
Interviewer
“A new search index needs every existing row of a 2-billion-row table, then the live stream. How do you copy it?”
Throttled to 20k rows a second, one reader needs about 28 hours.
Your move. Pick an approach:
Deep dive 3 of 6
Interviewer
“Connector crashes, database crashes and snapshots all re-send events. How do you keep every sink exactly right?”
Your move. Pick an approach:
Deep dive 4 of 6
Interviewer
“The orders primary dies and a standby is promoted. How does the connector carry on without a gap?”
Up to PostgreSQL 15, a logical slot exists only on the primary.
Your move. Pick an approach:
Deep dive 5 of 6
Interviewer
“A migration renames the column phone to mobile. Search, fraud and the lakehouse read that topic. How do you ship it?”
Your move. Pick an approach:
Deep dive 6 of 6
Interviewer
“A new fraud feature needs every orders row, and the lakehouse loader was down all weekend. How do they catch up?”
Neither may hurt the source database.
Your move. Pick an approach:
Build-up · 8 steps
Follow one price change through
The complete platform for one source database, repeated 200 times on shared Kafka, Connect and registry clusters.
Step 1 of 8
Commit the price change
The catalogue service commits the price UPDATE. The change lands in the WAL and streams to the standby, which holds a synced copy of the CDC slot.