Layer 3 · Go deeper · 12 min
Go deeper
Size it, build it, then make the calls an interviewer will push you on.
Back of the envelope
Size it up
Tap through each step of the arithmetic.
How much storage keeps 30 days searchable and a year archived?
What would a full-text index on everything cost instead?
How big must Kafka be to hold 72 hours?
How many log-store nodes does the 3M lines/s peak need?
Build-up · 5 steps
Build the high-level design
One functional requirement at a time, ending with a pipeline that works for one region before floods, outages or cost.
Step 1 of 5
Ship logs, no code changes
Apps only write to stdout, so logging can never block them. A node agent tails the file, adds Kubernetes metadata and buffers to disk, and the gateway answers 200 only once Kafka has the batch.
6 interview moments
You’re the candidate
The interviewer pushes on the hard parts. Pick an approach and see how it holds up.
Deep dive 1 of 6
Interviewer
“You ingest 43 TB of raw logs a day and keep 30 days searchable. How do you index them?”
Your move. Pick an approach:
Deep dive 2 of 6
Interviewer
“Logs matter most during incidents, exactly when the backend is overloaded. How do you avoid losing them?”
Your move. Pick an approach:
Deep dive 3 of 6
Interviewer
“3,000 services log any fields they like; one logs status as 200, another as “OK”. How do you store them?”
Your move. Pick an approach:
Deep dive 4 of 6
Interviewer
“A bad deploy crash-loops checkout: its 400 pods jump from 4,000 to 400,000 lines a second. What protects everyone else?”
Your move. Pick an approach:
Deep dive 5 of 6
Interviewer
“Auditors want a year of logs; engineers mostly read the last hour. Where does each day of logs live?”
Your move. Pick an approach:
Deep dive 6 of 6
Interviewer
“Someone searches “error” over 30 days while others debug an incident. How do searches stay fast and fair?”
Your move. Pick an approach:
Build-up · 8 steps
Follow one line through the final design
Follow checkout’s payment-timeout line through eight paths. One pipeline writes; four readers consume Kafka, so search, tail, alerts and the archive fail independently.
Step 1 of 8
Collect on the node
The agent joins stack traces, collapses repeats, redacts secrets and caps each container before anything leaves the node. If the gateway is slow, zstd batches spill to a 2 GB disk buffer; the app never waits.