Layer 3 · Go deeper · 12 min
Go deeper
Size it, build it, then make the calls an interviewer will push you on.
Back of the envelope
Size it up
Tap through each step of the arithmetic.
How many bytes a day once samples are compressed?
How much storage for 15 days raw and years of rollups?
How much RAM, and how many ingesters, hold the recent data?
How many samples does a 30-day panel over 500 pods read?
Build-up · 4 steps
Build the high-level design
One functional requirement at a time, ending with a platform that works for one region on one large machine.
Step 1 of 4
Collect every 10 seconds
Pods come and go every few minutes, so the collector learns its targets from the Kubernetes API. It adds labels like pod and zone, writes up as 1 or 0, and appends samples to a TSDB.
6 interview moments
You’re the candidate
The interviewer pushes on the hard parts. Pick an approach and see how it holds up.
Deep dive 1 of 6
Interviewer
“Ten million samples a second, almost all appends, never updated, deleted in bulk. Where do you store them?”
Your move. Pick an approach:
Deep dive 2 of 6
Interviewer
“200,000 containers, some behind NAT, plus batch jobs that live 8 seconds. How do their metrics reach the platform?”
Your move. Pick an approach:
Deep dive 3 of 6
Interviewer
“Big Billion Days: a developer adds
user_idto the checkout latency histogram “just for today”. How do you protect the platform?”With 2M users, that is 140M new series in an hour, hashed across every ingester.
Your move. Pick an approach:
Deep dive 4 of 6
Interviewer
“Checkout runs on 10 pods; one sits on a sick disk. How do you report a p99 that tells the truth?”
Your move. Pick an approach:
Deep dive 5 of 6
Interviewer
“Checkout has a 99.9% SLO over 30 days. Which rule should wake the on-call?”
Your move. Pick an approach:
Deep dive 6 of 6
Interviewer
“The worst monitoring failure is silence that looks like health. How do you make sure the alerting path itself never goes quiet?”
Your move. Pick an approach:
Build-up · 9 steps
Follow one sample, then one alert
Three paths share storage but fail independently. The alerting path’s only dependency outside the region is the heartbeat check.
Step 1 of 9
Scrape
Pulling inside the cluster keeps up and local pacing. Short-lived jobs push to the same agent, which drops forbidden labels, pre-aggregates and queues batches in a local WAL.