100M
active series, each sampled every 10 seconds
That is 10M samples a second, never pausing. Recent data sits in RAM, three copies of every series. Series, not bytes, set the bill.
Pull or push?
Bad: Every app pushes
Dead looks like idle, and a buggy sender can flood you.
Good: Scrape locally, push across
A local agent scrapes, so a dead pod reads up = 0, then pushes batches out.
A sample is 16 bytes. An on-time, unchanged one costs 2 bits.
Store how much the gap between timestamps changed, and which bits of the value changed. Facebook averaged 1.37 bytes a sample.
Someone adds user_id as a label, “just for today”.
Labels multiply. One histogram has 840,000 series; user_id adds 140M in an hour, more than the whole platform. Memory runs out, and every team loses data.
1.97 s
real p99, while averaged p99s say 281 ms
Nine pods report 90 ms, a tenth on a sick disk 2 s. Never average percentiles. Add up bucket counts across pods, then take the p99.
With a one-hour wait, a total outage pages no sooner than a 0.2% blip.
The trick
Page when the month’s error budget burns 14.4× too fast, over both the last hour and the last 5 minutes.
Who pages you when monitoring is down?
- Every rule runs twice; three Alertmanagers send one page
- Nothing on the alert path uses production Kafka or databases
- Always-firing alert pings an outside service; silence pages you
Remember
Active series set the cost: cap labels, merge histograms before the p99, page on burn rate, and watch the watcher from outside.