Design metrics, monitoring and alerting

It is 9:47 pm, the last over of an IPL final, and one in twenty subscription payments on a streaming app has started to fail. Somewhere in a fleet of 50,000 hosts and 200,000 containers, a database connection pool has run dry. Whether an engineer is looking at the right graph two minutes later — or reads about it on social media an hour later — depends on the metrics platform: the system that collects numbers such as http_requests_total and node_cpu_seconds_total from every process every 10 seconds, keeps them for years, answers queries in milliseconds, and pages the right on-call exactly once. It sounds like “a database for numbers”. The design gets interesting because the write rate is enormous and never pauses (10 million samples a second), because one careless label can multiply the data a millionfold, because the obvious way to compute a percentile is wrong, and because the monitoring system has to keep working on the one day that matters most: the day everything it watches is broken.

Understand the problem

A monitoring platform is three products glued together: a collection pipeline, a time-series database and an alerting engine. Say that out loud, and draw the line between metrics and their two siblings. A metric is a number sampled at a fixed interval — how many requests, how slow, how full — and it costs the same whether a service handles 10 requests a second or 10 million. That is its superpower, and also why it cannot tell you what happened to order 4711. Per-event questions belong to logs and traces.

Functional requirements

1. Services and hosts should be able to expose metrics — counters, gauges and histograms with labels — and the platform should collect them every 10 seconds. For example, http_requests_total{service="checkout", route="/pay", code="500"} from every checkout pod.
2. Engineers should be able to query metrics with a PromQL-style language and build dashboards over any range, from the last 5 minutes to the last 5 years.
3. Teams should be able to define alert rules that notify the right on-call through a pager, Slack or a ticket — one notification per incident, not one per pod.

Non-functional requirements

• Scale. 50,000 hosts and 200,000 containers produce about 100M active series; at a 10-second scrape interval that is 10M samples/s. Plan for 2× bursts: a fleet-wide deploy or an IPL-final traffic spike creates millions of new series in minutes.
• Freshness. A sample is queryable within 15 s of being scraped, at p99. Alerts are only as fresh as the data they read.
• Query latency. A last-hour dashboard panel returns in under 200 ms at p50 and 2 s at p99; a 13-month trend in under 5 s.
• Alert latency and availability. A hard failure pages the on-call within about 90 seconds plus the rule’s own waiting window. The alerting path targets 99.99% availability: it must be more available than anything it watches.
• Retention. Raw 10-second data for 15 days; 5-minute rollups for 13 months, so this year’s Big Billion Days can be compared with last year’s; 1-hour rollups for 5 years.
• Durability, not exactness. Losing a node must lose no data, but a few missing samples are acceptable. Facebook’s Gorilla paper makes the same bet: individual points can be lost without spoiling aggregates.
• Isolation. About 500 teams share the platform. One team’s bad deploy must not slow down or blind any other team.
• Independence. The platform must keep working when production is down, so it cannot depend on the shared Kafka, databases or service mesh it monitors.

Out of scope

• Logs and traces. They are per-event data with different storage and query patterns; see design a distributed logging system.
• Exact business analytics such as revenue per user or daily active users. Exactness and per-user dimensions belong in an analytics pipeline, as in web analytics.
• Delivering the page. SMS, voice and push delivery with retries is a notification system; the Alertmanager hands off to it.
• Machine-learned anomaly detection. Paging rules should be simple and explainable; there is a follow-up on this at the end.
• The dashboard product itself. Grafana-style dashboards are treated as query clients.

The set-up

Plan the approach

Satisfy the three functional requirements with the simplest design that works on one machine, then spend the interview in the deep dives. Name the three paths early — a write path (collect → distribute → store), a read path (query → merge) and an alerting path (evaluate → group → notify) — and say that the alerting path must never break. The deep dives follow from the numbers.

Core entities

• Series — a metric name plus labels, such as http_requests_total{service="checkout", route="/pay", code="500", pod="checkout-7f9c"}. Its ID is a hash of the sorted labels; about 100M are active at any moment.
• Sample — a millisecond timestamp and a float64: 16 bytes raw. A native histogram sample holds a whole distribution.
• Chunk — a run of consecutive compressed samples of one series, say 120 of them (20 minutes at 10 s).
• Block — every chunk plus a label index for a two-hour window. Immutable once written.
• Rule — a recording rule (a query whose result is stored as a new series) or an alerting rule (a query, a waiting window, labels, and annotations such as a runbook link).
• Alert — a label set with a state (pending, firing, resolved) and a start time.
• Route and silence — label matchers that choose a receiver, or mute alerts for a time window.
• Tenant — a team, with limits on active series, samples per second and query cost.

API

There are four surfaces: how data gets in, how queries get out, how rules are managed, and how humans silence alerts.

Exposition (pull). Every target serves its current values as text. Counters only ever go up; the histogram is a set of cumulative bucket counters plus a sum and a count.

GET /metrics HTTP/1.1
Host: checkout-7f9c:9090

# TYPE http_requests_total counter
http_requests_total{route="/pay",code="200"} 1048213
http_requests_total{route="/pay",code="500"} 1311
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{route="/pay",le="0.1"} 990120
http_request_duration_seconds_bucket{route="/pay",le="0.5"} 1046001
http_request_duration_seconds_bucket{route="/pay",le="+Inf"} 1049524
http_request_duration_seconds_sum{route="/pay"} 61822.4
http_request_duration_seconds_count{route="/pay"} 1049524

Remote write (push). Collector agents send compressed batches to the platform’s write endpoint, tagged with the team that owns them.

POST /api/v1/push HTTP/1.1
Host: metrics-write.ap-south-1.internal
Content-Type: application/x-protobuf
Content-Encoding: snappy
X-Tenant: team-payments

The body is a batch of series, each with its full label set and a few samples. A 2xx means stored; 400 means invalid data, so do not retry; 429 means the team is over its limit, so back off.

Query. A range query evaluates one expression at every step between a start and an end (URL encoding left out for readability):

GET /api/v1/query_range?query=sum by (code) (rate(http_requests_total{service="checkout"}[5m]))
    &start=2026-10-27T15:30:00Z&end=2026-10-27T16:30:00Z&step=30s

{
  "status": "success",
  "data": {
    "resultType": "matrix",
    "result": [
      { "metric": { "code": "200" }, "values": [[1793115000, "4120.5"], [1793115030, "4188.1"]] },
      { "metric": { "code": "500" }, "values": [[1793115000, "2.1"], [1793115030, "231.7"]] }
    ]
  }
}

Values travel as strings so that NaN and infinities survive JSON.

Rules. Teams manage rule groups as code; CI pushes them with PUT /api/v1/rules/payments. This group records two error ratios and pages on a fast burn of the error budget (explained in the deep dives):

groups:
  - name: checkout-slo
    interval: 15s
    rules:
      - record: service:http_errors:ratio_rate5m
        expr: |
          sum by (service) (rate(http_requests_total{code=~"5.."}[5m]))
            / sum by (service) (rate(http_requests_total[5m]))
      - record: service:http_errors:ratio_rate1h
        expr: |
          sum by (service) (rate(http_requests_total{code=~"5.."}[1h]))
            / sum by (service) (rate(http_requests_total[1h]))
      - alert: CheckoutBurningErrorBudget
        expr: |
          service:http_errors:ratio_rate1h{service="checkout"} > (14.4 * 0.001)
            and service:http_errors:ratio_rate5m{service="checkout"} > (14.4 * 0.001)
        labels:
          severity: page
          team: payments
        annotations:
          summary: "Checkout is spending its 30-day error budget 14x too fast"
          runbook: "https://runbooks.internal/checkout/errors"

Silences. During a planned failover drill, an engineer mutes matching alerts for two hours, with a name and a reason:

POST /api/v2/silences
Content-Type: application/json

{
  "matchers": [
    { "name": "cluster", "value": "mum-2", "isRegex": false },
    { "name": "team", "value": "payments", "isRegex": false }
  ],
  "startsAt": "2026-10-27T18:30:00Z",
  "endsAt": "2026-10-27T20:30:00Z",
  "createdBy": "asha@payments",
  "comment": "Planned Postgres failover drill, change CHG-4417"
}

Back-of-envelope estimates

These numbers decide how many machines you need and which resource runs out first. (For the method, see back-of-envelope estimation.)

Active series. Hosts: 50,000 × 500 series (CPU cores, memory, disks, network interfaces) = 25M. Containers: 200,000 × 300 series (request counters and histograms per route and status, runtime metrics) = 60M. Infrastructure — load balancers, databases, Kafka, Kubernetes objects — adds about 15M. Total ≈ 100M active series.

Samples. 100M ÷ 10 s = 10M samples/s, or 10M × 86,400 = 864 billion samples a day.

Bytes per sample. Raw, a sample is an 8-byte timestamp and an 8-byte float: 864 billion × 16 B ≈ 13.8 TB a day. Gorilla-style compression got Facebook to 1.37 bytes per sample, and the Prometheus storage docs quote 1–2 bytes. Plan with 2 bytes including the index, as the Mimir capacity guide does: 864 billion × 2 B ≈ 1.73 TB a day, eight times smaller than raw.

Storage for the whole retention.

• Raw for 15 days: 1.73 TB × 15 ≈ 26 TB.
• 5-minute rollups for 13 months: each series makes 288 points a day, and each point keeps five aggregates (count, sum, min, max and a counter value) — call it 10 bytes. 100M × 288 × 10 B = 288 GB a day, × 395 days ≈ 114 TB.
• 1-hour rollups for 5 years: 100M × 24 × 10 B = 24 GB a day, × 1,826 days ≈ 44 TB.
• Total ≈ 184 TB if every series is kept for the full period. If, as at Uber, only metrics that opt in get long retention — say one in five — it drops to 26 + 0.2 × 158 ≈ 58 TB. All of it sits in object storage, the cheapest durable bytes you can buy.

Memory. Recent data lives in memory on ingesters, three copies of every series: 100M × 3 = 300M in-memory series. The Mimir guide budgets 1 core and 2.5 GB per 300,000 in-memory series, plus 50% memory headroom: 300M ÷ 300k = 1,000, so 1,000 cores and 2.5 TB of RAM (3.75 TB with headroom). With 16-core, 64 GB machines holding 4M series each (13.3 cores; 33 GB, or 50 GB with headroom), that is 300M ÷ 4M = 75 ingesters, 25 per zone, each appending 10M × 3 ÷ 75 = 400k samples/s.

Distributors. At 1 core per 25,000 samples/s: 10M ÷ 25k = 400 cores, doubled for bursts = 800 cores, for example 100 pods of 8 cores.

Network. At an assumed 15 bytes per sample on the wire after batching and compression, ingest is 10M × 15 B = 150 MB/s ≈ 1.2 Gbit/s, tripled on the way to the ingesters.

Queries. 1,000 dashboards open at peak × 20 panels ÷ a 30-second refresh ≈ 670 queries/s, five times that during an incident. Rules are the steady load: 30,000 recording and alerting rules every 15 s = 2,000 evaluations/s, around the clock.

Samples per query. A last-hour panel for a 500-pod service reads 500 × 360 = 180k samples, which is trivial. The same panel over 30 days reads 500 × 259,200 = 129.6M samples, about 260 MB, at raw resolution — against 500 × 8,640 = 4.32M from 5-minute rollups and 500 × 720 = 360k from hourly ones.

High-level design

Build the platform one requirement at a time; by the end it works for one region, on one large machine.

Collect metrics from every host and container

Start with the data model. A series is a metric name plus a set of labels; a sample is a timestamp and a value. Every process links a client library that keeps its counters, gauges and histograms in memory and serves them as text at /metrics. Hosts run a small agent for CPU, disk and network numbers, and databases or Kafka brokers get exporters that translate their statistics into the same format. A collector keeps a live target list from service discovery — pods come and go every few minutes — and scrapes each target every 10 seconds.

The flow:

1. The collector asks the Kubernetes API for pods marked as scrapeable, and watches for changes.
2. Every 10 s it sends GET /metrics to each target, with a timeout shorter than the interval.
3. It parses the response and attaches target labels such as job, pod, zone and region, so the application never needs to know where it runs.
4. It writes a synthetic up series — 1 if the scrape worked, 0 if not — and appends every sample to the TSDB.

Query metrics and build dashboards

A dashboard panel is a saved query, re-run every 30 seconds. The query language works on series selected by labels: rate(http_requests_total{service="checkout"}[5m]) turns each counter into a per-second rate over a 5-minute window, and wrapping it in sum by (code) (…) adds the pods together, keeping only the status code. A range query is the same expression evaluated at every step between a start and an end.

The flow:

1. The query service parses the expression and pulls out its label matchers.
2. It looks each matcher up in the inverted index and intersects the sorted lists of series IDs — the same posting-list trick a search engine uses.
3. It reads the compressed chunks of the matching series for the time range, decodes them, evaluates rate() per series and sums across them.
4. The dashboard draws one line per status code.

Keep expensive aggregations out of every refresh with recording rules: queries the platform evaluates every 15–60 seconds and stores as new series, such as service:http_errors:ratio_rate5m. A dashboard over 300 pods then reads one pre-summed series instead of 300.

Alert the right on-call

An alert rule is a query plus a condition, evaluated on a timer. A rule evaluator runs every rule every 15 seconds. Each label set the expression returns becomes an alert, which moves from pending to firing once the condition has held for the rule’s for: duration. Firing alerts go to an Alertmanager, which turns a stream of alerts into a few useful notifications.

The Alertmanager does five jobs, all driven by labels (see the Alertmanager README):

• Deduplicate the same alert from two evaluators, or re-sent every 15 seconds.
• Group alerts that share alertname and cluster into one notification. It waits group_wait (30 s in the example configuration) for siblings, sends updates at most every group_interval (5 minutes), and repeats an unresolved page every repeat_interval (3 hours).
• Route with a tree of matchers on team and severity: the payments pager for severity="page", a Slack channel for severity="ticket".
• Inhibit: while “cluster unreachable” is firing, mute every other alert from that cluster — they are consequences, not news.
• Silence matching alerts during planned maintenance, with an author and a reason.

You now have a working platform: one collector, one TSDB, one rule evaluator and one Alertmanager. It also has every problem the deep dives exist for: 100M series need about 100M ÷ 300k × 2.5 GB ≈ 830 GB of RAM on one box (by the Mimir ratio) before any replication; that box is a single point of failure for every alert in the company; and nothing stops one label from blowing it up.

Deep dives

How do we store 10 million samples a second cheaply?

Monitoring data has an unusual shape. Writes are relentless, tiny and almost all appends to the newest end of each series. Reads mostly want the last few hours, and they want many series at once (“every pod of checkout”). Nothing is ever updated, and old data is deleted in bulk. A general-purpose database pays for features this workload never uses.

Bad: one row per sample in a relational database. A table samples(series_id, ts, value) with an index on (series_id, ts). You gain SQL and familiar tools. You pay in size and write amplification: with a row header and an index entry, each 16-byte sample costs roughly 70 bytes — about 60 TB a day — and every insert updates a B-tree. Time-series extensions for relational databases narrow the gap and suit thousands of samples a second; at 10 million, you are running a transaction engine you do not need.

Good: a wide-column store. Use Cassandra or HBase with the row key “series ID + two-hour bucket” and one column per timestamp. Writes shard across nodes and data is sorted for range scans — roughly what OpenTSDB, Facebook’s older ODS on HBase, and Uber’s first M3 (Cassandra, plus Elasticsearch to search labels) did. You pay three ways. Bytes per sample stay close to raw. Compaction keeps rewriting data that will never change; Uber’s M3 write-up says such systems spend a large share of CPU, memory and disk I/O on it. And reads are slow: Facebook put Gorilla, an in-memory layer, in front of HBase and cut query latency 73 times.

Great: a purpose-built time-series database. Four ideas, from Gorilla and the Prometheus storage engine:

1. An in-memory head, protected by a write-ahead log. New samples are appended to an in-memory “head” covering roughly the last two hours, and to a write-ahead log on local disk in 128 MB segments. A crash replays the log; nothing acknowledged is lost. Recent data — what alerts and most dashboards read — is served straight from RAM.
2. Compression built for time series. Timestamps arrive at nearly fixed intervals and values change slowly, so store the change in the gap between timestamps (delta of delta) and the XOR of each value with the previous one, in variable-length codes. Facebook measured 96% of timestamps compressing to a single bit and 51% of values to a single bit, for an average of 1.37 bytes per sample instead of 16. That 12× saving is what let the last 26 hours of Facebook’s monitoring data fit in 1.3 TB of RAM across 20 machines when Gorilla launched.
3. Immutable two-hour blocks. Every two hours the head is cut into a block: a directory of compressed chunks, an index and a metadata file. Blocks are never modified; background compaction merges them into larger ones (in Prometheus, up to 10% of the retention period or 31 days). Retention is deleting whole directories — no row deletes, no vacuum. Gorilla found that blocks longer than two hours barely improve compression.
4. An inverted index of labels. Each block maps every label pair to a sorted list of series IDs, so a selector is a fast list intersection, as in Fig. 2.

You gain about 1.4–2 bytes per sample, sequential appends, nearly free retention and recent data in memory. You pay with RAM that grows with active series, not traffic; with awkward handling of late samples, since blocks are immutable; and with a single-node engine — the Prometheus docs say plainly that local storage is not clustered or replicated. The scaling deep dive fixes that.

Should collectors pull metrics, or should services push them?

Large systems have made opposite choices: Prometheus pulls by scraping an HTTP endpoint on every target, while Google’s Monarch has clients push to nearby ingestion routers. Both work at huge scale; the trade-offs are about control, liveness and network reach (see push vs pull architecture).

Pull (scrape). The collector reads its target list from service discovery and asks each target for its current values every 10 seconds. You gain three things. The collector sets the pace, so a buggy service cannot flood the platform. Liveness is free: a failed scrape records up = 0, so a dead pod shows up as a fact rather than as silence — a cheap, honest health check. And debugging is a curl to /metrics. You pay with reach: the collector needs a network path to every target, which breaks across NAT, firewalls, other VPCs and phones; a job that lives for 8 seconds can finish between two scrapes and never be seen; and someone must shard targets across collectors.

Push. Each process, or an agent beside it, sends samples to a central endpoint. You gain the opposite set: only outbound connections are needed, and short-lived jobs report before they exit. You pay with control. The receiver must defend itself against a client that suddenly sends ten times its usual rate, so it needs per-sender rate limits. And silence is ambiguous — a service that stops sending might be idle or dead — so you need heartbeats or “this metric is absent” alerts. The pull world’s patch for batch jobs, a push gateway that holds the last values for scraping, shares the flaw: it keeps serving a dead job’s last numbers until someone deletes them.

Great: pull inside the failure domain, push across the network. Run an agent per cluster (or per node) that discovers and scrapes local targets — keeping up, local rate control and the curl debugging story — and remote-writes compressed batches over a few long-lived connections to the central distributors. Short-lived jobs push to the same local agent. The agent is also the right place for hygiene: relabel rules that drop forbidden labels, renames, and pre-aggregation (see the cardinality deep dive). It keeps a queue backed by its own write-ahead log, so a 20-minute network blip delays data instead of losing it. Large platforms tend to converge on this shape: Grafana Mimir is fed by Prometheus servers and agents that remote-write into it, and Uber’s M3 accepted both pushed StatsD metrics and Prometheus remote writes.

How do we scale writes and reads past one machine?

100M series need about 830 GB of RAM before replication, and no single box should stand between an outage and the pager.

Bad: a bigger server. Even with a terabyte of RAM, it is one machine: when it reboots, every dashboard and alert in the region goes dark, and when its disk dies, the history goes with it.

Good: shard by hand — one TSDB server per team or cluster, run in pairs. Each team or Kubernetes cluster gets two identical servers scraping the same targets; a global query layer fans out to all of them and removes duplicates at query time. You gain isolation and simplicity: a team’s mistake fills only its own server. You pay in toil: someone must split big teams across servers as they grow, and cross-team questions fan out to hundreds of servers. Google lived this with Borgmon, whose users sharded targets and built query evaluation trees by hand — one reason the Monarch paper gives for replacing it.

Great: a horizontally scalable cluster that splits the work into stateless and stateful parts.

• Distributors (stateless) receive remote writes, validate labels, enforce each team’s limits, and hash each series to find its home.
• Ingesters (stateful) hold the head of their share of series in memory with a WAL, cut two-hour blocks and upload them to object storage.
• Compactors merge uploaded blocks, drop the duplicate copies that replication created, and build 5-minute and 1-hour rollups.
• Queriers (stateless) read recent data from ingesters and older blocks from object storage through store gateways, which cache block indexes.
• A query frontend in front of the queriers splits, caches and queues queries.

How a series finds its ingesters. The distributor hashes the tenant and the sorted label set into a 32-bit token and looks it up on a consistent-hashing ring where every ingester owns many tokens. Walking clockwise, it picks the next three ingesters in three different zones and sends the sample to all of them; the write succeeds when two acknowledge. Uber’s M3 makes the same choice: a quorum write to three replicas in each region.

Why shard by series and not by metric name? A popular metric such as http_requests_total has millions of series; sharding by name would pile them onto one node. Hashing the full label set spreads them evenly, at the price that a query for one metric must ask every ingester — which is why queriers fan out in parallel.

What you gain: lose one ingester, or a whole zone, and every series still has two live replicas, so writes and reads carry on. Because ingesters hold only the last few hours, most of the data — and its cost — lives in object storage, where durability is someone else’s problem. What you pay: five component types to operate, and three copies of every recent sample in RAM. Some newer designs put a dedicated, replicated log such as Kafka between distributors and ingesters instead; if you do, it must be the platform’s own cluster, never the shared production one it watches.

Reads over long ranges. Three mechanisms keep a 13-month dashboard fast:

• Split and cache. The query frontend cuts a 30-day query into 30 one-day queries, runs them in parallel on different queriers, and caches each finished day. A past day’s answer never changes, so most of a refreshed long-range panel comes from cache.
• Rollups. The compactor writes 5-minute and 1-hour versions of every block; Thanos, for example, builds the 5-minute level for blocks older than 40 hours and the 1-hour level after 10 days. Each rollup point keeps count, sum, min, max and a counter value, so rate(), averages and maxima still work. The querier picks the coarsest resolution that fits the query’s step.
• Recording rules precompute the aggregates people look at every day, such as the error ratio per service summed over hundreds of pods, so a dashboard reads one series instead of 300.

What happens when someone adds user_id as a label?

It is the Big Billion Days sale, a customer complains about slow checkouts, and a developer adds user_id to the checkout latency histogram “just for today”. Every distinct label combination is a new series, so labels multiply. The histogram already has 40 routes × 5 status codes × 14 series per histogram (12 buckets, a sum and a count) × 300 pods = 840,000 series. With 2M users each producing about five route–status–pod combinations an hour, user_id adds 2M × 5 × 14 = 140M new series in an hour — more than the entire platform holds today. Ingesters run out of memory, crash, replay their WAL and crash again. Because series are hashed across every ingester, every team loses data, not just checkout.

Churn adds to this even without bad labels: every redeploy renames pods, so if all 200,000 containers roll once a day, about 200,000 × 300 = 60M series are created and retired daily.

The Prometheus instrumentation guidance is blunt: every label set costs RAM, CPU, disk and network; keep most metrics’ cardinality below 10, look hard at anything over 100, and never use user IDs, email addresses or other unbounded values.

Bad: trust developers and clean up after the outage. You gain freedom; you pay with a shared outage every few weeks, discovered when the platform team’s own pager goes off.

Good: hard limits at the front door. Distributors and ingesters enforce per-team limits: active series per team, series per metric, samples per second with a burst allowance, labels per series and label-value length. Existing series keep flowing; new series beyond a limit are rejected with a clear error and counted in a per-team “samples discarded” metric. You gain a blast radius of one team’s new series. You pay with friction — limits need owners and overrides — and the offending team loses its new data until someone notices.

Great: limits, plus detection, prevention and a better home for the question.

• Detect early. Treat cardinality as a metric: active series per team and per metric, and new series created per minute. Alert the owning team when a metric grows tenfold in an hour, and publish a “top 20 metrics by series” page. Exact distinct counts over 100M series are expensive; HyperLogLog sketches estimate them in kilobytes.
• Prevent at the edge. Relabel rules in the agent drop forbidden labels (user_id, request_id, email) and normalise paths (/orders/48213 becomes /orders/:id); a CI check rejects instrumentation that uses them.
• Aggregate before storing. If nobody needs per-pod latency next year, sum the pod label away in the agent or a streaming aggregator before storage. Uber’s M3 aggregates 500 million metrics per second down to the 20 million it persists — a 25× reduction. Monarch aggregates at ingestion too, for example disk I/O per user across millions of disk servers, which would otherwise be tens of billions of series.
• Send per-entity questions elsewhere. “Which user?” is a question for logs, traces and exemplars (a trace ID attached to a histogram bucket), or for an analytics store. Metrics answer “how many, how fast, how full”.

How do we compute a correct p99 across 500 pods?

“p99 latency” is what every SLO and dashboard asks for, and it is the easiest number in the system to get wrong.

Bad: each pod computes its own p99, and you average them. Client libraries can compute quantiles in-process (Prometheus calls these summaries). They are precise for one pod, but quantiles do not aggregate: the Prometheus docs are explicit that you cannot combine them across replicas. Picture checkout on 10 pods. Nine handle 7,000 requests a second between them with a p99 of 90 ms. Pod 9 sits on a sick disk and handles the other 3,000, all taking between 1.2 and 2 s. The average of the ten p99s is (9 × 90 + 2,000) ÷ 10 = 281 ms, comfortably under a 500 ms alert. The real p99 across all 10,000 requests sits at rank 9,900: past the 7,000 fast ones, 2,900 of the way through pod 9’s 3,000 — about 1.97 s. Thirty percent of users wait more than a second while the dashboard stays green.

Good: bucketed histograms, merged before the quantile. Each pod counts requests into fixed buckets — 100 ms, 250 ms, 500 ms, 1 s, 2.5 s and so on — as cumulative counters. Counts add up across pods, so the query sums the buckets first and estimates the quantile last:

histogram_quantile(0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket{service="checkout"}[5m])))

In the example, 7,000 requests land at or under 100 ms and 3,000 between 1 and 2.5 s. Rank 9,900 falls in the 1–2.5 s bucket, and linear interpolation gives 1 + (9,900 − 7,000) ÷ 3,000 × 1.5 ≈ 2.45 s. Not exact — the true value is 1.97 s — but in the right place, and the alert fires. You pay twice: every bucket is its own series (12 buckets plus a sum and a count is 14 series per label set), and the error is as wide as the bucket. So put a boundary exactly on each SLO threshold: if the objective is “99% under 300 ms”, a 0.3 s bucket makes “fraction under 300 ms” exact.

Great: native (exponential) histograms or mergeable sketches. Native histograms store a whole distribution as one series with sparse, exponentially growing buckets, so resolution is high wherever the data is and empty buckets cost nothing; the Prometheus docs recommend them over both classic histograms and summaries. Other systems use mergeable sketches such as t-digest. Monarch made the distribution a first-class value type because its predecessor lacked good support for one. You gain accuracy and far fewer series; you pay with newer tooling that every hop — client, agent, storage and query engine — must understand.

How do we page on what users feel, and not on noise?

A page wakes a human. The SRE workbook judges an alerting rule on four things: precision (how many pages were real), recall (how many real problems paged), detection time and reset time.

Bad: thresholds on causes. “CPU above 80% for 5 minutes on any host”, across 50,000 hosts, pages several times a night for batch jobs that are supposed to use the CPU — and stays quiet when a bad config makes every request fail fast with idle CPUs. Precision and recall are both poor, and on-call learns to ignore the pager.

Good: thresholds on symptoms, with a wait. Alert on what users experience — error ratio, latency — and add for: to ride out blips: “error ratio above 0.1% for 10 minutes”. Better, but the workbook shows two flaws. The wait does not scale with severity: with a one-hour for:, a total outage pages no sooner than a 0.2% blip, and burns 140% of the month’s error budget before anyone is woken. And a service that fails completely for 5 minutes out of every 10 resets the timer each time and never pages at all.

Great: multiwindow, multi-burn-rate alerts on the SLO. Start from the error budget. A 99.9% SLO over 30 days allows 0.1% of requests to fail — the equivalent of 30 × 24 × 60 × 0.001 ≈ 43 minutes of total outage. The burn rate is how fast you spend it: burn rate 1 empties the budget exactly at the end of the 30 days. Page when the burn rate is high enough to matter, measured over a long window to prove it is significant and a short window to prove it is still happening:

The arithmetic: 14.4 × 1 h ÷ 720 h = 2% of the budget; 6 × 6 h ÷ 720 h = 5%; 1 × 72 h ÷ 720 h = 10%. At 14.4×, the whole month’s budget would be gone in 720 ÷ 14.4 = 50 hours.

You gain pages that map to real user pain, fast detection for big outages — at 100% errors the 1-hour average crosses 1.44% in about 52 seconds — and alerts that clear minutes after recovery. You pay with more rules to manage (generate them from one SLO definition per service rather than writing them by hand) and with noisy maths for low-traffic services, where three failed requests can look like a 14× burn; add a minimum request count, or synthetic traffic, there.

Grouping and inhibition do the rest: while “region ap-south-1 unreachable” fires, the hundreds of service alerts it implies stay muted. Cause-based alerts — a disk full in four hours, a certificate expiring in seven days — still exist, as tickets rather than pages.

Who watches the watchers?

The worst failure of a monitoring system is not a crash. It is silence that looks like health: the rule evaluator stalls, no alert fires, and everyone assumes all is well.

Bad: one rule evaluator and one Alertmanager, running inside the production cluster. If either stops, nothing pages. If the production cluster’s network, DNS or service mesh fails, the alerting path fails with it — on exactly the day you need it.

Good: redundancy everywhere on the alert path. Run two rule evaluators on the same rules (or shard rule groups across a ring with replication). Each sends every alert to every Alertmanager in a cluster of three; the Alertmanager README warns against load-balancing between them. The Alertmanagers gossip their notification log and silences — every 200 ms by default — so only one of them sends each page. During a network partition between them, each side may page: the design prefers two pages to none.

Great: assume the watcher will fail, and design for it.

• A dead man’s switch. Add an alert that always fires (expr: vector(1)) and route it every minute to an external heartbeat service on a different provider. If the heartbeats stop for five minutes, that service pages through a separate path. One rule tests the whole chain.
• No circular dependencies. Monarch keeps its data in memory partly so that the alerting path never depends on Bigtable, Colossus or Spanner — storage systems that rely on Monarch to be monitored. Your version: no shared production Kafka, database or service mesh on the alerting path, and static fallbacks for service discovery.
• Meta-monitoring. A small, separate set of collectors watches the platform’s own components — ingester memory, rule-evaluation lag, failed notifications — and each region’s stack watches the others.
• Regional autonomy. Each region ingests, stores and alerts on its own data, so losing another region costs the global view, not alerting. Monarch’s zones work this way, and its global queries return partial data rather than blocking; M3 likewise writes locally and fans queries out across regions (multi-region design).

The final design

Put the pieces together. Each region runs the same stack; the write path, the read path and the alerting path share storage but fail independently.

Follow the data through the system:

1. Scrape. In each cluster, an agent discovers and scrapes pods and hosts every 10 s; short-lived jobs push to it. It drops forbidden labels, pre-aggregates and queues batches in a local WAL.
2. Remote write. Agents send compressed batches to the region’s distributors through a load balancer.
3. Distribute. Distributors validate labels, apply per-team limits (429 on overflow), hash each series and write it to three ingesters in three zones; two acknowledgements make it durable.
4. Ingest and flush. Ingesters append to memory and their WAL, and every two hours upload an immutable block to object storage.
5. Compact. Compactors merge blocks, drop replica duplicates, write 5-minute and 1-hour rollups, and delete blocks past retention.
6. Query. Dashboards call the query frontend, which splits long queries by day and serves cached days; queriers read recent hours from ingesters and older data from object storage, at the coarsest resolution that fits.
7. Evaluate. Rulers run recording rules and SLO burn-rate alerts every 15 s through the queriers, two copies of each rule group.
8. Notify. Every ruler sends firing alerts to all three Alertmanagers, which agree through gossip, then group, inhibit and route: one page to the on-call, one message to Slack.
9. Watch the watcher. An always-firing watchdog reaches an external heartbeat check every minute; if it goes quiet, the checker pages through a separate path. A global query layer fans out across regions.

Check it against the requirements. Ingest: 100 distributor pods and 75 ingesters carry 10M samples/s with 2× headroom, and losing an ingester or a whole zone loses nothing, because every series has three replicas and writes need two. Freshness: a sample is queryable from ingester memory as soon as it is appended, a few seconds after the scrape, well inside 15 s. Query latency: recent panels read RAM, long panels read rollups, and the frontend caches finished days. Alert latency: a hard failure is scraped within 10 s, evaluated within 15 s and grouped for 30 s — about a minute, inside the 90-second budget — plus the rule’s own window. Retention: 26 TB of raw blocks and 32–158 TB of rollups, all in object storage. Isolation: per-team limits at the distributors, and cardinality alerts before limits bite. Independence: nothing on the alerting path depends on shared production systems, each region alerts on its own data, and the watchdog proves the chain works from outside.

What interviewers look for by level

Follow-up questions

What if a team wants error rates per merchant, for 2 million merchants?

That is per-entity analytics wearing a metrics costume: 2M merchants × 5 status codes × 3 regions is 30M series for one metric, almost a third of the platform. Keep metrics at the level you alert on — per service, payment method or bank — and send per-merchant events to a columnar store built for high-cardinality group-bys (OLTP vs OLAP). For “the 100 merchants with the most failures right now”, a streaming top-K with count-min sketches answers in kilobytes. Exemplars bridge the two: a latency bucket carries a trace ID, and the trace carries the merchant.

What if an agent is cut off for an hour and then sends a backlog?

Its WAL-backed queue has kept every sample, but replaying an hour at full speed competes with fresh data, and fresh data is what alerts read. Send newest first, cap the backfill rate per tenant, and accept late samples only within a bounded out-of-order window on the ingesters, whose memory cost grows with that window. Monarch takes a harder line: it drops writes that arrive too late (for aggregated data, anything outside a short admission window), so a congested zone catches up with the present instead of drowning in the past.

What if a whole region goes dark?

Its dashboards go dark with it, but nothing else should. Every region alerts on its own data, and cross-region meta-monitoring notices the silent region within a minute and pages — a region going quiet is itself a symptom. Global queries return the surviving regions’ data, flagged as partial. If the region holds the only copy of valuable history, copy its bucket to another region asynchronously; blocks are immutable, so it is a plain copy job (disaster recovery).

What if we want anomaly detection instead of static thresholds?

Use it for tickets and dashboards first, not pages. A paging rule should be simple enough that a sleepy on-call can see in ten seconds why it fired; learned thresholds are hard to explain at 3 am and drift with traffic. A useful middle ground is a readable comparison with the past — “UPI success rate is 30% below the same minute last week” — which catches a payment method quietly dropping to zero during a festival sale without a model anyone has to trust.

What if we must sell this as a multi-tenant SaaS?

Everything per-team becomes per-customer and enforced: a tenant ID on every write; per-tenant limits on series, samples and query cost; and shuffle sharding, so each tenant’s series land on a small random subset of ingesters and a noisy tenant can hurt only the few customers who share its subset (noisy neighbours and fairness). Price on active series and samples, the real cost drivers, so customers feel the cost of a label, and give every tenant its own rule namespace and Alertmanager configuration.

How to say it in the interview

“I’ll design an internal metrics platform for about 50,000 hosts and 200,000 containers. That’s roughly 100 million active series; at a 10-second interval, 10 million samples a second. Raw data stays for 15 days and rollups for years, and the alerting path has to be more reliable than anything it watches.

Collection is hybrid. An agent in each cluster discovers targets from the Kubernetes API and scrapes them, so we get liveness for free through the up metric. It drops forbidden labels, buffers in a local WAL, and remote-writes batches to the region.

Storage is a purpose-built time-series database, not a relational one. Stateless distributors validate each write, enforce per-team limits and hash every series onto a ring of ingesters: three replicas in three zones, two acknowledgements to succeed. Ingesters keep the last two hours in memory with a write-ahead log and compress samples Gorilla-style — delta-of-delta timestamps and XOR’d values — to under two bytes each. Every two hours they ship immutable blocks to object storage, where compactors deduplicate and build 5-minute and 1-hour rollups.

The number that drives cost is active series, so cardinality gets limits at the distributors, detection per team, and aggregation at the agent. Per-user questions go to logs and analytics, not labels.

Queries go through a frontend that splits long ranges by day and caches finished days; queriers read recent data from ingesters and older data from rollups in object storage. Percentiles come from histograms merged across pods before the quantile is computed — never from averaging p99s.

For alerting, rulers evaluate SLO burn-rate rules every 15 seconds: page if we burn budget 14.4 times too fast over both the last hour and the last five minutes. Alerts go to all three Alertmanagers, which gossip, so the on-call gets one grouped page. Finally, the watcher is watched: an always-firing watchdog heartbeats to an external service, nothing on the alert path depends on production Kafka or databases, and each region alerts on its own data.”

Concepts used in this design

• Observability: logs, metrics and traces — where metrics fit, and what they cannot answer.
• Reliability: SLIs, SLOs and error budgets — the budget that burn-rate alerts spend.
• Push vs pull architecture — scraping, remote write and the hybrid.
• Service discovery — how collectors find 200,000 short-lived targets.
• Health checks done right — the up metric, and why silence is not health.
• Write-ahead logging — crash safety for the in-memory head and the agents’ queues.
• Search data structures — inverted indexes from label pairs to series.
• Consistent hashing — the ring that places series on ingesters.
• Quorums and leaderless replication — three replicas, two acknowledgements.
• Blob storage (S3) — the cheap, durable home for immutable blocks and rollups.
• Count-min sketch and HyperLogLog — estimating cardinality and heavy hitters cheaply.
• Gossip protocol — how Alertmanagers agree on who already paged.
• Noisy neighbours and fairness — per-team limits and isolation.
• Multi-region and active-active — regional autonomy with a global view.
• Design a distributed logging system — the sibling pipeline for per-event data.
• Design a notification system — delivering the page once the Alertmanager decides to send it.

References and further reading

1. Gorilla: A Fast, Scalable, In-Memory Time Series Database (Pelkonen et al., VLDB 2015)vldb.orgDelta-of-delta timestamps and XOR value encoding; 1.37 bytes per sample instead of 16; 96% of timestamps and 51% of values in one bit; two-hour blocks; 2 billion series at Facebook; 73× faster queries than the HBase store.
2. Prometheus documentation: Storageprometheus.ioTwo-hour blocks of chunks plus index; an in-memory head protected by a WAL in 128 MB segments; compaction up to 10% of retention or 31 days; 1–2 bytes per sample; local storage is not clustered or replicated.
3. Prometheus documentation: Instrumentation best practicesprometheus.ioEvery label set is a new series with RAM, CPU, disk and network cost; keep cardinality below about 10 for most metrics and never label by user ID or email.
4. Prometheus documentation: Histograms and summariesprometheus.ioSummary quantiles cannot be aggregated across replicas; bucketed histograms can; each classic bucket is its own series; native histograms with exponential buckets are preferred.
5. Prometheus Alertmanager (README and example configuration)github.comDeduplication, grouping, routing, inhibition and silences; group_wait 30s, group_interval 5m, repeat_interval 3h in the example; HA clusters that gossip (200 ms default) and must each receive every alert.
6. Planning Grafana Mimir capacitygrafana.comSizing ratios: 1 core per 25,000 samples/s for distributors; 1 core and 2.5 GB per 300,000 in-memory series for ingesters; replication factor 3 by default; 2 bytes per sample for compacted blocks; 50% memory headroom.
7. Thanos documentation: Compactor and downsamplingthanos.io5-minute downsampling for blocks older than 40 hours and 1-hour after 10 days; downsampled chunks keep count, sum, min, max and counter; deduplicating replicated blocks.
8. M3: Uber’s open source, large-scale metrics platform for Prometheus (Uber Engineering)uber.com6.6 billion series; 500 million metrics/s aggregated to 20 million/s persisted; quorum writes to three replicas per region; per-metric retention policies; compaction overhead in Cassandra; regional writes with global query fan-out.
9. Alerting on SLOs (The Site Reliability Workbook, Google)sre.googlePrecision, recall, detection and reset time; why duration clauses fail; burn rates; multiwindow, multi-burn-rate thresholds of 14.4× over 1 hour, 6× over 6 hours and 1× over 3 days.
10. Monarch: Google’s Planet-Scale In-Memory Time Series Database (Adams et al., VLDB 2020)vldb.orgAutonomous regional zones under a global query layer; in-memory data to avoid circular dependencies on storage systems it monitors; availability over consistency; clients push; collection aggregation with an admission window.