One slow checkout leaves three records and one ID
The metric says something’s wrong, and how much. The span shows where the time went. The log line says why.
28M
series from one metric labelled by user_id
Route × status × pod gives 70,000 series. Add user_id for 2 million users: 400 times more, most written once. Per-user questions belong on spans and logs.
Average ten pods’ p99s and the dashboard says 190 ms
Pod 7 serves 10% of requests, all near 1 s, so the true p99 is ≈ 1 s. Sum histogram buckets, then compute once.
How do spans from two services join one trace?
Each hop sends a traceparent header: the same trace-id, and the caller’s span ID as parent. Across queues, you copy it yourself.
At 5,000 requests a second and 25 spans each: 125,000 spans a second. Keep them all?
Sample twice
Head-sample at the root to protect the pipeline; tail rules then keep every error and slow trace.
What should page someone?
Bad: CPU at 90%
A cause: it belongs on a dashboard.
Good: Checkout failing or slow
A symptom users feel. Page on SLO burn: 14.4× over 1 h and 5 min.
Page → cause in four hops
- RED: p99 2.1 s, CPU just 45%
- Only payment slowed, since a 19:35 deploy
- Exemplar trace: 1.79 s waiting for the pool
- Its logs:
pool_max=5, down from 50
The interview line
Metrics say something’s wrong, traces show where, logs say why, and one trace ID joins them. Page on SLO burn rate, not causes.