Reliability: SLIs, SLOs and error budgets

Reliability is whether your service does the right thing, correctly and fast enough, as your users experience it. You measure it with SLIs (the share of events that went well), commit to a target with an SLO, and treat the gap to 100% as an error budget you may spend on change. An SLA is the looser, contractual promise underneath, with a penalty attached.

Picture it first: the tiffin service’s promise

A tiffin service delivers 1,000 lunches a day to offices across the city. Customers don’t care how many scooters are running. They care that the right lunchbox reaches their desk, still warm, by 1:15 pm.

So every evening the owner counts lunches delivered right and on time ÷ lunches ordered. That ratio is her indicator.

She decides 99 in 100 is good enough over a month; a perfect record would need two riders on every route. That target is her objective. It lets her get 300 of 30,000 monthly lunches wrong, an allowance she spends on purpose: a faster route, a new rider. That is her error budget.

When monsoon floods eat the allowance by the 20th, her rule kicks in: no new routes until the numbers recover. That is an error budget policy.

Corporate clients have a contract: below 97 in 100, the month’s bill drops by 10%. That SLA is looser than her own 99, so she hears about trouble before it costs money. And she wants to know by 12:30 that today’s Andheri route is going badly: burn-rate alerting.

Map it back: lunches are requests, “right, warm, on time” defines a good event, the month is your SLO window, and the experiments are your deploys.

Reliability and the four terms

Availability asks whether the service answered. Reliability asks whether it did its job: the right answer, fast enough, as the user experienced it. A service can be up and still unreliable:

• a wallet API that returns 200 OK with a balance ten minutes stale;
• a search that answers in 9 seconds, after the app gave up at 5;
• a restaurant list that silently falls back to yesterday’s cached menu.

So reliability is a bundle of indicators: availability, latency, correctness, freshness. The nines table, dependency math and MTTR live in availability and the nines; this article turns a target into a working system.

Keep four terms separate:

• SLI (indicator): a measurement, the share of valid events that were good.
• SLO (objective): a target for an SLI over a window.
• Error budget: 1 − SLO, the failure you are allowed. At 99.95%, one request in 2,000.
• SLA (agreement): a contract that names a consequence for missing a level, set looser than the SLO.

SLIs: good events over valid events

Google’s SRE workbook recommends one shape for nearly every SLI:

SLI = good events ÷ valid events × 100%

It runs from 0% (nothing works) to 100% (nothing broken), the error budget falls out as 100% minus the SLO, and every SLI feeds the same tooling. Writing one down means answering three questions:

• Which events are valid? Real user requests; not health checks, load tests or your own probes. Client errors such as 404 normally count as good, since the service correctly refused a bad request; the workbook’s example and CloudWatch count every non-5xx as a success. But a 503 because you shed load is your failure.
• What makes an event good? No 5xx, and under the latency threshold if this SLI covers latency, and not degraded if quality matters.
• Count requests or minutes? A request-based SLI (OpenSLO’s Occurrences) weights by traffic, so a 3 am failure counts less than a 9 pm one. A time-based SLI (Timeslices; CloudWatch says period-based) marks each minute good or bad and counts good minutes. Request-based tracks user pain more faithfully; time-based copes better with low traffic and matches how SLAs are written (both, on one outage).

The SLI menu

You rarely need to invent an SLI. Google’s SRE books group them by what the system does:

Pick five or fewer SLI types per service, as the workbook advises; the rest belong on dashboards.

Where to measure an SLI

One SLI specification (“order-tracking requests answered within 500 ms”) can be implemented at several points between user and database. Google Cloud compares them on fidelity (closeness to what the user felt), coverage and cost.

• Server metrics are cheapest and richest in labels, but a crashed process logs nothing and time queued in the load balancer never shows up.
• Load balancer logs see every request that reaches you, including 502s when backends vanish; the workbook’s worked example uses them because they already exist and sit closer to the user. They miss DNS, CDN and network failures.
• Synthetic probes run every minute regardless of traffic, so they catch a broken front door (an expired certificate, a bad CDN rule) and give quiet services a signal, but only on scripted paths.
• Client-side timing (real-user monitoring, RUM) is the truest picture, but it includes what you can’t fix, such as a weak 2G signal: a good proxy for happiness, a poor trigger for a page.

Writing an SLO

A complete SLO names the SLI, target, window and measurement point: “99.95% of valid pay requests get a non-5xx answer, measured at the load balancer, over a rolling 28 days.” Leave a part out and two people will compute two different numbers.

Rolling or calendar window

• Rolling (the last 28 days, recomputed continuously) matches how users remember: an outage on the 31st is not forgotten on the 1st. The workbook suggests whole weeks, so every window holds the same number of weekends; both it and Google Cloud recommend starting at four weeks.
• Calendar (a month or quarter) matches billing, SLAs and planning, but a request-based budget’s size is uncertain until the period ends, and it resets on the 1st whatever users think.

A common split: rolling 28 days for operating and alerting, calendar periods for reports and contracts.

Latency SLOs

“p99 under 300 ms” and “99% of requests complete within 300 ms” name the same threshold, but the second makes a better SLI: it gives a budget directly (1% may be slow) and adds up across servers by summing counts. Percentiles don’t add up; the Prometheus documentation calls averaging per-server percentiles statistically nonsensical (percentiles and tail latency).

Set two grades for the typical case and the tail, each with its own budget; the workbook’s example is 90% under 100 ms and 99% under 400 ms. And label latency by outcome (success or error), or a flood of instant 500s makes your latency SLI look better.

Why not 100%?

• Users can’t tell. Someone on a 99%-reliable phone cannot tell 99.99% from 99.999%, as the SRE book puts it.
• 100% forbids change. Google’s example policy attributes roughly 70% of its outages to changes.
• Each nine costs more than the last, and your dependencies aren’t perfect either (what each nine costs).
• Don’t overachieve, either. Users build on what you deliver. Google’s Chubby lock service was so reliable that teams assumed it never failed, so its SREs began taking it down on purpose.

Choosing a starting number

1. Pick the critical user journeys and write an SLI specification for each.
2. Look at the last four weeks. Say the pay API delivered 99.968% success, a p90 of 412 ms and a p99 of 1,180 ms. Round availability down and latency up, as the workbook does: 99.95%, 90% under 450 ms, 99% under 1,200 ms.
3. Check it against users, not just history. The SRE book warns that copying current performance can lock you into heroics; ask below what level users complain. Tightening a loose target is easy; relaxing a promise is painful.
4. Get three signatures: product (good enough for users), developers (will act when the budget is gone) and on-call (defensible without heroics). The owner is whoever can trade features for reliability, often the product owner.
5. Review monthly at first, then quarterly, and track an aspirational SLO if users need better than you can deliver yet.

Error budgets in minutes and requests

The error budget is 1 − SLO. Read it two ways.

In time. A 30-day window has 30 × 24 × 60 = 43,200 minutes. At 99.9%, 0.001 × 43,200 = 43.2 minutes of full outage, or the equivalent in partial ones (the full table).

In requests. 99.9% means one bad request in 1,000. At 50 million requests a month, that is 50,000,000 × 0.001 = 50,000 failed requests to spend.

Now price some incidents. With steady traffic, 50,000,000 ÷ 43,200 ≈ 1,157 requests arrive per minute:

• A 10-minute total outage: 10 × 1,157 ≈ 11,574 failures, 23% of the budget; at a 9 pm peak of three times the average, about 69%.
• A bad deploy failing 2% of requests for 3 hours: 0.02 × 1,157 × 180 ≈ 4,167, 8%.
• One shard of five down for 30 minutes: 0.2 × 1,157 × 30 ≈ 6,944, 14%. Partial outages are charged only for the users they hurt.

Each objective has its own budget. “99% of requests under 400 ms” adds 1% × 50,000,000 = 500,000 slow requests, tracked separately.

Budgets also tell you what to fix. In the workbook’s game example, a 97% SLO allowed 109,897 failed requests in four weeks. A bad release cost 14,066 (13%) and a 20-hour database restore about 72,000 (65%). But with two or three bad releases a year against one server failure in five years, releases cost about twice as much budget over time: release safety was the better investment.

The dashed diagonal is a steady burn rate of 1, on pace to reach zero on day 30. Above it you are ahead; below it you are spending too fast.

The error budget policy

A budget with no consequence is just a KPI. The policy is a short written agreement, approved by product, development and on-call, that says what happens and who decides. The workbook’s example is a good template:

• In budget: releases follow the normal process.
• Budget exhausted over the trailing four weeks: halt all changes except P0 bugs and security fixes until the service is back within SLO.
• Reliability work is mandatory if the team’s own code or process caused the miss.
• One incident over 20% of the four-week budget: a postmortem with at least one P0 action item.
• Exceptions: a company-wide network outage, another team’s service (if that team has frozen its own releases), or load-test traffic.
• Escalation: disagreements go to one named person, the CTO in Google’s example.
• Non-goal: it is not a punishment; it gives the team permission to put reliability first.

The response can be graduated: the SRE book mentions slowing releases as the budget nears zero. And before a Big Billion Days-style sale, many teams freeze changes regardless of budget; write that down as its own rule.

Two gaps teams often leave open:

• Misses caused by a dependency. One school says don’t freeze, it wasn’t you; the other freezes anyway, because fewer changes means fewer outages. The workbook notes the second makes users happier. Choose and record it.
• Budget left over. Ending every month with 90% unspent means over-investing in reliability. The workbook’s decision matrix: SLO met, little toil and happy users, release faster; SLO missed but users happy, loosen it; SLO met but users unhappy, tighten it.

Alerting on burn rate

You want a page when something eats a significant share of the budget, and silence otherwise; the workbook scores rules on precision, recall, detection time and reset time. The obvious rules fail. Paging when the 10-minute error rate tops 0.1% fires on blips worth 10 ÷ 43,200 = 0.02% of the budget. Adding for: 1h means a total outage pages after an hour, having burned 60 ÷ 43.2 = 139% of the budget.

Instead, alert on how fast you are spending. Burn rate = observed error rate ÷ (1 − SLO). At 99.9%, a 0.1% error rate is a burn rate of 1: the budget lasts exactly the window. A total outage burns at 1,000 and empties a 30-day budget in 30 × 1,440 ÷ 1,000 = 43.2 minutes. Two formulas do all the work:

• Time to empty = SLO window ÷ burn rate.
• Budget spent in an alert window = burn rate × alert window ÷ SLO window.

So one hour at burn rate 14.4 spends 14.4 × 1 h ÷ 720 h = 2% of a 30-day budget. Turn it around to pick thresholds: burn rate = share of budget × SLO window ÷ alert window. 5% in 6 hours gives 0.05 × 720 ÷ 6 = 6; 10% in 3 days gives 0.10 × 720 ÷ 72 = 1. With a 28-day window, 2% in an hour is 0.02 × 672 = 13.44, the figure in AWS’s CloudWatch documentation.

Multiwindow, multi-burn-rate alerts

One burn rate isn’t enough: a 1-hour rule at 14.4 never notices a steady 3× burn that empties the budget in 10 days. So the workbook combines several, starting from these for a 99.9% SLO:

The logic: if a problem will empty the budget within hours or days, wake someone; if it will take weeks, a ticket will do. Detection scales with severity: a total outage trips the 1-hour rule after 0.001 ÷ 1 × 60 min × 14.4 ≈ 52 seconds, while a 1.5% error rate takes about 58 minutes. Either way, 2% of the budget is gone when the page arrives.

Why the short window? A long window gives precision but has a long memory: after the fix, the 1-hour average stays high for up to an hour and the alert keeps firing. So each rule also checks a window one-twelfth as long and fires only when both exceed the threshold: “is it still burning right now?”

The 1-hour average rises by 10% ÷ 60 per bad minute, so it crosses 1.44% after 1.44 × 60 ÷ 10 = 8.6 minutes, when exactly 2% of the budget is gone. After the rollback at minute 15, the 5-minute average drops below the line at 19.3 minutes and the page clears; the 1-hour average alone would stay above it until 66 minutes. The incident cost 15 × 10% ÷ 43.2 ≈ 3.5% of the budget. Since one meltdown can trip every rule at once, suppress lower-severity alerts while a page is open.

Journeys, dependencies and an example SLO

Write SLOs for what users do, not for each microservice. The workbook calls these critical user journeys: search, add to cart, pay. Measure one by joining log events, with client-side instrumentation or with a scripted synthetic journey.

Dependencies need SLOs too. A critical dependency (when it is down, you are down) needs an SLO at least as high as the journey that relies on it. If it can’t get there, engineer around it with a cache, store-and-forward or graceful degradation, and don’t assume two zones turn 99.9% into 99.9999% (fault tolerance).

Grade by importance: the workbook gives premium customers 99.99% and free ones 99.9%.

Here is an illustrative SLO set for a UPI-style pay API, all over a rolling 28 days:

At 40 million pay requests per 28 days, availability allows 0.0005 × 40,000,000 = 20,000 failed requests and correctness 400 mismatches, each still reconciled by hand; the SLO decides when engineering stops feature work to fix the cause. When the payer’s bank fails, the user’s payment still failed, so the completion SLI counts it; track the bank’s share as a separate dependency SLI.

The availability objective in the vendor-neutral OpenSLO format:

apiVersion: openslo/v1
kind: SLO
metadata:
  name: pay-api-availability
  displayName: Pay API availability
spec:
  service: payments
  description: Pay requests answered without a 5xx or timeout; business declines count as good.
  indicator:
    metadata:
      name: pay-api-good-ratio
    spec:
      ratioMetric:
        counter: true
        good:
          metricSource:
            type: Prometheus
            spec:
              query: sum(lb_requests_total{route="/v1/pay", synthetic="false", outcome!~"5xx|timeout"})
        total:
          metricSource:
            type: Prometheus
            spec:
              query: sum(lb_requests_total{route="/v1/pay", synthetic="false"})
  timeWindow:
    - duration: 28d
      isRolling: true
  budgetingMethod: Occurrences
  objectives:
    - displayName: 99.95% of real pay requests answered
      target: 0.9995

budgetingMethod: Occurrences makes it request-based; Timeslices would count good minutes instead.

Anti-patterns

• Too many SLOs. Forty objectives means none is defended. Group request types into a few classes instead; the workbook’s buckets are CRITICAL (99.99%), HIGH_FAST (99.9%, 90% under 100 ms, 99% under 200 ms), HIGH_SLOW (99.9%, 1 s and 5 s), LOW (99%) and NO_SLO. The SRE book’s test: if quoting an SLO never wins a priority argument, drop it.
• SLOs on causes. Users never feel CPU at 90%. Keep it on dashboards for diagnosis (observability); put SLOs on symptoms.
• A 100% target, or five nines everywhere: no budget, no change.
• An SLO nobody acts on. Without a signed policy, running out of budget changes nothing.
• Averaging latency. The SRE book shows a typical request at about 50 ms while 5% take 20 times longer; the mean hides it. Count requests under a threshold.
• Measuring only at the server. You miss everything upstream, and when clients give up, your error count falls as things get worse.
• Counting the wrong events. Health checks inflate the denominator; retries that eventually succeed hide the failure the user waited through; 4xx responses caused by your own bug count as good.
• Copying today’s performance or the SLA. Today’s 99.999% as the SLO promises heroics; an SLO equal to the SLA leaves no margin before money changes hands.

Cheat sheet

How to say it in the interview

“I define reliability from the user’s side. For a food-delivery order-tracking API, I’d pick three SLIs: availability, the share of tracking calls without a 5xx; latency, the share answered within 300 ms; and freshness, the share of reads where the rider’s location is under 10 seconds old. I’d measure at the load balancer, plus a synthetic probe for quiet hours. Targets: 99.9% available, 99% fast, 99% fresh, over a rolling 28 days. 99.9% leaves about 40 minutes, or one call in a thousand. While budget remains we ship daily behind canaries; if it runs out, the agreed policy freezes risky changes. I page on burn rate: 14.4× over an hour confirmed over five minutes, 6× over six hours, and a ticket for slow burns.”

What SLIs would you pick for this system? Start from the user journeys and use the menu: availability and latency for request paths, freshness and correctness for pipelines, durability for storage. Write each as good ÷ valid events, say where it is measured, and stop at about five.

Why not 100%? Users can’t perceive it through their own phones and networks, every change risks it, each nine costs far more than the last, and a 100% target leaves zero budget for change.

How do you alert on an SLO without paging for every blip? Alert on burn rate. Page when the 1-hour burn exceeds 14.4 (2% of a 30-day budget) or the 6-hour burn exceeds 6 (5%), each confirmed by a window one-twelfth as long so the page clears soon after the fix. Slow burns open a ticket.

What happens when the error budget runs out? Whatever the signed policy says: typically a freeze on all but P0 and security fixes until the service is back within SLO, reliability work first, and a postmortem for any incident over 20% of the budget.

How do SLOs change your design? 99.95% leaves 21.6 minutes a month, too little for a human to diagnose, so rollback must be automatic, changes ship as canaries and health checks must be accurate. “99% within 300 ms” becomes a timeout budget per hop; a 10-second freshness target rules out an hourly batch. Past four nines, only small blast radii, such as cells and 1% rollouts, can defend the budget.

Where to go next

• Availability and the nines: the downtime table, serial and parallel math, MTTR and SLA credits.
• Latency vs throughput vs bandwidth: percentiles and tail latency behind every latency SLO.
• Observability: logs, metrics and traces: the telemetry your SLIs are computed from.
• Deployment strategies: canaries and rollbacks that spend budget slowly.
• Health checks done right: fast, accurate detection inside the budget.
• Fault tolerance and the reliability patterns cheat sheet: techniques that keep incidents small.
• Cost-aware design: pricing a nine before you buy it.
• Ten principles senior engineers design by: measurable targets in the bigger picture.

References and further reading

1. Site Reliability Engineering, chapter 4: Service Level Objectives (Google)sre.googleDefinitions of SLI, SLO and SLA, the SLI families per system type, percentiles over averages, choosing targets, safety margins and Chubby’s planned outages.
2. Site Reliability Engineering, chapter 3: Embracing Risk (Google)sre.googleWhy 100% is the wrong target, the 99%-smartphone argument, and how an error budget aligns product and SRE incentives and makes teams self-policing.
3. The Site Reliability Workbook, chapter 2: Implementing SLOs (Google)sre.googleGood-over-total SLIs, the SLI menu per component type, measurement sources, rolling versus calendar windows, starter SLOs from history, the worked game-API budget, user journeys and dependencies.
4. The Site Reliability Workbook, chapter 5: Alerting on SLOs (Google)sre.googlePrecision, recall, detection and reset time; burn rate; the 14.4/6/1 multiwindow thresholds with 1/12 short windows; low-traffic and extreme-availability caveats; request-class buckets.
5. The Site Reliability Workbook, Appendix B: Example Error Budget Policy (Google)sre.googleA real policy: release freeze on exhaustion, the 20% postmortem rule, exceptions, escalation to the CTO, and the ~70% of outages caused by change.
6. SRE fundamentals: SLIs, SLAs and SLOs (Google Cloud blog)cloud.google.comWhy the SLA is set looser than the internal SLO, with the 99.9% versus 99.95% example.
7. Google Cloud Observability: Concepts in service monitoring, SLIscloud.google.comProperties of a good SLI, specification versus implementation, fidelity versus coverage versus cost per vantage point, the 28-day starting window, and separating fast errors from slow successes.
8. Amazon CloudWatch Application Signals: Service level objectivesdocs.aws.amazon.comA vendor view: period-based versus request-based SLOs, non-5xx responses counted as good, the burn-rate formula and the 13.44 threshold for a 28-day window.
9. OpenSLO specificationgithub.comA vendor-neutral YAML format for SLOs: ratio metrics, rolling or calendar time windows, and the Occurrences, Timeslices and RatioTimeslices budgeting methods.
10. Prometheus: Histograms and summariesprometheus.ioCounting requests under a latency threshold, and why averaging precomputed percentiles across instances is statistically meaningless.