Availability and the nines

Availability is the share of time, or of requests, in which your system does its job correctly and fast enough. The “nines” are shorthand for that share: 99.9% leaves about 43 minutes of downtime a month, 99.99% about 4. The real number is set by your dependencies, your recovery time and what your replicas secretly share.

Picture it first: the ATM outside your office

Ask three people whether the ATM near your office was available today. The bank’s dashboard says it was powered on all day. You say it was down: at 1 pm it showed “unable to process” twice. Your colleague says it was fine; she withdrew cash at 11.

To hand you cash, the ATM needs power, its network link, the bank’s core system and the card network all working at once. If any one is out, you get nothing: serial composition.

So the bank installs a second ATM next to the first. Now you are stuck only if both are broken at once: parallel composition. But both machines share one power line, one router and the same nightly software update. A bad update kills both, so the second machine bought far less than it seemed: a correlated failure.

Finally, how long you are stuck depends on how fast the bank notices, diagnoses and fixes the fault. A machine that breaks monthly but is back in 5 minutes loses 60 minutes a year; one that breaks twice a year for a day each time loses 2,880. That is MTBF and MTTR.

Map it back: the ATM is your service, a withdrawal is a request, core banking is a hard dependency, the second machine is a replica, and the shared power line is a failure domain such as an availability zone (AZ).

What availability measures

AWS’s Well-Architected guidance calls a workload available when it performs its agreed function successfully when required: write down the function, count only correct answers, and measure when users want the service.

What counts as “down”

A useful default:

• Server-caused errors count: 5xx responses, crashes, dropped connections. Client errors such as a 404 do not; AWS’s formula counts only valid requests.
• Too slow counts. An answer after the app gave up is a failure, even if the server logged a 200. Set a threshold per operation.
• Wrong answers count, and can be worse than none. Google’s SRE book notes that if users could see each other’s private contacts, taking the service down is the right call.
• Planned maintenance counts. AWS advises against excluding it: users want the service then too.
• Partial outages are the normal case: one shard, region, API or feature fails while the rest works.

Time-based or request-based

• Time-based: uptime ÷ (uptime + downtime). It matches how people talk (“we were down for 20 minutes”) but needs a rule for when a minute counts as down.
• Request-based: successful valid requests ÷ total valid requests. Google’s SRE book prefers this for global services, which are rarely cleanly up or down because some traffic is always served somewhere. At 10 million requests a day and 99.95%, you may fail 0.0005 × 10,000,000 = 5,000 requests a day.

The two can tell very different stories about one incident:

Most cloud SLAs use the hybrid in the middle: score each one- or five-minute window, call it “down” below a threshold, and add up the down windows. AWS’s whitepaper example: 27 five-minute windows in a year below 95% is 135 minutes, about 99.97% (135 ÷ 525,600 ≈ 0.026%).

Two traps catch almost everyone:

• Summing hides small APIs. A UPI-style app’s balance check serves 2,000,000 requests at 99.95%; its add-bank-account API serves 1,000 at 80%. Summed: (1,999,000 + 800) ÷ 2,001,000 = 99.94%. Averaged per API: 89.98%. Only the second shows that new users cannot link an account, so measure per API or per journey.
• Server-side numbers miss what never arrives. If DNS or the load balancer breaks, your servers log nothing. And in a bad outage clients back off or give up, so failures drop. Add client-side canaries (synthetic probes) and count bad minutes.

Objectives and error budgets built on these measures live in SLIs, SLOs and error budgets.

The nines, derived

“Nines” count the leading 9s: 99.9% is three nines. The useful number is the unavailability 1 − A, your downtime budget. Each extra nine divides it by ten.

Downtime budget = minutes in the period × unavailability:

• Year (365.25 days): 365.25 × 24 × 60 = 525,960 minutes. At 99.9%: 0.001 × 525,960 ≈ 526 minutes = 8 h 46 min. At 99.99%: 52.6 minutes.
• 30-day month: 30 × 24 × 60 = 43,200 minutes. At 99.9%: 43.2 minutes; at 99.99%: 4.32 minutes.
• Day: 86,400 seconds, so 99.99% leaves 0.0001 × 86,400 = 8.64 seconds.

Google’s SRE book uses a 365-day year, so it prints 8.76 hours and 52.56 minutes; the gap is about 22 seconds a year at three nines. Cloud SLAs usually measure the billing month, 28 to 31 days.

So at 99.9% a month, an incident a person fixes in half an hour fits. At 99.99%, an incident where a human must wake up and decide is already over budget. As the Calculus of Service Availability puts it, by the time a human is online to trigger a failover, you have probably spent your error budget.

Incident frequency × duration sets a hard ceiling too. One incident a month at 15 minutes each to detect and roll back is 12 × 15 = 180 minutes a year: at most 1 − 180 ÷ 525,960 ≈ 99.966%, whatever your architecture diagram says.

Serial composition: dependencies multiply

If a request needs components 1 to n all working, and they fail independently, the chance that all are up at once is the product:

A_serial = A1 × A2 × … × An

Take a checkout during a Big Billion Days-style sale: load balancer (99.99%), app tier (99.95%), orders database (99.95%) and a third-party payment gateway (99.9%). 0.9999 × 0.9995 × 0.9995 × 0.999 ≈ 0.9979, or 99.79%: 0.0021 × 43,200 ≈ 91 minutes of downtime a month, worse than the weakest link alone.

When every availability is close to 100%, unavailabilities add: 0.01% + 0.05% + 0.05% + 0.1% = 0.21%, so 99.79%. In minutes: 4.3 + 21.6 + 21.6 + 43.2 ≈ 91 a month. Use this version at a whiteboard.

Three things follow:

• Fewer hard dependencies, higher ceiling. AWS’s whitepaper makes it a rule: reducing dependencies can improve availability.
• Turn hard dependencies into soft ones, whose failure you can absorb: hide the recommendations widget, serve the last-known delivery estimate from a cache, send the confirmation SMS through a queue. A soft dependency drops out of the product.
• Treat the product as an estimate. AWS notes that dependencies often beat their published SLAs, so multiplying SLAs gives a conservative order of magnitude; your own deploys and bugs usually dominate.

The rule of the extra nine

Treynor and colleagues turn this into the rule of the extra 9: each critical dependency should offer one more nine than the service that depends on it, or be about 10× as reliable, so its contribution is noise.

Here is why. Say checkout targets 99.95%, a budget of 0.05% (21.6 minutes a month). Keep half for your own bugs and deploys and split the rest across five critical dependencies: 0.025% ÷ 5 = 0.005% each. Each must deliver 99.995%, roughly one extra nine.

Two refinements from the same paper:

• Count unique dependencies once, however deep they sit. An auth service called by three of your dependencies is still one critical dependency; second-level dependencies don’t need two extra nines.
• If a dependency can’t give you the extra nine, make it soft (cache, fail open, degrade), add an independent second instance (a second payment gateway behind a router), or lower your own promise.

Parallel composition: what redundancy buys

If any one copy can serve, the system fails only when every copy is down at once. With independent failures, multiply the unavailabilities:

A_parallel = 1 − (1 − A1) × (1 − A2) × … × (1 − An)

Two independent replicas at 99.9%: 1 − 0.001 × 0.001 = 99.9999%, about 32 seconds a year. AWS’s shortcut: when every number is all 9s, redundant copies add their nines (3 + 3 = 6).

Returns diminish fast. In AWS’s whitepaper, a 99.5% subsystem with two spares is already 1 − 0.005³ = 99.9999875%, about 4 seconds a year; going to 10 spares multiplies the sparing cost by 5 to save those 4 seconds. Beyond three spares, components that are at least 99% available gain nothing users notice.

Mind the unit of failure, too. Ten instances in one AZ are not redundant against losing that AZ. If you need 10 at peak, AWS compares 2 AZs × 10 (20 instances) with 3 AZs × 5 (15). Both keep 10 after losing an AZ, but the second needs 1.5× peak capacity instead of 2×. Pre-provisioned headroom like this is static stability: survivors carry the load without launching anything mid-incident.

The formula hides four assumptions, and each fails in practice:

1. Failures are independent. Usually false; see the next section.
2. Failover is instant. A load balancer configured to health-check every 5 seconds, with 2 failures to mark a target unhealthy, needs about 10 seconds to stop sending it traffic; promoting a database replica takes longer (see failover and health checks done right).
3. The survivor has capacity. If one node cannot carry peak alone, losing its twin becomes an overload outage.
4. The switch is in series. The load balancer or failover controller multiplies the pair: 0.9999 × 0.999999 ≈ 99.99%. The switch, not the pair, sets the number.

Correlated failures: where the math lies

In production, “independent” replicas share a lot: the AZ (power, cooling, network), the binary from the same deploy, the same config push, the same bug triggered by the same input (a leap second, a malformed message, a certificate expiring on the same day), the same DNS and auth services, and often the same operator.

Model shared fate with c, the fraction of time a common cause takes both copies down:

A_pair ≈ 1 − (c + (1 − A1) × (1 − A2))

If a bad config push that reaches both replicas costs 20 minutes a month, c = 20 ÷ 43,200 ≈ 0.046%, and the pair is 1 − (0.00046 + 0.000001) ≈ 99.95%. The independent part is noise; the shared part is the whole story.

Fleet data backs this up. Ford and colleagues studied a year of failures in Google’s storage fleet (OSDI 2010): 37% of node failures were part of a burst of at least two nodes, often tied to a rack, switch or power domain, and models assuming independence typically overestimated availability by at least two orders of magnitude. Two copies in each of two data centres outlasted four copies in one.

Shared config is the most dramatic case. On 2 July 2019, Cloudflare pushed one new firewall rule whose regular expression backtracked badly. Rule changes could go out globally in one step, so it exhausted the HTTP-serving CPUs across the whole network at once: about 27 minutes of 502 errors. Many cities, one shared change.

How you claw independence back:

• Spread across failure domains: AZs for power and hardware, regions for regional events, cells so one bad tenant or push reaches a slice of users.
• Stagger every change, config included, one cell or zone at a time with automatic rollback (deployment strategies).
• Keep the recovery path off shared dependencies. If failover needs the control plane, auth or DNS that just broke, the spare is useless.
• Add diversity where stakes justify it: two payment gateways, two DNS providers, paid for in integration and reconciliation work.

Hunting down shared components layer by layer is the subject of single point of failure.

MTBF, MTTR and the cheapest nine

Availability depends on how often you fail and how long each failure lasts:

A = MTBF ÷ (MTBF + MTTR)

MTBF (mean time between failures) is the average stretch of normal running. MTTR (mean time to repair or recover) is how long a failure lasts as users experience it: detect (MTTD) + diagnose + repair or mitigate. AWS’s reliability pillar example: MTBF 150 days (3,600 hours), MTTR 1 hour, so 3,600 ÷ 3,601 ≈ 99.97%.

A worked example. A Swiggy-style order service has about one incident a month: MTBF ≈ 43,200 minutes. An incident takes 10 minutes before an alert fires, 20 to find that the last deploy did it, and 15 to redeploy the old build. MTTR = 45 minutes, so A = 43,200 ÷ 43,245 = 99.90%.

The team adds a synthetic probe that pages within 2 minutes, deploy markers on dashboards so the cause is obvious in 3, and automatic rollback that finishes in 5. MTTR = 10 minutes, so A = 43,200 ÷ 43,210 = 99.98%.

Getting there by failing less often instead needs MTBF ÷ (MTBF + 45) = 99.977%, so MTBF ≈ 135 days: 4.5× fewer incidents. Preventing nearly four incidents in five is far harder than recovering 4.5× faster.

Why is recovery usually the cheaper lever?

• Most incidents follow a change. Google’s SRE book reports that roughly 70% of outages are due to changes in a live system, and undoing a change doesn’t require understanding the bug. AWS’s whitepaper makes it a rule: mitigate impact first, resolve the problem later.
• Detection is a monitoring problem, not a redesign. Alerting on user-facing symptoms (errors, latency) from outside the system shrinks MTTD without touching the architecture; see observability.
• Routing around failure makes MTTR roughly equal to MTTD. With redundancy and fast health checks, “repair” means “stop sending traffic there”. AWS’s restart table shows why: about 15 minutes to launch a new virtual server, 2 seconds to start a container.

The Calculus of Service Availability names three levers: make outages rarer, smaller (sharding, cells, geographic isolation) and shorter. Smaller counts directly in a request-based measure: an outage in one cell of five costs a fifth as much.

MTBF and MTTR are averages, and AWS cautions that software failures don’t follow a predictable wear-out curve. Use the formula to reason about levers; measure what you actually deliver.

Cloud SLAs and composite SLAs

An SLA is a contract: a promised level, how it is measured, and what the provider pays if it misses. An SLO is your internal target, usually stricter. Azure’s guidance keeps them deliberately separate.

Read a real one. The Amazon Compute SLA for EC2 (last updated May 2022) says:

• Region-level: 99.99% monthly uptime, only if your instances run across two or more AZs. “Unavailable” means all of them have no external connectivity at the same time; uptime is 100% minus the share of Unavailable minutes in the month.
• Instance-level: 99.5% for a single instance.
• Credits, not refunds: 10% of that Region’s monthly EC2 bill below 99.99%, 30% below 99.0%, 100% below 95.0%, applied only to future EC2 payments. You must claim, with request logs as evidence, by the end of the second billing cycle after the incident.
• Exclusions: force majeure, Internet problems outside AWS, and failures caused by your own software or actions.

Run the numbers. You spend $20,000 a month on EC2 in one Region, and a bad month delivers 99.9%: 43 minutes with every AZ unreachable. That earns a 10% claim: a $2,000 credit. If those 43 minutes hit a sale-day peak when checkout takes ₹2 crore an hour, you lost about 2 × 43 ÷ 60 ≈ ₹1.4 crore of orders.

Composite SLAs follow the serial rule: multiply everything on the path. Azure’s example: 99.95% × 99.99999% ≈ 99.95%. With illustrative SLAs of 99.99% for compute, 99.95% for a managed database and 99.99% for a load balancer, the composite is 0.9999 × 0.9995 × 0.9999 ≈ 99.93%, before your own code has a single bug.

Two cautions:

• Scope. Azure notes that the App Service SLA doesn’t cover features like deployment slot swaps, and a Front Door SLA applies only to a narrow, specified probe scenario.
• Deployment shape. EC2’s 99.99% needs multiple AZs; one instance gets 99.5%.

What each extra nine costs

Each nine divides the downtime budget by ten, so everything that eats the budget must get about ten times better: faster detection and rollback, more reliable dependencies, redundancy across larger failure domains. Hence the rule of thumb that each nine costs roughly an order of magnitude more; Google’s SRE book warns that one increment may cost 100× the previous one, in redundant capacity and in engineers building reliability instead of features. AWS’s reliability pillar adds exhaustive failure testing, automated recovery, slower releases and fewer usable dependencies.

So price a nine before buying it, using the SRE book’s method: multiply revenue by the availability gained. A grocery checkout taking ₹20 crore a month that moves from 99.9% to 99.99% protects at most 0.0009 × ₹20 crore = ₹1.8 lakh a month, less once you count users who retry. If that nine costs two engineers for a year plus a second region, revenue alone won’t justify it; trust, regulation or contracts might.

Users can’t perceive nines beyond their own connection, either. The SRE book notes that a user on a 99% reliable phone cannot tell 99.99% from 99.999%, and Google measured typical ISP background error rates between 0.01% and 1%.

When you do need another nine, these are the usual purchases:

The patterns behind these rows are collected in the reliability patterns cheat sheet and fault tolerance.

Choosing a target per feature

One number for the whole system is almost always wrong. Journeys differ in what a failure costs and what they depend on, so:

1. List the critical journeys, not services: search, add to cart, pay, track the order.
2. Price a failure for each: lost revenue, retries, support load, trust, regulation.
3. Check feasibility: multiply the hard dependencies, apply the extra-nine rule, test incidents × MTTR against the budget.
4. Write it as an SLO with a window and a definition of failure, and publish a looser SLA if you need one (SLIs, SLOs and error budgets).

AWS’s reliability pillar gives reference tiers: 99% for batch and ETL, 99.9% for internal tools, 99.95% for online commerce and point of sale, 99.99% for video delivery and broadcast, 99.999% for ATM transactions and telecoms. Typical starting points (not standards):

Cheat sheet

How to say it in the interview

“I’d define availability per journey as the fraction of valid requests that succeed within the latency target, measured at the load balancer and by external probes over 30 days. For checkout I’d target 99.95%, about 22 minutes a month. Its four hard dependencies multiply to roughly 99.8%, so I’d make notifications and recommendations soft, run the database across three zones, and add a second payment gateway behind a router. Redundancy only pays if failures are independent, so changes roll out one zone at a time with automatic rollback. And with 22 minutes a month, recovery must be automatic.”

Your dependencies are 99.9%. Can you offer 99.99%? Not if they are hard and in series: one 99.9% dependency alone uses 43 minutes a month against your 4.3. Make it soft with a cache, queue or fallback, put two independent instances behind a router (99.9999% on paper, less after shared causes), or lower your target.

Time-based or request-based? Request-based for SLOs and alerting, because it weights partial outages by the traffic they hurt. Add bad-minute counting and external probes, because request counts look healthier when clients stop calling.

Are two replicas enough? Two independent 99.9% replicas give six nines on paper, but shared AZs, deploys or config pushes dominate the real number, and the survivor must carry full load. Spread them across failure domains and stagger changes; use three if one may be down for maintenance when another fails.

What does a 99.99% cloud SLA actually promise? A service credit, not uptime. Under EC2’s Region-level SLA, if all your instances across two or more AZs lose connectivity long enough to drop below 99.99% in a month, you can claim 10% of that Region’s EC2 bill (30% below 99%, 100% below 95%) against future bills, with evidence.

Why not aim for five nines everywhere? That is about 26 seconds a month, so every dependency needs roughly six nines and every recovery must be automatic. Cost rises about tenfold per nine, while mobile users often can’t tell the difference. Spend nines where failure is expensive, such as payments.

Where to go next

• Reliability: SLIs, SLOs and error budgets: turning a target into an objective, a budget and alerts.
• Single point of failure: finding the shared components that break independence.
• Failover: how fast the switch really is, and its data-loss cost.
• Fault tolerance: keeping faults from becoming failures.
• Reliability patterns cheat sheet: timeouts, retries, circuit breakers, bulkheads.
• Health checks done right: how detection time is set.
• Cell-based architecture: shrinking the blast radius to 1/N of users.
• Multi-region and active-active and disaster recovery: surviving a whole region.

References and further reading

1. Site Reliability Engineering, chapter 3: Embracing Risk (Google)sre.googleTime-based versus request-based (aggregate) availability, why each increment of reliability costs far more than the last, the 99%-smartphone argument, ISP background error rates and the revenue value of a nine.
2. Site Reliability Engineering, Appendix A: Availability Table (Google)sre.googleCross-check for the nines table (it uses a 365-day year) and the case for aggregate metrics when outages are partial.
3. The Calculus of Service Availability (Treynor, Dahlin, Rau and Beyer, ACM Queue / CACM, 2017)cacm.acm.orgAvailability as MTTF / (MTTF + MTTR), the rule of the extra 9, counting unique dependencies once, the three levers, and the 2-minute alert plus 5-minute response allowances.
4. Availability and Beyond: Understanding and Improving the Resilience of Distributed Systems on AWS (whitepaper)docs.aws.amazon.comMTBF, MTTD and MTTR; hard dependencies and spares, with the diminishing returns of sparing and the AZ example; restart times; and, in “Measuring availability”, summing versus averaging APIs, bad five-minute windows, and clients that stop calling.
5. Site Reliability Engineering, chapter 1: Introduction (Google)sre.googleThe finding that roughly 70% of outages are due to changes in a live system, and the progressive-rollout, detection and rollback practices that follow.
6. AWS Well-Architected Reliability Pillar: Availabilitydocs.aws.amazon.comApplication categories per availability tier, the serial and redundant formulas with worked numbers, the MTBF/MTTR estimate, planned downtime and why higher tiers cost more.
7. Amazon Compute Service Level Agreement (EC2)aws.amazon.comA real SLA: the 99.99% Region-level and 99.5% instance-level commitments, the definition of Unavailable, service-credit tiers, claim rules and exclusions.
8. Azure Well-Architected Framework: Architecture strategies for defining reliability targetslearn.microsoft.comComposite SLOs as a product of the contributing factors, SLA versus SLO, and why a provider SLA covers a narrower scope than the whole service.
9. Availability in Globally Distributed Storage Systems (Ford et al., OSDI 2010)static.googleusercontent.comFleet data on correlated failure bursts and how models that assume independence overestimate availability by orders of magnitude.
10. Details of the Cloudflare outage on July 2, 2019 (Cloudflare blog)blog.cloudflare.comA single rule pushed globally took down every HTTP-serving CPU at once; the timeline shows detection, diagnosis and repair minutes.