Health checks done right

A health check is a question that automation asks your instance on a timer, such as “should I send you traffic?” or “are you stuck?”, followed by an action: stop routing, restart, replace, or fail over a region. Done right, it pulls one broken server out of rotation in seconds. Done wrong, a check that leans on a shared database turns that database’s one-minute hiccup into a fleet-wide outage.

Picture it first: the dispatcher and the supplier

A food-delivery app works with dozens of cloud kitchens, and several people check on each kitchen for different reasons:

• The dispatcher asks, “Can you take another order?” A kitchen that says no stops getting orders but stays open.
• The shift manager looks for a cook frozen mid-task and replaces them, at the cost of twenty minutes of cold stove.
• The franchise owner rebuilds a kitchen whose gas line has failed.
• The city head sends orders to the next city if every kitchen here is dark.

Then the vegetable supplier, who serves every kitchen, is an hour late. If the dispatcher asks “has your supplier delivered?”, every kitchen says no, and the app shows “no restaurants open” although each has an hour of stock. If the shift manager’s rule is “no vegetables, replace the cook”, every kitchen swaps cooks at once, and nobody is warmed up when the truck arrives.

Map it back: kitchens are instances, the dispatcher is the load balancer, the shift manager is the kubelet running liveness probes, the franchise owner is the autoscaling group, the city head is DNS failover, and the supplier is a shared dependency such as the database. Each question was reasonable; asking it about something everyone shares caused the outage.

Four consumers, four questions

A health check is only as good as the action taken on its answer, and the four consumers act very differently.

• Load balancers and service discovery (an ALB, Envoy, a Kubernetes Service) decide whether the next request goes here. The next passing check undoes a removal.
• The orchestrator (Kubernetes’ kubelet) decides whether the process is wedged. A restart loses in-flight requests, caches and warm-up.
• The autoscaling group decides whether the machine is broken. EC2 Auto Scaling uses EC2 status checks by default and the load balancer’s verdict only if you turn it on.
• DNS failover (Route 53 health-checked records) decides whether an endpoint or region can serve. Clients follow only as their DNS caches expire.

Google’s SRE book draws the same line: process health (“is this binary responding at all?”) matters to the cluster scheduler, service health (“can it serve this class of requests right now?”) to the load balancer. Most failures in this article come from mixing the two up.

The rule of thumb: the more drastic and slow to undo the action, the narrower and more certain the check behind it must be.

Liveness, readiness and startup probes

Kubernetes makes the split explicit with three probes, all run by the kubelet against each container.

Startup probe: has the app finished booting? Until it succeeds, no liveness or readiness probes run; if it fails failureThreshold times in a row, the container is restarted. It lets a slow boot coexist with a strict liveness probe.

Liveness probe: is the process stuck in a way only a restart fixes, such as a deadlock? On failure the kubelet restarts the container. The docs warn that badly designed liveness probes cause cascading failures, restarting containers exactly when load is high. If your process already crashes when it breaks, you may not need one.

Readiness probe: should this pod get traffic right now? On failure the pod’s Ready condition turns false and its IP is removed from every matching Service. The container keeps running and being probed for its whole life, and rejoins once it passes.

Each probe uses one mechanism: httpGet passes on any status from 200 to 399 (the kubelet reads at most 10 KiB of body and judges only the code); tcpSocket passes if the port accepts a connection, which proves little; grpc passes when the standard gRPC health service answers SERVING; exec passes when a command exits 0, but forks processes on every probe, which adds up on dense nodes.

This configuration follows the rest of the article: a boot budget, a shallow and patient liveness probe, and a quicker readiness probe on its own endpoint.

containers:
  - name: orders-api
    image: example/orders-api:1.42
    ports:
      - containerPort: 8080
    startupProbe:                  # up to 30 × 2 s = 60 s to boot
      httpGet: { path: /livez, port: 8080 }
      periodSeconds: 2
      failureThreshold: 30
    livenessProbe:                 # restart only after ~30 s of no progress
      httpGet: { path: /livez, port: 8080 }
      periodSeconds: 10
      timeoutSeconds: 2
      failureThreshold: 3
    readinessProbe:                # out of the Service in ~10 s
      httpGet: { path: /readyz, port: 8080 }
      periodSeconds: 5
      timeoutSeconds: 2
      failureThreshold: 2

The docs also describe a shortcut: one cheap endpoint for both probes, with a higher failureThreshold on liveness, so a pod goes unready well before it is killed. Kubernetes’ own API server goes further: it deprecated /healthz in v1.16 in favour of separate /livez and /readyz.

Shallow and deep checks

The AWS Builders’ Library sorts checks by how much they test:

• Liveness checks: the port accepts connections, or a basic request returns 200. Cheap and blind to most bugs.
• Local health checks: resources not shared with peers. Is the disk writable? Is the worker process behind the proxy answering? These rarely fail on many servers at once.
• Dependency health checks: can this server reach its database and downstream services? They catch one server’s expired credentials or broken connection pool, but also fail on every server when the dependency itself is down.
• Anomaly detection: compare each server’s errors and latency with its peers’, from outside.

The trap is the third bullet. When the thing a deep check tests is shared, every instance reports the same failure at once, and the automation cannot tell “one bad server” from “one bad database”.

Walk through the left panel. Twenty orders-api pods run SELECT 1 in both probes, configured as above, and the primary takes 60 seconds to fail over.

• Readiness fails after about 2 × 5 = 10 s. The Service has no endpoints, so every request fails, including cache-served reads and endpoints that never touch the database.
• Liveness fails after about 3 × 10 = 30 s. Every pod restarts, finds the database still down and fails again, landing in CrashLoopBackOff: restart delays of 10 s, doubling up to 5 minutes. When the database returns at 60 s, some pods are still waiting out back-off, then all reconnect at once with cold caches.

A one-minute blip has become a multi-minute outage of a service that could have stayed partly up. As AWS puts it, without fail-open protection a health check on a dependency turns a soft dependency into a hard one.

Azure’s and AWS’s guidance converge on one split:

• Liveness never checks dependencies. Restarting your process cannot repair a shared database.
• Readiness checks local state: started, warm, not draining. Include a dependency only if every request needs it, it can fail for this instance alone, and the consumer fails open; leave optional ones out.
• Dependency status goes to monitoring and to slower, rate-limited automation that sees the whole fleet and can call a human.

Fail open: when every instance looks sick

If one instance fails its check, removing it is right. If every instance fails at once, the cause is almost certainly shared, and removing them all guarantees an outage. So good traffic managers fail open: below some level of health, they stop trusting the checks and spread traffic across everyone.

• Application Load Balancer. If a target group holds only unhealthy targets, the ALB routes to all of them. Target group health settings can raise that line: below a minimum count or percentage of healthy targets, it sends traffic to every target. AWS says Network Load Balancers fail open too.
• Route 53. In the docs’ words, “if no record is healthy, all records are healthy”. If both sides of a failover pair are unhealthy, it returns the primary.
• Envoy. Below its panic threshold, 50% by default, it ignores health and balances across all hosts (or none, if configured to fail traffic).
• Kubernetes Services do not fail open. If no pod is ready, there are no endpoints and requests fail. Readiness is where a deep check has no safety net.
• EC2 Auto Scaling throttles itself. By default it replaces at most 10% of desired capacity at a time, and may mark even fewer instances unhealthy when the load balancer reports all of them sick.

Fail open is a backstop, not a licence for deep checks. AWS names the gaps: a database that is slow or failing a few queries makes checks fail on some servers some of the time, so instances flap and the all-unhealthy trigger never fires. And at a 50% threshold, losing 4 of 10 instances is no panic, yet the remaining 6 carry 10 ÷ 6 ≈ 1.67× their normal load. So Amazon teams keep fast load balancer checks local and leave dependency checks to slower central systems that stop and page a human when too many servers look sick.

How fast can you detect a dead instance?

With interval I, timeout T and unhealthy threshold N, an instance that hangs just after passing a check is removed after about N × I + T, or one interval sooner if it dies just before a check. Then add propagation: every load balancer node, kube-proxy or DNS cache must hear the news.

ALB defaults are a 30-second interval, 5-second timeout, unhealthy threshold 2 and healthy threshold 5:

• Removal: 2 × 30 + 5 = 65 s worst case, about 6,500 requests to a dead target at 100 requests a second.
• Return: 5 consecutive passes, 5 × 30 = 150 s after recovery. A newly registered target needs only one pass.
• Tuned (5 s interval, 2 s timeout, threshold 2): 2 × 5 + 2 = 12 s, about 1,200 requests.

Kubernetes readiness defaults give 3 × 10 + 1 = 31 s, and liveness restarts after the same. For region failover, Route 53 checks from many locations every 30 s (or 10 s); its default threshold of 3 means roughly 90 s (or 30 s), plus cached record TTLs. It calls an endpoint unhealthy only when 18% or fewer of its checkers see it healthy, so one cut-off checker region cannot fail you over.

Faster is not free:

• False positives. A 1-second timeout fails on any longer garbage-collection pause. If 1% of probes fail transiently, a pod probed every 10 s (8,640 probes a day) is ejected 8,640 × 0.01 ≈ 86 times a day at threshold 1, but 8,640 × 0.01³ ≈ 0.009 at threshold 3, if failures were independent. Under load they cluster, so the real rate is higher just when capacity matters most.
• Probe load. Each ALB node checks every target. With, say, 6 nodes, a 5 s interval and 200 targets, a check that runs one query sends 6 × 200 ÷ 5 = 240 queries a second to the database.

So tune by consumer: readiness and load balancer checks fast (5 s × 2–3), because removal is cheap to undo; liveness slow, with a timeout of 2 s or more, because a restart is not; DNS failover deliberately slow.

Passive checks and gray failure

Active probes test the path you chose to probe; users take a different one. Microsoft researchers named the result gray failure, whose key feature is differential observability: the failure detector says healthy while applications suffer.

The AWS article opens with a real one. A rare bug made some Amazon.com web servers render blank error pages on every request. Their health check confirmed the rendering process answered, not that it answered correctly. And because the broken servers were now very fast, and the load balancing of the time favoured fast servers, they attracted extra traffic: a black hole. Other gray failures include a disk that fails only on writes, or a server that answers 400 instead of 500.

Passive health checks judge real responses. Envoy’s outlier detection ejects a host after 5 consecutive 5xx or connection errors by default, or when its success rate is far below its peers’, with at most 10% of hosts out at once (the circuit breaker article has the details). One trap: by default a passing active check un-ejects a host, so if your probe misses the real request path, a gray-failed host returns early; successful_active_health_check_uneject_host: false turns that off.

At fleet level, AWS runs anomaly detection, comparing every server’s error rate and latency with its peers’ from load balancer access logs. It needs servers doing similar work on similar hardware, and catches clock skew, stale code and wrong error codes that no self-check can see.

Use both. Failure detection between cluster peers, with leases and accrual detectors, is a different tool: see heartbeats.

What a good health endpoint returns

A health endpoint is called thousands of times a day by automation. Design it like an API:

• Separate endpoints. /livez: is the process making progress? /readyz: should I get traffic? Detailed status for humans goes on an authenticated path or internal port.
• Fast and cheap. Answer from in-memory flags, no I/O, tiny body. The kubelet stops reading after 10 KiB; Route 53’s string matching reads only the first 5,120 bytes.
• Dependencies from the background. AWS suggests a background loop that updates a flag, so probes stay fast and dependency load is predictable. Watch the flag’s age: if the loop dies, you stop seeing failures and recoveries. As Azure notes, the refresh interval adds to detection delay.
• Real progress for liveness. A handler on its own thread that always returns 200 cannot see a deadlocked worker pool. Have the work loop bump a timestamp, and fail when it goes stale.
• Reserved capacity. A check queued behind user requests times out under load. AWS gives servers more workers than the proxy’s connection limit, up to double, or lets health checks bypass the concurrency limit.
• No leaks. No secrets, stack traces or internal hostnames in public responses; a hard-to-guess path is not access control. Do report the build version internally, so a central job can spot a zombie running old code.

Never put these in liveness: database or cache queries, calls to other services, anything shared with peers, anything slow, anything a restart cannot fix.

A minimal version in Go:

type Health struct {
    started  atomic.Bool  // set once warm-up has finished
    draining atomic.Bool  // set by the SIGTERM handler
    lastLoop atomic.Int64 // unix seconds; the worker loop bumps it on every pass
}

func (h *Health) Livez(w http.ResponseWriter, _ *http.Request) {
    if time.Now().Unix()-h.lastLoop.Load() > 30 {
        http.Error(w, "worker loop stalled", http.StatusServiceUnavailable)
        return
    }
    w.Write([]byte("ok")) // no database, no network, no locks
}

func (h *Health) Readyz(w http.ResponseWriter, _ *http.Request) {
    switch {
    case !h.started.Load():
        http.Error(w, "warming up", http.StatusServiceUnavailable)
    case h.draining.Load():
        http.Error(w, "draining", http.StatusServiceUnavailable)
    default:
        w.Write([]byte("ok")) // dependency status is exported as a metric instead
    }
}

Graceful startup

An open port is not readiness. Gate traffic on the work that makes the instance useful:

• Readiness after warm-up. Load configuration, open pools, fill caches, then flip started. Google’s SRE book describes keeping restarted servers in a lame duck state while they pre-warm, because a fresh Java server can need far more resources for its first few minutes.
• A startup probe for slow boots. If boot can outlast initialDelaySeconds + failureThreshold × periodSeconds (30 s with defaults), liveness kills it every time. Give startup its own budget; the Kubernetes docs’ example allows 30 × 10 = 300 s.
• Slow start for cold caches. An ALB’s slow start mode ramps a new target’s share linearly over 30 to 900 seconds (off by default). It does not work with the least-outstanding-requests or weighted-random algorithms.
• A grace period for the autoscaler. EC2 Auto Scaling won’t replace a new instance for failed checks until its health check grace period ends: 300 s by default in the console, but 0 via the CLI or an SDK, which can terminate instances still booting.

Graceful shutdown

Every deploy and scale-in removes instances, so shutdown is your most frequent health transition. Done carelessly, each rolling deploy can drop a few requests per pod.

When Kubernetes deletes a pod, two things start at the same time: the control plane marks its endpoint not ready, which kube-proxy, ingress controllers and external load balancers pick up over the next seconds, and the kubelet begins shutdown. It runs any preStop hook, sends SIGTERM to process 1 in each container, and sends SIGKILL to whatever is left when terminationGracePeriodSeconds (30 s by default, counted from before preStop) runs out.

So an app that exits the instant it sees SIGTERM drops requests that proxies are still sending it. The fix is a short pause, then an orderly drain.

In configuration, using the kubelet’s built-in sleep hook (on older clusters, an exec of sleep does the same):

spec:
  terminationGracePeriodSeconds: 30   # the default, written down on purpose
  containers:
    - name: orders-api
      lifecycle:
        preStop:
          sleep:
            seconds: 5                # keep serving while endpoints update

Then, on SIGTERM, the app sets draining so /readyz returns 503 to any load balancer still probing, stops accepting connections, finishes in-flight requests within a deadline (Go’s http.Server.Shutdown with a 20-second context) and exits. Google calls this the lame duck state: able to serve, but asking clients to go elsewhere. Line up three things:

• Grace period ≥ preStop + drain + margin. 5 + 20 + 5 = 30 s fits the default; long requests need more.
• Load balancer drain ≥ your longest request. An ALB stops sending new requests to a deregistering target and waits up to its deregistration delay (300 s by default); if the target closes connections first, those clients get a 5xx.
• Signals reach your process. If process 1 is a shell script that does not forward SIGTERM, your app never hears it and is killed at 30 s mid-request.

Choosing the right check

Going deeper: how health checks go wrong

The overload spiral

Under overload, checks fail for reasons unrelated to health: busy threads, garbage-collection stalls, exhausted connections. The SRE book describes the snowball: servers drop out and survivors take their load. With 10 servers at 70% CPU, the fleet carries 7 servers’ worth of work; remove 2 and each survivor runs at 7 ÷ 8 ≈ 88%, remove a third and they hit 100%. Shed excess requests with fast 503s and keep health checks answering. In an emergency the SRE book even suggests disabling health checks briefly so the system can stabilise.

Flapping on a half-broken dependency

A dependency failing 5% of calls makes a deep check fail now and then on random servers, so capacity wobbles and no fail-open threshold is crossed. AWS stresses testing dependency checks against partial failures, not just full outages.

Workers with no load balancer

Nothing probes a queue consumer. One with a full disk keeps pulling messages and failing them fast, a black hole for work. AWS’s advice: check local health before polling for more, and alarm on processing errors and queue age.

Cheat sheet

How to say it in the interview

“I design health checks per consumer, because each one drives a different action. Liveness is shallow, just ‘is the worker loop making progress?’, never touches the database, and has loose thresholds, because a restart is expensive and can’t fix a shared outage. Readiness covers local state, warmed up and not draining, and is tight: out of rotation in about 10 seconds. Dependency health is a metric, and the load balancer fails open if every target goes unhealthy. Detection is threshold times interval plus timeout, about 65 seconds on ALB defaults, so I’d tune to 5 seconds times 2 and add outlier detection on real 5xx rates for gray failures. On deploys, a preStop sleep and draining keep rolling restarts error-free.”

Should the health check call the database? Never in liveness. Not in the load balancer check either, unless the database can fail for that instance alone and the load balancer fails open. Poll it in the background, export a metric, and let central automation or a human decide.

What happens when every instance fails its check? An ALB, NLB or Route 53 fails open and routes to everyone; Envoy does so below 50% healthy. A Kubernetes Service drops to zero endpoints, and failing liveness probes restart every pod into a crash loop. A fleet-wide fault must never look like an instance-level verdict.

How fast can you detect a dead instance? About threshold × interval + timeout, plus propagation: 65 s on ALB defaults, 31 s on Kubernetes readiness defaults, about 12 s with 5-second checks and a threshold of 2. Faster means more false ejections and probe traffic, so keep thresholds above 1 and checks cheap.

What’s the difference between liveness and readiness? Failing readiness takes the pod out of the Service but keeps it running, so it can recover. Failing liveness restarts the container. Use readiness for anything temporary, liveness only for states a restart truly fixes.

How do you deploy without dropping requests? Gate readiness on warm-up and slow-start new instances. On the way out, a short preStop sleep covers endpoint propagation, then SIGTERM fails readiness, drains in-flight requests and exits inside the grace period, with the load balancer’s deregistration delay at least as long as the longest request.

Where to go next

• Heartbeats: failure detection between cluster peers, with leases and accrual detectors.
• Load balancing: L4 vs L7: what a load balancer can check at each layer and how it picks targets.
• Backend failure modes: the slow, partial and gray failures that probes miss.
• Circuit breaker: the caller-side partner to passive outlier detection.
• Deployment strategies: canary, blue-green and feature flags: health signals as deploy gates.
• Autoscaling done right: replacement, warm-up and grace periods at fleet scale.
• Service discovery: how health status reaches the clients that pick instances.
• Multi-region and active-active: DNS failover and what “the region is unhealthy” should mean.

References and further reading

1. Implementing health checks (Amazon Builders’ Library)aws.amazon.comLiveness, local, dependency and anomaly checks, fail-open, keeping fast checks local, prioritising health checks under load, and the blank-page server that became a black hole.
2. Kubernetes documentation: Liveness, Readiness, and Startup Probeskubernetes.ioWhat each probe does on failure, the four check mechanisms, the default timing fields, the 10 KiB body limit and redirect handling.
3. Kubernetes documentation: Pod Lifecyclekubernetes.ioThe termination flow (preStop, SIGTERM, the 30-second default grace period, SIGKILL), endpoint removal in parallel with shutdown, and crash-loop back-off from 10 seconds to 5 minutes.
4. Health checks for Application Load Balancer target groups (AWS)docs.aws.amazon.comDefault interval, timeout and thresholds, the 200 success code, one passing check for new targets, and fail-open when every target is unhealthy.
5. Target groups for your Application Load Balancers (AWS)docs.aws.amazon.comDeregistration delay (300-second default), slow start mode, and target group health thresholds for routing failover and DNS failover.
6. How Amazon Route 53 chooses records when health checking is configureddocs.aws.amazon.com“If no record is healthy, all records are healthy”, and what failover records return when both sides are unhealthy.
7. Envoy documentation: Outlier detectionenvoyproxy.ioPassive health checking on real traffic, the ejection algorithm and its interaction with active health checks (defaults from the outlier detection API reference).
8. Envoy documentation: Panic thresholdenvoyproxy.ioThe 50% default below which Envoy ignores health status and balances across all hosts, or none.
9. Health Endpoint Monitoring pattern (Azure Architecture Center)learn.microsoft.comShallow checks for liveness and deep checks for readiness, cascading failure from deep probes, the staleness cost of cached status, and securing health endpoints.
10. Addressing Cascading Failures (Google SRE book)sre.googleProcess health versus service health checking, how overload makes health checks fail, and disabling checks to let a system stabilise.