Who asks, and what happens when your check says no?
The more drastic and slow to undo the action, the narrower and more certain the check behind it must be.
Both probes run SELECT 1. The database fails over for 60 s.
Every pod’s check fails at once. Readiness empties the Service in ≈ 10 s; liveness restarts every pod at ≈ 30 s, into a crash loop.
What should each probe ask?
Bad: Is the database up?
Shared: all pods fail at once. Restarts can’t fix that.
Good: Only its own state
Liveness: is my loop progressing? Readiness: warm, not draining? Database health: a metric.
Every target fails its check at once. Remove them all?
Fail open
No: the cause is almost certainly shared. ALB, NLB and Route 53 route to everyone; Envoy below 50%. Kubernetes Services never do.
65 s
until ALB defaults pull a hung target
2 × 30 s interval + 5 s timeout. At 100 requests a second, ≈ 6,500 reach a dead target. Tuned checks: 12 s.
Endpoint removal and SIGTERM start together, so leave in order
- preStop sleep 5 s while proxies stop sending
- On SIGTERM,
/readyzreturns 503; drain in-flight requests - Exit inside the 30 s grace period, before SIGKILL
Amazon’s broken servers passed their checks and won more traffic
The check asked whether the renderer answered, not whether it answered correctly. Passive checks judge real responses; anomaly detection compares each server with its peers.
The line to say
Liveness never checks the database: a restart can’t fix a shared outage. Readiness checks local state; dependency health is a metric, not a kill switch.