Three levers: blast radius, time to detect, time to recover
Fleet-wide, a build failing every request for 10 minutes, plus 5 to roll back, burns 35% of a 99.9% month. On a 1% canary: 0.15 minutes.
Three ways to replace v1: who’s exposed, how fast you undo
Blue-green flips back fastest, but its flip exposes everyone and needs a second fleet. A canary risks 1–5% while it is judged.
Healthy compared with what?
Bad: Against yesterday
Monday’s release against the weekend mostly measures the weekend.
Good: Against a fresh baseline
A v1 copy started with the canary, same size and traffic. Warm caches and uptime cancel out.
0.3
errors expected on a quiet service’s 1% canary
At 50 requests a second, 1% sees 300 requests in 10 minutes. Only a catastrophe shows. Size canary steps in requests, not percent.
The new checkout misbehaves. Roll back the whole build?
Deploy ≠ release
Switch its flag off, in seconds. You deployed it dark, then released it to staff, 1%, 10% and 100% of users.
Knight Capital, 2012: one reused flag, one missed server
Seven of eight servers got the new code. The eighth ran long-dead code behind the reused flag: over $460 million lost in about 45 minutes.
Rollback breaks when v1 can’t read v2’s data
- Prepare: read both formats, still write the old
- Bake, and check every server has it
- Activate: write the new format
- Backfill, then remove the old reader
Remember
Shrink blast radius, detect fast, undo cheaply: canary against a baseline, release with flags, and keep data backward compatible.