28/s
fresh machines needed at the 11:00 peak
1,700 jobs a minute, each on a new VM that takes about 60 s to boot. So keep ≈ 1,700 runners booted and waiting.
Who runs next at 11:05?
Bad: One FIFO per pool
The hotfix waits ≈ 6 min behind a refactor’s 3,000 jobs.
Good: Team lanes, P0 first
The hotfix takes the next runner; the big team stays busy.
A cut-off runner reports “succeeded” after its replacement failed.
Its lease expired, so job 42 was requeued with a new token. Every write carries the token, so the stale one gets 409.
A PR can change the build script. Yet some jobs must change production.
The trick
Store no keys. Each job swaps a signed OIDC token for short-lived credentials; PR tokens never match prod.
45 → 6
minutes for a monorepo pull request
Most of the 45 minutes redo work another machine did an hour ago. Cache by content hash and test only what the change affects.
The canary hits 4% errors. The whole service barely moves.
So compare the 5% canary with a fresh baseline every minute. Three failed checks abort it at minute 4.
Ship what you tested, even with CI down
- Deploy by digest, never
:latest - Promote that digest; one lock per environment
- Commit each promotion to a config repo
- Cluster agents pull it and repair drift
Remember
CI leases throwaway runners fairly and caches by content. CD promotes one digest through a judged canary, and clusters pull it.