Config flows down ahead of time. No tenant request goes up.
The control plane onboards tenants, bills, flips flags and ships releases. The data plane serves every tenant call from its own in-memory copy.
10,000×
requests per control-plane write
Illustrative helpdesk SaaS. 50,000 requests a second against, say, 5 writes. The data plane gets the higher target; the control plane, with more moving parts, fails more often.
Just look up the tenant per request? 11 times the downtime.
99.99% × 99.9% ≈ 99.89%: 4.3 minutes a month becomes 47.5. And a service sized for 5 writes a second must answer 50,000 reads.
Then cache the lookups?
Bad: Hard 60-second TTL
A 30-minute control-plane outage becomes 29 minutes of errors, then a stampede.
Good: Last-known-good copy
Refresh in the background, never expire on read, alert on age. No failed requests.
Static stability: serve through a control-plane outage
- Push config before the tenant’s first request
- Read it from memory, never the network
- Keep last-known-good on disk for restarts
- Recover with spare capacity, not new resources
Cloudflare, July 2019: one rule, everywhere in seconds
A WAF regex exhausted CPU on every core, 27 minutes of 502s. Config is the one input every node shares, so roll it out like code.
The control plane is down; nodes fail static on old config. Then a leaked key is revoked.
The boundary
Security data gets a maximum age, then fails closed, plus a high-priority revocation path. Never fail open.
Remember
Push config ahead of need; roll it out like code. When the control plane is down, you lose change, not service.