Control plane vs data plane

A control plane decides what a system should look like: it creates, changes and deletes resources, sets policies and rolls out configuration. A data plane does the work, request by request, with the configuration the control plane produced. In a SaaS product, onboarding, billing, flags and releases are the control plane; a tenant’s API call is the data plane. The rule that follows: keep the control plane off the request path, so the data plane keeps serving when the control plane is down.

Picture it first: a restaurant chain’s head office

A restaurant chain runs 300 outlets from a head office in Bengaluru. Head office sets the menu and prices, opens outlets, and sends every billing terminal a fresh menu file each morning. The outlets take orders, cook and bill: thousands of orders an hour, against a few dozen head-office changes a day.

Now three bad days:

• Head office’s server crashes at lunch. Counters carry on, because every terminal still has this morning’s menu. Only change stops: no new dish, no price update.
• A chain that cut a corner had each terminal ask head office for an item’s price at billing time. Same crash, and all 300 outlets stop at once.
• A bad menu file that crashes the terminal software goes to all 300 outlets at 12:55. The one thing every outlet shares is the one thing that can break them all. A careful chain tries a new file in three outlets first.

Map it back. Head office is the control plane, the outlets are the data plane, and the menu file is configuration. Keeping this morning’s file is static stability, the price call is a control-plane call on the request path, and the three trial outlets are a staged rollout.

Two planes, two jobs

The words come from networking: a router’s control plane runs routing protocols to decide where traffic goes, and its data plane forwards each packet using the resulting table. AWS’s Builders’ Library defines a control plane as the machinery that adds, deletes and modifies resources and propagates those changes to where they take effect, and a data plane as “the daily business of those resources”.

A quick test: does the operation change what the system is, or use what it is? Launching an instance, creating a tenant, changing a plan or publishing a release is control plane; AWS calls these the administrative create, read, update, delete and list (CRUDL) APIs. The running instance, its disk reads and writes, and a tenant’s API call are data plane. The split is configuring versus serving, not reads versus writes.

In a SaaS product

AWS’s SaaS Architecture Fundamentals splits a SaaS environment into a control plane and an application plane, the data plane under another name. The control plane onboards, authenticates, manages, operates and analyses tenants through core services such as onboarding, tenant management, identity, billing, metrics and admin users. The application plane is the product and its backend services.

Two details are worth repeating. The control plane is not itself multi-tenant: its services are global, and tenants are the objects it manages, not its users. And AWS puts tenant provisioning in the application plane, while admitting some would put it in the control plane. Azure’s guidance draws the same line: in a bookkeeping SaaS, entering records is data plane, while onboarding a tenant and creating its database is control plane.

How the two planes differ

Take an illustrative B2B helpdesk SaaS with 2,000 tenants.

• Volume. The data plane peaks at 50,000 requests per second. The control plane sees sign-ups, plan changes, invites, flag flips and deploys: say 5 writes per second when busy. That is 50,000 ÷ 5 = 10,000 requests per control-plane write. AWS notes data planes often run at higher volume “by orders of magnitude”, so scale the two separately.
• Complexity. Onboarding, in Azure’s example, deploys a database (minutes), updates the tenant catalog, sends a welcome email, calls a third-party billing API that occasionally fails, and updates the CRM. AWS’s conclusion: with more moving parts, control planes are statistically more likely to fail.
• Availability. Tenants feel every failed request, but notice a control-plane outage only when changing something. So the data plane gets, say, 99.99% (4.3 minutes in a 43,200-minute month) and the control plane 99.9% (43.2 minutes); Azure recommends separate objectives for it. See availability and the nines.
• Blast radius. A data-plane bug usually hurts one stamp. Configuration, the control plane’s output, is the one input every stamp shares, so a bad push hurts everyone.
• Privilege. The control plane can create resources and reach every tenant’s settings. Azure warns that a breach there can be catastrophic and advises running it apart from tenant workloads, which doubles as a bulkhead.

The same split everywhere

The pattern is everywhere, and in each case a well-built data plane keeps going when its control plane stops.

Two rows make the point sharply. Istio says its Envoy proxies are the only components that touch traffic, while istiod turns routing rules into Envoy configuration and pushes it to the sidecars: the service mesh is the split made literal. And Route 53’s query-answering data plane is designed for a 100% availability SLA, while the APIs that change records run in one Region (US East, N. Virginia), outside the SLA, so AWS advises failing over with health checks, not record edits.

What lives where in a SaaS product

The control plane holds the tenant catalog (each tenant’s status, plan, limits, region and stamp), onboarding and lifecycle workflows run as idempotent steps on a workflow engine, tenant placement and rebalancing, identity administration, billing, the authoring of flags, entitlements and limits, and release orchestration.

The data plane is everything a tenant request or its background job touches: the API gateway or router, services, tenant databases, caches, queues and workers. It also holds read-only, in-memory copies of control-plane data: the tenant map, entitlements, flags and limits.

Identity straddles the line: a tenant’s SSO set-up is control plane, but checking tokens is per-request work, so verify them at the gateway with cached signing keys.

Three points are specific to SaaS.

One control plane, even for siloed tenants. Pooled, siloed or spread across stamps, tenants share one onboarding flow, catalog, billing pipeline and release train; that shared layer makes a stack per tenant SaaS rather than hosting (tenancy models). Azure says most solutions need only one. At scale it can split into a global control plane (placement, lifecycle) and stamp control planes (resources and maintenance inside one stamp). Below about 10 tenants, Azure suggests manual operations can do, with one central tenant list and reusable scripts.

Routing reads control-plane data on every request. The control plane owns the tenant-to-stamp map; the router reads it 50,000 times a second when routing requests to tenants. It fits in memory: even 100,000 tenants × 200 bytes is only 20 MB. Service discovery works the same way.

Metrics flow back up. Usage events and per-tenant metrics travel up asynchronously through a queue and become invoices (metering and usage-based billing), quota decisions and placement signals. If the control plane is down, the data plane buffers them; it never holds up a request to report one.

Static stability: serve through a control-plane outage

AWS calls a system statically stable when it keeps working while a dependency is impaired: it may miss updates the dependency should deliver, but whatever worked before keeps working. In EC2, the host running an instance has everything it needs locally to route that instance’s packets. If the EC2 control plane is impaired, new instances and security-group rules don’t arrive, but existing traffic flows, and a launched instance needs no control-plane API to keep running or to restart after a reboot.

Keep the control plane off the request path

The commonest way to lose static stability is one innocent synchronous call: look up the tenant, check the entitlement, fetch the flag. Serial availabilities multiply:

• Data plane alone: 99.99%, or 0.0001 × 43,200 ≈ 4.3 minutes down a month.
• Calling the control plane on every request: 0.9999 × 0.999 ≈ 99.89%, or 0.0011 × 43,200 ≈ 47.5 minutes.

That is 11 times the downtime, and a service sized for 5 writes a second must now answer 50,000 reads a second: the control plane has become a single point of failure for every tenant.

A cache in front helps only if entries never expire on the read path. With a hard 60-second TTL, lookups start failing a minute into an outage, so a 30-minute control-plane outage becomes a 29-minute data-plane outage, followed by a stampede of refreshes the moment it recovers.

Rules for a statically stable data plane

1. Push config ahead of need. A new tenant’s routing entry and limits go into a snapshot before its first request, and certificates rotate long before they expire.
2. Read from memory on the request path.
3. Refresh in the background and never expire on read. Alert on the config’s age instead of failing.
4. Keep the last-known-good copy on disk, so a node restarted mid-outage can still boot.
5. Pre-provision instead of reacting. Adding capacity mid-incident depends on the control plane, config downloads, discovery and credentials, just when they’re most likely impaired. AWS runs load-balanced services in three zones at about 66% of tested capacity: each zone carries a third of the load at two-thirds of its limit, so any two can carry all of it.
6. Recover with data-plane actions: shift traffic with health checks and spare capacity, not new resources or DNS edits.

Rules 2 to 4 in code, for a router holding the tenant map:

let current: Snapshot = loadLastKnownGood();       // from local disk; refuse traffic if none

async function refreshForever() {
  for (;;) {
    try {
      const next = await fetchSnapshot({ newerThan: current.version });
      if (next) {
        validate(next);                             // throws on a bad snapshot: keep the old one
        await saveLastKnownGood(next);              // a restart mid-outage still has config
        current = next;                             // one atomic swap: v41 or v42, never a mix
        reportStatus({ version: next.version, ok: true });
      }
    } catch (err) {
      reportStatus({ version: current.version, ok: false, error: String(err) });
    }
    metrics.gauge("config_age_seconds", ageOf(current)); // staleness is a metric, not an outage
    await sleep(60_000 + Math.random() * 10_000);       // jitter: no synchronised stampede
  }
}

export function stampFor(tenantId: string): string | undefined {
  return current.tenants.get(tenantId);             // memory read: no network call per request
}

How configuration flows down

Every control plane, from istiod to a SaaS tenant catalog, runs a version of the same loop.

1. Write and validate. A change arrives by console, API or pull request; the control plane checks invariants (every tenant maps to a real stamp) and stores it in a strongly consistent store such as PostgreSQL or etcd.
2. Snapshot. Build an immutable, numbered version (v42) per scope, such as a stamp or region. Immutability makes rollback trivial.
3. Distribute, by push or pull.
4. Apply atomically, or reject. Build the new structures beside the old ones and swap one pointer, so requests see v41 or v42, never a mix. Envoy acknowledges (ACK) or rejects (NACK) every update, and after a rejection its last valid configuration stays in force.
5. Report the version each node runs, and its health.
6. Gate. A rollout controller compares desired with reported versions and starts the next wave or rolls back.

Push or pull, snapshot or delta

Push streams changes to subscribed nodes, and it is fast: Cloudflare reported a p99 of 2.29 seconds for a change to reach every machine in its network, at about 350 changes a second. You pay in server-side state and in reconnect storms after an outage, which jittered retries soften.

Pull has nodes poll a known location. AWS Hyperplane merges customer changes into one file in S3, and every node fetches and loads it every few seconds, changed or not. That is constant work: load doesn’t rise when many customers change things at once, and a missed or corrupted update is replaced on the next pass. You pay up to one poll interval of delay.

Size decides how far full snapshots go. With 2,000 tenants × 200 bytes, the map is 400 KB, and 500 routers fetching it every 5 seconds move 500 × 400 KB ÷ 5 s = 40 MB/s. At 100,000 tenants (20 MB) the same loop moves 2 GB/s, so check the version first or send deltas, as Envoy’s incremental mode does. Deltas need ordering: tag each with the version it builds on, and resend a full snapshot when a node finds a gap.

Reconciliation loops

Kubernetes makes reality match desired state with controllers: loops that compare an object’s desired spec with the current state, act on the difference and report status. It prefers many small controllers to one big one, and expects any of them to fail. Unlike a one-shot workflow (“tenant created: run six steps”), a loop asks every cycle what should exist and what does, so a crashed worker or lost event is fixed on the next pass.

In SaaS, write the tenant record (status: active, stamp: stamp-07) and let a tenant controller reconcile the database, DNS record, routing entry and limits. Onboarding becomes an insert and offboarding a status change, both safe to retry. Keep the provider’s own config in Git as config as code: reviewed, validated in CI, rolled back by revert. Tenant-initiated changes still need an audited API.

Roll out configuration like code

Configuration is the one input every node shares, and its distribution systems are built for speed: the recipe for a global outage.

Cloudflare’s write-up of its July 2, 2019 outage is the standard example. At 13:42 UTC an engineer deployed a minor change to the WAF’s cross-site-scripting rules, which travel through Quicksilver, the system that pushes changes globally in seconds. A regular expression in the change backtracked enormously and exhausted the CPU on every core serving HTTP and HTTPS across the network, and visitors saw 502 errors. A global WAF kill switch ran at 14:07, traffic was back by 14:09, and the outage lasted 27 minutes. Three details matter:

• The rule went out in a “simulate” mode that logs without blocking, but that mode still executes the rule.
• Software releases went through stages (an employees-only location, a small share of free traffic, canary locations, the world). Rules could go global at once, even when not urgent, because they sometimes must answer attacks fast.
• The fixes included staged rollouts for rules, with an emergency global path kept for active attacks, and performance tests for rules.

A staged config rollout borrows from a canary deployment:

• Validate before publishing, including a performance test for anything executable and a check that a real data-plane binary loads the snapshot.
• Roll out in waves along fault boundaries: one cell, one region, half, all. AWS deploys EC2 software zone by zone, on different days.
• Bake and gate each wave on errors, latency, CPU and NACK rate. A failed gate rolls back automatically.
• Make rollback a pointer to the previous immutable snapshot, never a rebuild. Cloudflare’s rollback plan meant running the full WAF build twice, which took too long.
• Add tenant rings in SaaS: internal tenants first, then friendly ones, then everyone by tier.

If detecting and undoing a bad change takes 5 minutes either way, a global push exposes 100% × 5 = 500 percent-minutes and a 1% canary 1% × 5 = 5. A canary doesn’t make you faster; it makes being slow cheaper. Cloudflare’s first alert came 3 minutes after its deploy began, by which time the change was everywhere.

Going deeper: failure modes and recovery

Fail static, fail closed or fail open

When the data plane can’t refresh a policy, it can fail static (keep the last-known-good policy), fail closed (refuse) or fail open (allow everything). Static is the right default, but decide per type of data:

Fail static delays every change, including urgent ones. A tenant suspended for non-payment working an extra hour is fine. A leaked key needs a small, separate, high-priority revocation path, and fail static must never quietly become fail open; see multi-tenancy and isolation.

Circular dependencies

Control planes often depend on the data plane they manage: admins sign in through the product’s own SSO, and the deploy pipeline pulls images from a registry on the cluster it deploys to. In Cloudflare’s 2019 outage, engineers couldn’t authenticate to their internal control panel because Cloudflare Access was down, and customers couldn’t reach the dashboard or API because those pass through Cloudflare’s own edge; a follow-up was a way to take them off the edge in an emergency. AWS names avoiding circular dependencies as part of static stability. Host the control plane on a lower layer than the one it manages, keep break-glass credentials and an out-of-band path, and rehearse them.

Bootstrapping and recovery

A fresh node must find config even when the control plane is down: the control plane, else a copy in object storage, else the last-known-good file on disk, else defaults baked into the image. The tenant catalog is small but is the map of the business, so replicate it across regions and test restores as part of disaster recovery. Azure lists what a control-plane outage costs even while traffic flows: no onboarding, no billing, no way to disable a tenant during a security incident, and maintenance debt piling up. Afterwards, release the backlog of deferred changes in waves, like any rollout.

Other names

Network gear often adds a management plane (CLI, SSH, SNMP) beside the routing control plane; Azure calls Azure Resource Manager its control plane, where others say management plane. The test is the same: does the operation change the system, or serve through it?

Cheat sheet

How to say it in the interview

“I’d split the platform into a control plane and a data plane. The control plane owns the tenant catalog, onboarding, billing, flags and releases: low volume, long workflows, one shared system even for siloed tenants. The data plane is the request path: router, services, tenant databases. It carries thousands of times the traffic and gets the higher availability target, so it never calls the control plane synchronously. Instead it holds versioned snapshots of the tenant map, entitlements and limits in memory and on disk, and serves the last-known-good version if the control plane is down. Config changes roll out like code: validated, canaried on one cell, then waves, with automatic rollback.”

Why not call the control plane and cache the answers? A hard TTL only delays the dependency: with 60 seconds, a 30-minute control-plane outage becomes a 29-minute data-plane outage plus a refresh stampede. Push data ahead of need, never expire it on read, and alert on its age.

A tenant is suspended while the control plane is down. What happens? Nothing until it returns, because the data plane fails static. That’s fine for non-payment; for security revocations I’d add a small high-priority path and a maximum age, after which nodes fail closed.

How do you stop one bad config from taking down every region? Validate it, publish immutable versions, roll out to one cell and then in waves along fault boundaries, gate each wave on health, and roll back by repointing to the previous version.

How does a tenant move between stamps safely? The control plane copies the data, publishes a map version with the new stamp, waits until every router reports that version, and only then cuts writes over.

Where to go next

• What is a tenant?: the entity the control plane manages, and its lifecycle.
• Tenancy models: silo, pool and bridge: why one control plane serves every model.
• Routing requests to tenants: the per-request read of control-plane data.
• Metering and usage-based billing: the usage that flows back up.
• Cell-based architecture: deployment stamps and geodes: fault boundaries for data planes and config waves.
• Deployment strategies: canary, blue-green and feature flags: staged rollouts for code and config.
• Sidecar, ambassador and service mesh: istiod and Envoy as the two planes.
• Fault tolerance: the wider toolkit around static stability.
• Design a multi-tenant SaaS platform: both planes in one design.

References and further reading

1. AWS SaaS Architecture Fundamentals: Control plane vs. application planedocs.aws.amazon.comThe SaaS split into a control plane (onboard, authenticate, manage, operate, analyse) and an application plane; control-plane services are global to all tenants, not multi-tenant.
2. Amazon Builders’ Library: Static stability using Availability Zonesaws.amazon.comAWS’s definitions of both planes, the EC2 example, data planes running at orders of magnitude more volume, and over-provisioning three zones at about 66% instead of reacting to failures.
3. AWS Fault Isolation Boundaries: Static stabilitydocs.aws.amazon.comControl planes are statistically more likely to fail; existing resources stay available without them; avoid circular dependencies and control-plane calls in the recovery path.
4. AWS Well-Architected Reliability Pillar: REL11-BP04 Rely on the data plane and not the control plane during recoverydocs.aws.amazon.comControl planes as CRUDL APIs; Route 53’s data plane is designed for a 100% availability SLA while its control plane sits in one Region outside the SLA.
5. Azure Architecture Center: Considerations for multitenant control planeslearn.microsoft.comControl-plane responsibilities, what a control-plane outage costs, isolating it from tenant workloads, long-running onboarding workflows, and global, stamp and tenant control planes.
6. Kubernetes documentation: Controllerskubernetes.ioControllers as non-terminating control loops that move current state towards desired state and report status back through the API server.
7. Istio documentation: Architectureistio.ioA mesh split into Envoy sidecars (the only components touching traffic) and istiod, which turns routing rules into proxy config, pushes it and issues certificates.
8. Envoy documentation: xDS REST and gRPC protocolenvoyproxy.ioVersioned config, ACK and NACK for every update, the last valid config staying in force after a rejection, and state-of-the-world versus incremental updates.
9. Cloudflare blog: Details of the Cloudflare outage on July 2, 2019blog.cloudflare.comA WAF rule pushed globally in seconds exhausted CPU worldwide for 27 minutes; no staged rollout for rules, and the outage blocked access to Cloudflare’s own control panel.
10. Amazon Builders’ Library: Reliability, constant work, and a good cup of coffeeaws.amazon.comHyperplane nodes pulling a full config file from S3 every few seconds, and why constant-work loops self-heal where event-driven workflows build backlogs.