Design a CI/CD deployment system

Every change your engineers make passes through one system before it reaches users: the CI/CD platform. It builds and tests each push and pull request, puts a green tick or a red cross on the PR, and then carries the artifact that passed — the same bytes, not a rebuild — through staging, a canary and production, rolling back by itself when the canary looks sick. Here you design it for an Indian consumer-internet company with about 5,000 engineers and 3,000 repositories, one of them a monorepo holding 400 services. Three things make it interesting. It is a job scheduler with brutal daily peaks. It runs code it cannot fully trust on machines that can reach production secrets. And its deploy half is the most dangerous button in the company, because it ships every change.

Understand the problem

Split the system in two before you draw anything. CI (continuous integration) answers “is this commit good?”. It is a batch compute platform, judged by queue time and minutes per pipeline. CD (continuous delivery) answers “is this artifact safe in production?”. It is a low-volume, high-stakes control system, judged by blast radius and time to roll back. They share a pipeline file and a web UI, but they fail in different ways, and a strong answer gives each half the design it needs.

Functional requirements

1. Engineers should be able to define pipelines as code in a YAML file in the repository — a graph of jobs with dependencies and matrices — triggered automatically on push, pull request, tag, schedule or a manual dispatch.
2. Every job should run on a clean, isolated runner, with its log visible live and its result reported as a status check on the pull request.
3. Pipelines should be able to store and reuse artifacts and caches: container images, test reports, dependency caches and build caches.
4. Teams should be able to deploy an artifact through environments — staging, canary, production — with approvals, automated gates and automatic rollback on bad metrics.
5. Anyone should be able to see pipeline and deploy history and answer “who shipped what, where, and when?”.

Non-functional requirements

• Scale. About 50,000 pipelines and 400,000 jobs a day, nearly all in a 10-hour working day, peaking around 11:00 and 16:00.
• Queue time. p95 under 30 seconds from “ready” to “running” on Linux pools; scarce macOS and GPU pools get looser targets.
• Log latency. A line printed on a runner shows up in the browser within 2 seconds.
• Isolation. Every job gets a fresh machine, destroyed afterwards. Code from forks or unreviewed branches never sees a production secret, and no long-lived cloud keys live in CI.
• Deploy safety. About 500 service deploys a day. A bad build is detected and rolled back within 10 minutes, having served at most 5% of traffic.
• Availability. The CI control plane targets 99.9%, about 43 minutes of downtime a month. Running production services must not depend on CI at all, and there must be a way to ship a fix while CI is down.
• Audit. Every deploy records who, which digest, where, when and who approved, kept for a year.

Out of scope

• Hosting Git. We consume webhooks and APIs from the Git host; we don’t design it.
• The build tools. Compilers, Gradle and Bazel run inside jobs; we make them fast, we don’t write them.
• Mobile app-store releases, a separate workflow.
• Feature flags. They are the other half of safe releases — covered in canary, blue-green and feature flags — and live in their own service.
• Infrastructure provisioning. Terraform plans and applies are just jobs to this platform.

The set-up

Plan the approach

Build each requirement simply, then spend most of the time in the deep dives. Say the shape early: “a control plane that turns events into job graphs, a scheduler that leases jobs to throwaway runners, side services for logs and artifacts, and a separate deploy path that promotes one immutable artifact through environments.”

Core entities

• Pipeline definition. The YAML file, always read at the run’s own commit.
• Run. One execution: repository, commit SHA, trigger, status and who started it.
• Job. A node in the run’s graph: needs, runner labels (linux, arm64, large), state, attempt and lease.
• Runner. An ephemeral machine with labels, in a pool. It runs exactly one job.
• Lease. Job, runner, token and expiry. The current token’s holder owns the job.
• Artifact. A blob or image named by its digest, with provenance: which run and commit built it.
• Cache entry (key, scope, size, last access) and log chunk (job, attempt, sequence number, bytes).
• Environment. Staging or production for a service: protection rules, approvers, secrets and a deploy lock.
• Deployment. Service, environment, digest, wave, status and approvals — the audit record.

A pipeline file for one service in the monorepo:

name: payments-api
on:
  pull_request: { paths: ["services/payments/**"] }
  push: { branches: [main], paths: ["services/payments/**"] }
concurrency:
  group: payments-${{ github.ref }}
  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
jobs:
  build:
    runs-on: [linux, x64, large]
    steps:
      - run: make build
  lint:
    runs-on: [linux, x64]
    steps:
      - run: make lint
  test:
    needs: build
    runs-on: [linux, x64]
    strategy:
      matrix: { shard: [1, 2, 3, 4] }
    steps:
      - run: make test SHARD=${{ matrix.shard }}/4
  image:
    needs: [test, lint]
    if: github.ref == 'refs/heads/main'
    runs-on: [linux, x64]
    steps:
      - run: make image   # pushes payments-api@sha256:…
  deploy-prod:
    needs: image
    environment: production   # approvals, deploy lock, canary policy
    deploy: { service: payments-api, strategy: canary }

API

Events in. The Git host calls a webhook; we answer fast and work asynchronously.

POST /v1/hooks/git
X-Hub-Signature-256: sha256=9f2c…
X-Delivery-Id: 7c1e-44b0

{ "event": "push", "repo": "food-app/monorepo", "ref": "refs/heads/main",
  "after": "a1b2c3d", "pusher": "priya" }

→ 202 Accepted   (run created asynchronously; duplicate delivery IDs are ignored)

The runner protocol. Runners only make outbound HTTPS calls. A runner long-polls for a job and a lease, then heartbeats, appends logs and completes, each call carrying the lease token.

POST /v1/runners/r-88f1/acquire        (held open up to 50 s)
→ 200 { "job_id": 42, "attempt": 1, "lease_token": 8, "lease_expires_in_s": 60,
        "spec": { "steps": [...], "oidc_audience": "cloud-prod" } }

POST /v1/jobs/42/heartbeat   { "lease_token": 8 }               → 200 | 409 lease lost
POST /v1/jobs/42/logs        { "lease_token": 8, "seq": 311, "data": "…64 KB…" }
POST /v1/jobs/42/complete    { "lease_token": 8, "conclusion": "failure", "exit_code": 2 }

Logs out. The browser follows a job with server-sent events, resuming from the last sequence number it saw.

GET /v1/jobs/42/logs?follow=true
Accept: text/event-stream
Last-Event-ID: 311

Deploys.

POST /v1/deployments                    body: service, digest, environment, strategy → 201 deployment id
GET  /v1/deployments/d-901              status, current wave, analysis results, approvals
POST /v1/deployments/d-901:approve      a required reviewer approves the next gate
POST /v1/deployments/d-901:abort        stop and shift traffic back to the stable version
POST /v1/services/payments-api:rollback body: to_digest (defaults to the last healthy one)
GET  /v1/services/payments-api/history  who shipped which digest where and when

Back-of-envelope estimates

These numbers size the fleet and show which resources matter (method: back-of-envelope estimation).

Pipelines. 5,000 engineers push about 8 times a day to open PRs: 40,000 PR pipelines. Add about 6,000 merges to main, and 4,000 tag, scheduled and manual runs: 50,000 pipelines a day. At an average of 8 jobs each, that is 400,000 jobs a day.

Peak rate. Nearly all of it lands in a 10-hour working day: 400,000 ÷ 600 minutes ≈ 670 jobs a minute. The 11:00 and 16:00 peaks run about 2.5× that: ≈ 1,700 jobs a minute, about 28 a second. Plan for 2,000 a minute on release days.

Concurrent runners. Little’s law: jobs in flight = arrival rate × time in system. Jobs average 8 minutes, so 1,700 × 8 ≈ 14,000 concurrent jobs at peak, about 5,400 in an ordinary hour, a few hundred at night.

Cost. 400,000 × 8 minutes = 3.2 million runner-minutes ≈ 53,000 VM-hours a day. At an assumed ₹12 per hour for a 4-vCPU, 16 GB VM, that is about ₹6.4 lakh a day. A fixed fleet sized for the peak would cost 14,000 × 24 h × ₹12 ≈ ₹40 lakh a day — six times more. Autoscaling is not optional.

Provisioning rate. Runners are ephemeral, so every job needs a new machine: about 28 VMs a second at peak. If a VM takes about 60 seconds to boot and register, jobs would wait twice the 30-second target. The fix is a warm pool, sized by Little’s law again: 28 VMs/s × 60 s ≈ 1,700 runners booted and waiting at peak, about 12% on top of the busy fleet.

Control-plane traffic. 14,000 runners heartbeating every 15 seconds: about 930 calls a second. Long-polls: 1,700 ÷ 50 s ≈ 34 a second. Log appends, one a second from each job that is printing: ≈ 10,000 a second — about 90% of all calls (10,000 of roughly 11,000), a hint that logs deserve their own service.

Logs. A typical job prints about 20,000 lines of 100 bytes: 2 MB. 400,000 × 2 MB = 800 GB a day raw, about 100 GB compressed 8×, so 9 TB for 90 days. Live ingest at peak: 14,000 jobs × 2 MB ÷ 480 s ≈ 60 MB/s.

Caches. Say 60% of jobs restore a 300 MB dependency cache: 240,000 × 0.3 GB = 72 TB of cache reads a day, 2 GB/s on average over the working day and about 5 GB/s at peak — 80 times the log traffic.

Artifacts. 6,000 main-branch pipelines push 1.5 images each, about 60 MB of new layers per image: ≈ 540 GB a day, about 50 TB with retention. Test reports from 30% of jobs at 10 MB add 1.2 TB a day, 36 TB over 30 days. Storage is cheap next to compute.

Deploys. 500 deploys × about 5 stages = 2,500 transitions a day, a few a minute. The deploy half is hard because of correctness, not load.

High-level design

One requirement at a time. By the end of this section the platform works for a few hundred engineers; the deep dives make it work for 5,000.

Pipelines are defined as code and triggered automatically

A trigger service receives the Git host’s webhook, checks the signature, drops duplicates by delivery ID (webhooks arrive at least once) and queues a run request before answering 202. A planner fetches .ci/pipeline.yaml at the pushed commit, evaluates triggers and path filters, expands matrices and writes the run, its jobs and their edges to the runs database (PostgreSQL is plenty at about 450,000 rows a day).

The result is a directed acyclic graph. needs edges say what must succeed first; a failure skips everything downstream unless a job opts in with always(). A matrix becomes one real job per combination — GitHub caps a matrix at 256 jobs per run, a guardrail worth copying.

Two details matter from day one. Idempotency: the run key is (repository, SHA, event), so a redelivered webhook finds the existing run (see idempotency). Concurrency groups: a new push to a PR makes the previous run pointless. GitHub’s semantics are a good model: at most one running and one pending run per group, a newer pending run replaces the older one, and cancel-in-progress cancels the running one too — one of the cheapest savings in the system.

Every job runs on a clean runner and reports back

Jobs whose needs are met become queued, and a scheduler owns them. Runners are throwaway machines: start, take one job, run it, report, die. They long-poll over outbound HTTPS: an acquire call is held open up to 50 seconds and returns the moment a matching job exists.

Handing out a job creates a lease: “runner r-88f1 owns job 42 until 10:03:15, token 8”, renewed by a heartbeat every 15 seconds. If heartbeats stop, the lease expires and the job is requeued. Every state change is a conditional update on the job row, so two scheduler replicas never hand out the same job.

Two things leave each job: logs, in small chunks to a log service so the browser can show them live, and status, posted to the Git host as a check run that branch protection can require.

Artifacts and caches are stored and reused

Images, bundles and test reports go to an artifact store — object storage plus a container registry — named by content digest, so “the image that passed tests” is one immutable thing: payments-api@sha256:4be1…. Caches are optional speed-ups a job must never depend on. Both sit in the runners’ region and zones; at 5 GB/s of cache reads, cross-region traffic would be slow and expensive.

Artifacts are deployed through environments with gates

A deploy is not a script on a runner: it can take an hour, must survive any machine dying, and needs credentials no build job should hold. So a job with an environment goes to a deploy service, a durable state machine (a workflow engine fits) that checks protection rules — reviewers, allowed branches, wait timers, freezes — and walks the artifact through staging and production. For now it applies digests to each cluster’s API and reads error rates from metrics; the deep dives make both safer.

Everyone can see history and audit deploys

Every state change — run created, job leased, approval given, wave promoted, rollback — is appended to an event log and projected into the runs database. “Who shipped payments-api to production on Tuesday at 14:05?” is one indexed query on (service, environment, time).

At 5,000 engineers this breaks in six places: jobs wait minutes at 11:00, one team’s burst starves everyone, a PR can read a production secret, monorepo builds take 45 minutes, logs swamp the scheduler, and a bad build can reach every user.

Deep dives

How do we get every job a runner within 30 seconds — fairly?

At 11:00 about 1,700 jobs a minute arrive, each needing a fresh machine of the right OS, architecture and size, each owned by a team that thinks it is urgent.

Bad: runners poll the jobs table. Every idle runner runs SELECT … WHERE state = 'queued' … FOR UPDATE every two seconds: with 1,700 warm runners, 850 queries a second against your busiest table, all racing for the same oldest rows. SKIP LOCKED eases the lock fights, not the load, and a burst wakes every poller at once. Fine for 50 runners; at this scale the database becomes the queue, and a busy database becomes everyone’s problem.

Good: a queue per runner pool, long-polls and leases. The scheduler keeps an in-memory queue per pool of identical runners (linux-x64-4cpu), backed by the runs database. A runner long-polls its pool; the scheduler pops a job, writes the lease conditionally and answers. One write per job instead of hundreds of scans, and a dead runner is noticed within a lease period. What you pay: each pool is one FIFO, so when a refactor enqueues 3,000 jobs at 11:05, a payments hotfix a second later waits behind all of them.

Great: a matcher, fair-share lanes, priorities, leases with fencing, and warm pools. Five pieces, each small:

• Matching. A job’s runs-on labels must be a subset of a pool’s; among matches, pick the cheapest, so linux never lands on a GPU box.
• Lanes and fair share. Inside each pool, one lane per team. A picker serves the lane that has used least of its share (deficit round-robin), and each team’s concurrency cap bites only when the pool is saturated; an idle pool is anyone’s. Same idea as noisy neighbours and fairness.
• Priority classes on top: P0 for deploys, hotfixes and main-branch builds (a red main blocks everyone), P1 for pull requests, P2 for nightly work, aged upward so it never starves for hours.
• Leases with fencing. A 60-second lease, renewed every 15. When it expires, the job is requeued with a new token, and every runner write — heartbeat, log chunk, completion — carries its token, so a partitioned old runner gets 409 instead of marking job 42 “succeeded” after its replacement failed it. Retry only infrastructure failures, at most three times; a failing test is a result.
• Autoscaling on demand, not CPU. Scale each pool on queue depth and arrival rate, keep about 60 seconds of demand warm (1,700 runners at peak), and pre-warm before 11:00 and 16:00 from yesterday’s curve (see autoscaling done right).

The fair-share piece is easiest to see in the bad case it prevents:

The six minutes: if the burst can claim about 500 extra runners a minute, 3,000 jobs take six minutes to drain. On the right, the P0 lane goes first, so the hotfix takes the next free runner.

Two guardrails complete the scheduler. Hard limits stop runaway work: a job timeout (GitHub defaults to 360 minutes; 60 suits an internal platform), the 256-job matrix cap, a maximum queue wait (GitHub: 24 hours) and a maximum run lifetime including approvals (GitHub: 35 days). State is cheap to rebuild: the runs database is the source of truth, so a restarted scheduler replays queued jobs and re-arms lease timers. Two replicas, each owning half the pools, meet 99.9% without heroics.

How do we run untrusted code without leaking secrets?

A CI job is arbitrary code: a PR can change the build script, a dependency’s install hook runs whatever its author wrote, a compromised action runs with your token. Yet some jobs must push images and change production. Keep the two kinds apart.

Bad: long-lived shared runners with secrets in the environment. Big VMs run many jobs for weeks, with the Docker socket mounted and every secret exported as an environment variable. One malicious postinstall script uses /var/run/docker.sock to read other containers’ secrets, or plants a backdoor for the next deploy job. GitHub’s guidance is blunt: such runners “can be persistently compromised by untrusted code in a workflow”. You gain speed; you pay with the keys to production.

Good: one ephemeral container per job. A fresh pod, deleted after, so nothing persists. But jobs still share a kernel, and building images inside a container usually means privileged mode or a mounted daemon — the escape hatch you were closing.

Great: one microVM per job, trust tiers, no stored secrets.

• A VM boundary per job. Firecracker-style microVMs boot in a fraction of a second with a few megabytes of overhead, so each job affordably gets its own kernel and image builder, destroyed afterwards. GitHub recommends the same shape for self-hosted fleets: just-in-time runners that run at most one job.
• Trust tiers set by the platform, not the pipeline file. The trigger decides the tier; the tier decides pool, network, token permissions and which secrets exist at all.
• Short-lived credentials instead of secrets, through OIDC (figure below).
• Environments as a second gate. Production credentials belong to the production environment, whose jobs start only after its rules pass: allowed branch, a reviewer other than the author, freeze calendar.
• Masking is a seatbelt. Redaction is exact-match, so a base64-encoded secret or one inside a JSON blob slips through. Register derived values as secrets, and treat a leaked log as a leaked secret: delete and rotate.
• Egress control and provenance. Untrusted pools reach package mirrors, not the internal network or the metadata service. Trusted builds sign a provenance statement — which repository, commit and pipeline built this digest — with a key no step can read, and the deploy service refuses digests without it.

The first row is GitHub’s model: fork PRs get no secrets and a read-only token. The cache column stops cache poisoning: a low-trust run that could write main’s cache could plant a node_modules that a privileged build later executes.

Here is how a deploy job gets cloud access with no stored key, using OpenID Connect:

Each job gets a signed JWT with claims such as repository, ref and environment. The cloud role trusts only tokens whose subject is repo:food-app/payments:environment:prod, and its credentials last only for the job. Nothing long-lived exists to steal. What you pay: every cloud account must trust your issuer, and claim conditions must be precise — a policy matching any branch of any repository is a stored secret with extra steps.

How do we make builds fast?

Take a typical monorepo pull request built from scratch: 1 minute to check out, 6 to install dependencies, 18 to compile, 20 to test. 45 minutes, mostly redoing work another machine did an hour ago.

Bad: rebuild everything, every time. Correct and simple. If 15,000 of the day’s 50,000 pipelines are in the monorepo, it costs 675,000 runner-minutes — over 11,000 VM-hours, about ₹1.35 lakh a day — plus 45 minutes of waiting per push.

Good: dependency caches keyed by the lockfile. Key the cache on what decides its contents — OS plus a hash of the lockfile, linux-npm-9f2c41. Exact hit: restore and skip the install. Miss: restore the newest entry matching a prefix (linux-npm-), install the difference, and save under the new key. GitHub’s cache works this way: ordered restore-keys, the current branch searched before the default branch, eviction after 7 days unused, 10 GB per repository by default. You gain about 5 minutes per job; you pay in storage, bandwidth and an attack surface, since cache contents are not signed.

Great: a content-addressed build cache, affected-target testing, sharding and flaky-test quarantine.

• Remote build cache. Tools such as Bazel split a build into actions with declared inputs and outputs. An action’s key hashes its command, environment and input digests; an action cache maps the key to output digests, and a content-addressable store (CAS) holds the blobs. If the build is hermetic — no undeclared inputs like the clock or an unpinned compiler — the same key always means the same outputs, so any runner’s result is safe for every other. A PR touching 2% of targets recompiles 2%: 18 minutes becomes about 2.
• Affected targets only. Test only what depends on the changed files; unchanged test targets already have a cached “passed”. 20 minutes becomes about 5.
• Sharding by history. Split those 5 minutes over 4 runners using past timings: about 1.5 minutes of wall clock, for 4 runner startups.
• Image layers. Order Dockerfile steps from stable to volatile and use the registry as a layer cache, so only the top layers are rebuilt.

Put together for that monorepo pull request:

Across 15,000 monorepo pipelines that saves 35 runner-minutes each: about 8,750 VM-hours and ₹1 lakh a day, and 39 minutes of waiting per push. What you pay: hermeticity is real work (pinned toolchains, no network in build steps), the cache tier must serve several GB/s, and a cache bug poisons builds company-wide — so only trusted main-branch builds write the shared cache.

Flaky tests are a speed problem too. If 30 tests each fail 1% of the time at random, a pipeline goes red for no reason with probability 1 − 0.99³⁰ ≈ 26%: one PR in four re-runs, and engineers learn to ignore red. A test that fails then passes on the same commit is flaky by definition; quarantine it automatically (it runs but cannot fail the build) and give its owners a ticket with a deadline.

How do we stream logs live and keep them cheaply?

About 10,000 appends a second, 60 MB/s, 800 GB a day — and an engineer who wants the next line within 2 seconds.

Bad: upload at the end, or a row per line. End-of-job upload gives no live view, and a VM that dies takes its only copy of the log — the one you needed. A row per line is 400,000 × 20,000 = 8 billion rows a day.

Good: chunked appends, polled by viewers. The runner flushes every second or 64 KB with a sequence number; the log service appends to a per-job stream in a hot store; browsers poll. Live enough, but you pay for thousands of polling browsers and a hot store that only grows.

Great: separate hot and cold paths.

• Ingest checks the lease token (a zombie cannot write into the retry’s log), drops duplicate sequence numbers, and appends to a per-job stream in a hot tail — Redis streams or a log broker — capped near 1 MB per job: at most 14 GB for 14,000 live jobs.
• Fan-out nodes hold one upstream subscription per job however many people watch, and push chunks over server-sent events (real-time connections). The event ID is the sequence number, so a reconnecting browser sends Last-Event-ID: 311 and resumes without gaps.
• Compaction at job end: concatenate, compress about 8× (2 MB to 250 KB), write one object per attempt with an index of byte offsets per step, so “jump to the failed step” is one ranged read.
• Limits on the runner. Redact before bytes leave the VM, and cap each log (say 64 MB) so a runaway set -x loop cannot fill the hot tail.

Ingest is stateless and scales on its own, and the scheduler now sees about a tenth of its old traffic. Storage is easy: about 9 TB at a 90-day retention, deploy logs kept a year. Search across logs is the distributed logging system problem — feed it the compacted files.

How do we deploy without taking production down?

If even 1 build in 100 is bad in a way tests missed, 500 deploys a day send five bad builds toward production. The goal is not zero bad builds; it is that each is seen by few users, for a few minutes, and removed without a human.

Bad: deploy everywhere at once. A bad build reaches 100% of users together, someone notices on a dashboard minutes later, and rollback is another full deploy. On a sale evening, those minutes are lakhs of failed checkouts.

Good: rolling updates with health checks, or blue-green. A rolling update replaces pods in batches, waiting for readiness probes, so a build that crashes on start stops early. But a probe asks “is the process up?”, not “are payments succeeding?” — a build that adds 3% errors on one endpoint rolls all the way out (health checks done right). Blue-green flips traffic to a full-size new copy: rollback is a flip back, but you pay double capacity and every user meets the new version at once.

Great: progressive delivery — a canary judged against a baseline, then waves. Every service gets the same sequence, with its own numbers:

1. Staging with smoke tests, using the digest that will reach production.
2. Canary in one cell: 5% of that cell’s traffic, plus a baseline — fresh pods of the old version, the same size — so new pods are compared with equally new pods, not a fleet warm for days.
3. Analysis every minute for 10 minutes on a few metrics users feel: error rate, p99 latency, saturation and a business metric such as payment success rate.
4. Promote to 25% and 100% of the cell, then in waves — the region’s other cells, then the next region — each after a 15–30 minute bake (cell-based architecture).
5. On failure, abort: canary weight to zero, mark the digest bad so nothing promotes it again, page the owners.

In Argo Rollouts terms, the steps are a canary strategy and the judgment is an AnalysisTemplate:

strategy:
  canary:
    analysis:
      templates: [{ templateName: payments-health }]
      startingStep: 1
    steps:
      - setWeight: 5
      - pause: { duration: 10m }
      - setWeight: 25
      - pause: { duration: 10m }
      - setWeight: 100
---
kind: AnalysisTemplate
metadata: { name: payments-health }
spec:
  metrics:
    - name: error-rate-vs-baseline
      interval: 1m
      failureLimit: 3            # three bad measurements and the rollout aborts
      successCondition: result[0] <= 1.5   # canary errors at most 1.5× the baseline's
      provider:
        prometheus:
          query: |
            sum(rate(http_requests_total{job="payments",track="canary",code=~"5.."}[2m]))
            / sum(rate(http_requests_total{job="payments",track="canary"}[2m]))
            / (sum(rate(http_requests_total{job="payments",track="baseline",code=~"5.."}[2m]))
            / sum(rate(http_requests_total{job="payments",track="baseline"}[2m])))

A fixed ratio is a reasonable start. Spinnaker’s judge (Kayenta) goes further: per metric, a Mann-Whitney U test flags the canary only if it is worse with 98% confidence and by more than a set effect size; groups combine into a weighted score; a critical metric fails the canary alone; and if half the metrics have no data, the canary fails rather than passing silently. Statistics cut false alarms; you pay in tuning.

Check the NFR: three failed one-minute checks and the rollout aborts at minute 4. The stable version never scaled down, so abort is a traffic change taking seconds — about 4 minutes in all, well inside 10 — and 5% of traffic at 4% errors is 0.2% of requests for about 3 minutes, against 4% of all requests until a human noticed.

What you pay: full rollouts take an hour or more, metrics must be labelled by version (track=canary|baseline|stable), and low-traffic services may not see enough requests in 10 minutes to judge — they need longer canaries or synthetic traffic. Run one canary per service at a time so signals never mix.

How do we make deploys converge and never race?

Safe rollouts are about how traffic moves; this is about what is deployed, and whether clusters really end up in that state.

Bad: CI pushes a mutable tag. A runner runs kubectl set image payments=registry/payments:latest on each cluster. The runner holds production credentials; :latest now is not :latest an hour ago, so what runs is not what was tested and rollback means rebuilding; a hotfix and a regular merge can deploy at once, the older finishing last; and a manual kubectl edit drifts unnoticed.

Good: locks, immutable digests and promotion. Deploy payments@sha256:4be1…, never a tag, and promote the same digest through every environment: build once, deploy many times. Lock each (service, environment) like a GitHub concurrency group — one running, one waiting, newer waiting replaces older — but never cancel-in-progress in production, because killing a half-finished rollout is worse than finishing it. Refuse digests older than the current one unless it is an explicit rollback, which simply redeploys the previous healthy digest.

Great: GitOps — clusters pull desired state from Git. The deploy service stops talking to clusters. To promote, it commits the digest to a config repository, and an agent in each cluster — commonly Argo CD — continuously compares Git with the live objects and applies the difference.

What pull buys you:

• No production credentials in CI, only permission to commit to the config repo, whose branch protection is your deploy approval.
• Drift repair. With self-heal on, a manual kubectl edit is reverted soon after it is detected.
• A free audit log. Every deploy and rollback is a commit.
• Independence from CI. Clusters need the config repo, the registry and themselves — not the scheduler, runners or log service.

The costs. Lag: Argo CD polls every 120 seconds plus up to 60 of jitter, so up to 3 minutes; a webhook makes it near-instant, with the poll as a safety net. Rollback semantics: with auto-sync on, Argo CD syncs once per commit and refuses its own rollback — so rollback is a Git revert or a commit of the previous digest, which keeps Git the single source of truth. One more system to run in every cluster.

So the deploy service owns order and policy — locks, approvals, waves, freezes, provenance — and expresses each decision as a commit; the agents and rollout controller own convergence, canary steps and abort.

The final design

The top half is CI — a control plane and throwaway runners. The bottom half is CD — a deploy service that writes to Git, and production cells that pull from it.

Follow one change from a laptop to every user:

1. Trigger. Priya pushes to her PR. The trigger service dedupes the webhook; the planner reads .ci/pipeline.yaml at that SHA, expands the graph, cancels the superseded run and posts “queued” checks.
2. Schedule. Ready jobs enter their pool’s queue in her team’s lane, at PR priority.
3. Lease. A warm microVM runner receives job 42 with lease token 8 and heartbeats every 15 seconds; if it dies, the job is retried with a new token.
4. Build. The job restores the lockfile-keyed cache and uses the remote build cache; on main it trades an OIDC token for registry credentials and pushes payments@sha256:4be1… with signed provenance.
5. Watch. Log chunks stream to the log service and on to browsers over SSE, then are compacted into one object; the result becomes a check on the PR.
6. Hand over. After merge, the deploy stage goes to the deploy service, which checks approvals, the freeze calendar and provenance, and takes the (payments, production) lock.
7. Promote. It commits the digest to the config repo: staging first, then cell 1.
8. Converge. That cell’s GitOps agent sees the commit (webhook, or within 3 minutes) and syncs.
9. Canary. The rollout controller starts a 5% canary and a baseline, pulls the image from the registry, analyses every minute, and steps up or aborts to weight 0.
10. Report. Status flows back; the deploy service waits out the bake and commits the next wave — or marks the digest bad, releases the lock and pages the owners.

Check it against the requirements: queue time under 30 seconds because the warm pool holds 60 seconds of peak demand, with lanes so a hotfix never waits behind a refactor; a clean microVM per job and no stored cloud keys; logs live in about a second, off the scheduler; monorepo pipelines in 6 minutes instead of 45; bad builds removed in about 4 minutes after reaching 5% of one cell; and production that keeps running, and can be rolled back by a Git revert, while CI is down.

What interviewers look for by level

Follow-up questions

What if every commit to the 400-service monorepo triggers everything?

Then a one-line change runs 400 services’ jobs, and the 11:00 peak is mostly waste. Path filters are cheap but miss shared libraries; affected-target detection from the build graph builds and tests only the reverse dependencies of the changed files, answering the rest from the remote cache. Add a merge queue that tests batches of approved PRs on top of the latest main, so main stays green. Most changes touch a handful of services; plan capacity for the rare core-library change that touches all 400.

What if a deploy needs a database migration?

Never tie a schema change to a code rollout that can roll back automatically: the canary aborts in seconds, the migration does not. Use expand and contract: first a migration that only adds (a nullable column, a table) and works with old and new code; then code that writes both and reads the new; then a backfill; and weeks later, once nothing reads the old shape, the drop. Migrations are their own gated stage before the rollout, and automatic rollback only moves code to versions the current schema supports. See online schema and data migrations.

What if we need macOS, iOS or GPU builds?

These pools are small, slow to grow and expensive — macOS needs Apple hardware, GPUs are scarce — so a big warm pool is not an option. Give each its own queue and queue-time target, strict per-team caps so one app’s matrix cannot hold all 60 Macs, run expensive suites on main and nightly rather than every PR push, and cache aggressively. Publish utilisation and queue time per pool so capacity debates use numbers. GPUs get their own design in design a GPU job scheduler.

What if a global service must roll out across regions — during a sale freeze?

Waves handle geography: one cell, the rest of the region, then the next region, each after a bake, ideally while that region’s on-call is awake. A freeze is a deploy-service policy, not an email: during a Big Billion Days-style sale week, production promotions are blocked except hotfixes with two approvers and a smaller canary. CI keeps running, so the week after the sale needs care: ship the backlog in more, smaller waves, not one big release.

What if a popular third-party action is compromised?

Its code runs inside your jobs with whatever they can reach, so limit both. Pin actions to a full commit SHA, not a tag, which an attacker can move (GitHub also now offers workflow dependency lockfiles recording exact commits, transitive ones included). Keep an allow-list mirrored internally, default tokens to read-only, and since deploy access comes from OIDC, there is no long-lived secret to steal. The deploy service only promotes digests whose signed provenance names a trusted pipeline.

What if the CI/CD system is down during a production incident?

This is why the control plane and data plane are separate. Running services do not notice: clusters, registry and agents do not depend on the scheduler, runners or log service. Rollback still works — revert the config commit and agents converge within 3 minutes. Shipping a fix needs a break-glass path: a separately hosted, hardened builder that produces a signed image, and a procedure for on-call to commit its digest with two-person approval — audited, and rehearsed quarterly so it works when needed.

How to say it in the interview

“I’ll design an internal CI/CD platform for about 5,000 engineers: 50,000 pipelines and 400,000 jobs a day, peaking around 1,700 jobs a minute at 11:00. I’ll treat CI and CD as two systems: CI is batch compute judged by queue time; CD is a low-volume control system judged by blast radius and time to roll back.

A webhook reaches a trigger service; a planner reads the pipeline file at that exact commit and writes a graph of jobs. A scheduler keeps a queue per runner pool with a lane per team, so priorities and fair share decide who goes next when runners are scarce. Runners are ephemeral microVMs — one job, then destroyed. They long-poll for work and hold a lease with heartbeats and a fencing token, so a dead runner’s job is retried safely. Little’s law sizes the fleet: 1,700 a minute times 8 minutes is 14,000 concurrent jobs, and because every job needs a new machine, a warm pool of about 60 seconds of demand keeps queue time under 30 seconds.

Security: fork PRs get no secrets, caches can’t be poisoned by low-trust runs, and jobs get cloud access by trading an OIDC token for credentials that expire with the job. Builds are fast because caches are keyed by content — lockfile hashes for dependencies, a content-addressed remote build cache for actions — plus affected-target testing and flaky-test quarantine. Logs stream in one-second chunks to a log service with a hot tail for live viewers, then get compacted into object storage.

For deploys, CI builds an image once and names it by digest. A deploy service promotes that digest through staging, a 5% canary compared against a baseline with automated analysis, and waves of cells, aborting automatically. It writes desired state to a config repo, and an agent in each cluster pulls and reconciles. So there are no production credentials in CI, drift gets repaired, rollback is a commit, and production keeps running even when CI is down.”

Concepts used in this design

• Canary, blue-green and feature flags — the rollout strategies the deploy service orchestrates.
• Design a job scheduler — queues, leases and retries in a more general setting.
• Workflow engines — durable, long-running deploy state machines.
• Long-running jobs — heartbeats, progress and retries for work that takes minutes.
• Heartbeats — detecting dead runners.
• Distributed locking — leases, fencing tokens and per-environment deploy locks.
• Noisy neighbours and fairness — fair share between teams on shared runner pools.
• Autoscaling done right — scaling on queue depth and pre-warming for known peaks.
• OAuth 2.0 and OpenID Connect — the identity tokens behind keyless cloud access.
• Multi-tenancy and isolation — running untrusted code next to trusted code.
• Blob storage — artifacts, caches and compacted logs.
• Real-time connections: long polling, SSE and WebSockets — runner long-polls and live log streaming.
• Health checks done right — why readiness probes alone don’t catch bad builds.
• Cell-based architecture — the units that deploy waves move through.
• Online schema and data migrations — expand and contract alongside code deploys.
• Control plane vs data plane — why production must not depend on CI.
• Reliability, SLIs, SLOs and error budgets — choosing canary metrics users actually feel.

References and further reading

1. GitHub Docs: Actions limitsdocs.github.com35-day workflow runs, 256 jobs per matrix, 6-hour jobs on hosted runners (5 days self-hosted), a 24-hour queue limit, 100 pending runs per concurrency group and 10 GB of cache per repository.
2. GitHub Docs: Workflow syntax for GitHub Actionsdocs.github.comneeds, strategy.matrix with fail-fast on by default, the 360-minute default job timeout, and concurrency groups: one running, one pending, cancel-in-progress and queue: max.
3. GitHub Docs: Secure use referencedocs.github.comWhy log redaction is not guaranteed, the pull_request_target risk, pinning actions to full commit SHAs, and why self-hosted runners should be ephemeral or just-in-time.
4. GitHub Docs: Events that trigger workflowsdocs.github.comFork pull requests get no secrets and a read-only token; pull_request_target runs in the base branch’s privileged context.
5. GitHub Docs: OpenID Connectdocs.github.comA signed per-job JWT whose sub, ref and environment claims a cloud role can match, exchanged for credentials that expire with the job.
6. GitHub Docs: Dependency caching referencedocs.github.comkey and restore-keys matching, branch-scoped caches, read-only caches for low-trust triggers against cache poisoning, 7-day idle eviction and the 10 GB default.
7. Bazel: Remote cachingbazel.buildAn action cache from action hashes to results plus a content-addressable store of outputs, shared safely between machines when builds are reproducible.
8. Argo CD: Automated sync policyargo-cd.readthedocs.ioOne automated sync per commit SHA, self-heal of live drift, no rollback while auto-sync is on, and a 120 s reconciliation interval plus up to 60 s jitter.
9. Argo Rollouts: Analysis and progressive deliveryargo-rollouts.readthedocs.ioAnalysisTemplates with successCondition, interval and failureLimit; a failed analysis aborts the rollout and sets the canary weight back to zero.
10. Spinnaker: How canary judgment worksspinnaker.ioKayenta’s default judge: a Mann-Whitney U test at 98% confidence per metric, weighted group scores, critical metrics, and failing when half the metrics have no data.