Deployment strategies: canary, blue-green and feature flags

A deployment strategy decides how a new version replaces the old one in production: all at once, a batch at a time, by switching between two fleets, or by giving a small slice of traffic to the new version first. Because most outages begin with a change, the goal is to limit how many users a bad change can reach, notice it fast and make undoing it cheap. Feature flags add a second lever: you deploy code dark and release the behaviour separately.

Picture it first: a new recipe in 200 kitchens

A biryani chain runs 200 kitchens and wants to change its recipe. It has six options:

• Close everything for a morning, retrain every cook, reopen. Simple, but nobody eats.
• Retrain 20 kitchens a day. No closures, but for ten days customers get either recipe.
• Build a second kitchen beside each one and switch which one feeds the counter; switch back if complaints spike. You pay for two kitchens.
• Serve the new recipe from 2 kitchens and compare their complaints with 2 similar kitchens on the same day, not with last Sunday.
• Cook it in the back, taste it, throw it away. Customers never see it.
• Put the new dish on a hidden menu: staff first, then 5% of app customers.

Map it back: kitchens are servers or cells, the recipe is a version, complaints are errors and slow responses, and the options are recreate, rolling update, blue-green, canary, shadow and feature flags.

Why change is the risk

Google’s SRE book says it in its introduction, under change management: “SRE has found that roughly 70% of outages are due to changes in a live system.” Binaries, config pushes, flag flips and schema changes all count. Its remedy is three automated practices: progressive rollouts, quick and accurate detection, and safe rollback.

Those map to three numbers you control: blast radius (the share of traffic a bad version reaches), time to detect and time to recover. Harm is roughly exposure × failure rate × (detect + recover time). The SRE workbook’s example: a release that fails 20% of requests, canaried on 5% of traffic, shows up as a 1% overall error rate.

Put numbers on it with a 99.9% monthly SLO: the budget is 0.001 × 43,200 min ≈ 43 minutes of full outage. A bad build that fails every request fleet-wide for 10 minutes before an alarm, plus 5 minutes to roll back, burns 15 minutes, about 35% of the month. On a 1% canary the same bug costs 0.01 × 15 = 0.15 minutes, a hundred times less.

Deploy is not release

Deploying puts a new build on servers. Releasing lets users experience new behaviour. Treating them as one event is the habit to break.

With a feature flag you deploy code that is switched off, confirm the deploy is harmless, then release by turning the flag on for staff, then 1%, 10% and 100% of users. If the feature misbehaves, you switch it off in seconds without rolling back the rest of the build. Facebook described this in 2017: web releases went to employees, then 2% of production, then 100% over a few hours, and many changes sat behind its Gatekeeper system so a problem could be switched off instead of reverted.

You gain release timing the product team controls and a fast undo for features. You pay with two code paths per flag, more combinations to test and flags to delete later.

Recreate and rolling updates

Recreate: stop everything, start everything

Kubernetes’ Recreate strategy kills every old pod and waits for them to go before creating new ones. Downtime is shutdown plus startup plus warm-up, and rollback is another recreate with another outage. Use it where two versions must never run together (a singleton consumer, a job that rewrites a file format) or where downtime is free (dev, nightly batch workers).

Rolling update: a batch at a time

RollingUpdate is the Kubernetes default. Two knobs bound it: maxSurge, how many pods may exist above the desired count, and maxUnavailable, how many may be missing. Both default to 25%; surge rounds up and unavailable rounds down. For 10 replicas that is 3 and 2: never more than 13 pods, never fewer than 8 available.

You gain no second fleet, only 25% extra pods at peak, and no downtime if readiness probes and draining work (health checks done right covers both). If new pods never become ready, the controller stops after removing at most maxUnavailable old pods, so a build that cannot start cannot take the service down.

You pay in three ways:

• Mixed versions. For the whole rollout, v1 and v2 serve together and read each other’s data. AWS’s rollback-safety article has a sharp example: raising a heartbeat interval from 5 to 10 seconds looks like a one-number change, but old servers expecting a heartbeat every 5 seconds start closing connections to new ones.
• Readiness is not correctness. A build that returns wrong answers with HTTP 200 passes readiness and rolls to 100%.
• Rollback is another rollout. kubectl rollout undo re-rolls the previous pod template at the same pace, minutes not seconds, and leaves a ConfigMap you edited in place edited. A rollout stalled past progressDeadlineSeconds (600 by default) only gets a ProgressDeadlineExceeded condition; something else must decide to roll back.

AWS’s pipelines replace at most a third of a region’s boxes at a time, which every service can absorb because each is sized to lose an Availability Zone, and they roll back a third at a time even when they deploy 5% at a time: undo should be faster than do.

Blue-green: two fleets, one switch

Martin Fowler’s description: keep two production environments, as identical as possible. One, say blue, is live. You deploy the new release to green, test it there, then switch the router so all requests go to green. Blue sits idle on the old version. If anything goes wrong, you switch back. On the next release the roles swap.

The switch can be a load balancer moving to a new target group, a Kubernetes Service selector changing from version: blue to version: green, or a DNS record. Avoid DNS: changes spread only as caches expire, and some clients hold old answers past the TTL.

You gain an atomic cut-over (users never hit a mix of versions), a green fleet tested in production before it takes traffic, and the fastest rollback there is while blue still runs. As Fowler notes, every release also rehearses a failover.

You pay for it:

• Double capacity while both fleets run, the main downside in the SRE workbook’s view. Scaling blue down after a bake saves money and ends the instant-rollback window.
• Everyone at once. The flip moves 100% of traffic, and comparing green-after with blue-before is a before/after test, which the workbook calls risky. Shift weight gradually between blue and green and you have a canary.
• Draining. In-flight requests and WebSockets stay on blue until they finish; an AWS Application Load Balancer waits 300 seconds by default before it finishes deregistering a target.
• Cold start. Green’s caches and connection pools are cold; warm them before the flip.

Canary releases

The SRE workbook defines canarying as a partial, time-limited deployment of a change plus an evaluation of it; the part that gets the change is the canary, the rest is the control. Who goes first can be a random sample, employees, or one region.

Splitting the traffic

• By replica count. Run 1 v2 pod beside 99 v1 pods behind one load balancer. Simple, but pod count limits the steps: Argo Rollouts without a traffic router approximates weights with replicas, so 10 replicas give steps of 10%.
• By weighted routing. A service mesh, ingress or load balancer sends exactly 1% to a small v2 deployment. Route by a hash of the user ID, so a user doesn’t bounce between versions.

Steps, pauses and gates

A typical plan ramps 1% → 5% → 25% → 50% → 100%, pausing at each step while an analysis runs. In Argo Rollouts:

# Argo Rollouts: canary steps with a background analysis
strategy:
  canary:
    canaryService: checkout-canary
    stableService: checkout-stable
    trafficRouting:            # weighted routing makes 1% possible with few pods
      istio:
        virtualService:
          name: checkout
    analysis:                  # runs during the steps; a failure aborts to 0%
      templates:
        - templateName: canary-vs-baseline
    steps:
      - setWeight: 1
      - pause: {duration: 10m}
      - setWeight: 5
      - pause: {duration: 30m}
      - setWeight: 25
      - pause: {duration: 30m}
      - setWeight: 50
      - pause: {duration: 30m}

If the analysis fails, the rollout aborts and the canary weight drops back to zero. Nobody has to be watching.

Is the canary healthy?

Compare the canary with a baseline, not with yesterday. The SRE workbook calls before/after evaluation risky because time moves metrics more than most releases do: a Monday release compared with the weekend mostly measures the weekend. Spinnaker’s guidance for Kayenta, the analysis engine built by Netflix and Google, goes further: start a fresh baseline on the current version at the same time, size and traffic as the canary, so warm caches and uptime cancel out.

Watch a few signals, split by version:

The workbook suggests a dozen metrics at most, since noisy ones teach operators to ignore the canary, and windows no longer than the canary itself. Then decide with statistics. Spinnaker’s default judge runs a Mann-Whitney U test per metric, flags a metric only at 98% confidence, and rolls the results into a score out of 100; suggested starting thresholds are 75 (marginal) and 95 (pass), with at least 50 data points per metric.

Keep an absolute guard too. Canary and control share databases and caches, so a bad canary can drag the control down while the comparison looks even; check SLOs as well. If the error budget is already spent, don’t start.

Why 1% of a quiet service tells you nothing

Statistics need volume. Take a service at 2,000 requests a second with a 0.1% error rate. A 1% canary gets 1,200 requests a minute and expects 1.2 errors a minute, so in one minute a release that doubles the error rate looks like luck. Over 10 minutes it sees 12,000 requests and expects 12 errors, give or take √12 ≈ 3.5. A doubling gives 24, about 3.5 standard deviations high: clear. At 50 requests a second, the same 1% sees 300 requests in 10 minutes and expects 0.3 errors; only a catastrophe shows.

Shadow traffic and dark launches

Shadowing (also called mirroring or teeing) copies live requests to the new version and throws its responses away. Users only see v1’s answers, so you can compare v2’s responses, latency and resource use on real traffic at no risk to them. Istio’s mirroring sends copies “fire and forget” and appends -shadow to the Host header so the receiver can tell.

The catch is side effects. A mirrored “place order” charges a card twice or sends a second SMS unless every outbound write, payment and message is stubbed or sent to a sandbox. Shared caches mislead too: the workbook notes that a shared cache inflates the hit rate and spoils performance numbers. Shadowing suits read paths, rewrites behind the same API and new storage engines checked by diffing results.

A dark launch does the same from real clients: before Facebook Chat launched in 2008, its pages quietly connected to the chat servers, queried presence and simulated sends without drawing any chat UI.

Feature flags

A flag is a decision point in deployed code, such as if flags.on("new-checkout", user). Pete Hodgson’s article on martinfowler.com sorts flags by how long they live and how dynamic the decision is:

Manage each kind differently: a release toggle gets a removal ticket the day it is born, an experiment belongs to a product owner, and an ops toggle must be fast to flip and regularly tested. Hodgson calls a long-lived ops toggle a manually managed circuit breaker.

Percentage rollouts by hashing

To give 5% of users a feature, don’t roll a random number per request, or the feature will flicker. Hash the user into a fixed bucket:

import hashlib

BUCKETS = 100_000  # 1% = 1,000 buckets

def bucket(flag_key: str, salt: str, user_id: str) -> int:
    digest = hashlib.sha256(f"{flag_key}.{salt}.{user_id}".encode()).hexdigest()
    return int(digest[:15], 16) % BUCKETS

def is_on(flag: dict, user_id: str) -> bool:
    if flag["killed"]:                    # the kill switch wins
        return False
    if user_id in flag["always_on"]:      # staff, beta testers
        return True
    return bucket(flag["key"], flag["salt"], user_id) < flag["percent"] * 1_000

LaunchDarkly’s documented algorithm has the same shape: a SHA-1 of the flag key, a per-flag salt and the user key, mapped onto 100,000 buckets. The same user always gets the same answer. Raising 5% to 20% keeps buckets 0–4,999 on and adds 5,000–19,999, so nobody loses the feature. And because the flag key is in the hash, each flag picks a different 5%, so one unlucky group doesn’t get every experiment.

Evaluate locally, update by push

Never call the flag service per request: a page that checks 20 flags would add 20 round trips and a hard dependency. LaunchDarkly’s server-side SDKs download the full rule set at startup, keep it in memory, stream changes, and evaluate each flag in-process. If the flag service goes down, processes keep their last known rules; one that starts with none uses the defaults in code, so make those the safe ones. Browser and mobile SDKs receive evaluated results instead, because rules can contain user lists.

Flags are changes too

A flag flip changes production exactly as a deploy does, and can reach every server in seconds. AWS ships feature flags and runtime config through their own pipeline with the same staged rollout and automatic rollback as code, because “a bad feature flag configuration change can have an impact on production just as a bad application code change can.” Cloudflare’s 2019 outage, told in backend failure modes, began with a rule change that went global at once. A workable rule: a move back to known-good behaviour (a kill switch) can be instant; a move towards new behaviour is staged like a deploy.

Flag debt: Knight Capital, 2012

Flags are cheap to add and expensive to keep: each one doubles the paths you could test, and stale ones hide landmines. The SEC’s 2013 order on Knight Capital shows how far that goes.

Knight’s new order-routing code reused a flag that had once switched on “Power Peg”, a function unused since 2003 but never deleted. A technician copied the new code to seven of the eight servers, and nobody reviewed the deployment. On August 1, 2012, orders carrying the reused flag reached the eighth server, which ran the dead code and sent child orders without stopping. The team then uninstalled the new code from the seven good servers, which made it worse: the flag now triggered Power Peg there too. In about 45 minutes Knight got more than 4 million executions in 154 stocks and lost over $460 million.

The lessons: never reuse a flag, delete dead code, verify every host got the deploy, and never roll back code without the config it shipped with. Hodgson’s cures for flag debt: a removal task for every release toggle, expiry dates, “time bombs” that fail the build when a flag outlives its date, and a cap on the number of flags.

Choosing a strategy

These are not rivals. A typical setup uses a canary or rolling update for every build, flags for every user-facing feature, shadowing for risky rewrites and blue-green where an atomic switch matters.

Waves: never everywhere at once

A good canary inside one region still lets a bad change reach every region if you promote it too fast. AWS states the first objective of its production deployments as preventing negative impact in multiple Regions at the same time, and in multiple Availability Zones of the same Region. Its typical pipeline:

1. One box first in each region or zone: a single VM or container, or a small share of Lambda invocations, typically serving at most 10% of that region’s or zone’s requests.
2. Wave 1: one region with little traffic, one Availability Zone or cell at a time.
3. Wave 2: one busy region, where customers are likely to exercise every new code path.
4. Later waves grow to three regions, then up to 12, then the rest, still one zone or cell per region at a time.
5. Bake after each stage: at least an hour after each one-box stage, at least 12 hours after the first regional wave, and two to four hours after each later wave, plus a minimum number of requests.

One aggregate alarm triggers rollback: fault rate, latency percentiles, CPU, memory, log errors, health checks, synthetic tests and the alarms of neighbouring services. One-box stages also alarm on that box alone, since its small share barely moves service-wide numbers. No deploy starts while an alarm fires, and deploys run only when people are around to help. The result: every region in four or five business days, with nobody watching.

That pace suits infrastructure other companies build on; a consumer web app with every feature behind a flag can move faster. Inside a region, cells give you the same staging with a smaller unit.

Rollback, roll forward and data

Rolling back means redeploying the last known-good artifact; AWS keeps deployment packages immutable and checksummed, so a rollback restores exactly what ran before. Rolling forward means shipping a fix. Default to rolling back: it is a tested path and needs no diagnosis, while every minute spent debugging is a minute of users failing.

When is rollback unsafe? AWS found the most common reason is a protocol change: a new version writes data the old one cannot read. Say v2 starts compressing what it stores. Once v2 has written compressed records, v1 cannot read them, and rolling back breaks reads. So every change should be N-1 compatible: the old version must read what the new one writes, and the new one must read what the old one wrote.

The fix is a two-phase deployment:

1. Prepare. Deploy v2, which reads both formats but still writes the old one. Rolling back is safe, because nothing new exists yet. Check that every server has it (Knight’s eighth server is what happens when you don’t).
2. Bake for a few days, so you won’t need to go back past this point.
3. Activate. Deploy v3, which writes the new format. Rolling back to v2 is safe, because v2 reads it.
4. Backfill old records, and only then remove the old reader.

AWS adds an ordering rule for services that talk to each other: readers before writers when rolling forward, writers before readers when rolling back. Schema changes follow the same pattern, expand/contract, covered step by step in online schema and data migrations. A deploy that “includes a migration” becomes several small deploys, each of which can be undone.

Clients you cannot roll back

Code on users’ phones can’t be recalled. Apple’s phased release spreads an automatic update over seven days (1%, 2%, 5%, 10%, 20%, 50%, then 100% of users) and can be paused for up to 30 days in total, but not reversed, and anyone can download the update manually. So ship new features dark behind remote flags you can kill, keep the backend serving old app versions for months, and retire an old API only once its client usage is low enough.

Cheat sheet

How to say it in the interview

“Since most outages come from changes, I design deployments around three numbers: how many users a bad version can reach, how fast I notice and how fast I undo it. Every build goes out progressively: one box, then a 1% canary judged against a fresh baseline on error rate, p99 latency, saturation and orders per minute, with automatic rollback if it fails. Then 5, 25, 50 and 100% within a region, and region by region in waves with bake time. User-facing features ship dark behind flags, so the release is a flag ramp I can kill in seconds. Every change stays backward compatible, so rollback is always safe, and schema changes go expand-then-contract.”

Blue-green or canary? Blue-green gives an atomic switch and the fastest rollback, but exposes everyone at once and costs a second fleet. A canary exposes 1–5% and judges it against a baseline, but needs traffic splitting and per-version metrics. For a busy service I’d canary; for a small app with a risky cut-over, blue-green. They combine: shift weight gradually between blue and green.

How do you roll back a deploy that included a migration? Design it so the migration never needs rolling back. Expand first with an additive, backward-compatible change, deploy code that works with both shapes, switch over, and contract in a later release. Rolling back the code is then safe because the old version still works with the expanded schema. Destructive steps go last, after a bake.

How do you decide a canary is healthy? Against a baseline on the old version, started at the same time and size, not against yesterday. A few SLI-based metrics per version (errors, p99, saturation, a business metric), windows no longer than the canary, enough requests for significance, a statistical judge, and an absolute SLO guard.

What’s the risk of feature flags? Test combinations grow with every flag, stale flags hide dead code (Knight lost over $460 million partly through a reused flag), flag flips can skip the deploy pipeline, and a flag service on the request path becomes a dependency. Evaluate locally, stage flag changes like code, test kill switches and delete release flags within weeks.

Where to go next

• Online schema and data migrations: expand/contract and backfills, so rollback stays safe.
• Health checks done right: readiness, draining and graceful shutdown that make rolling updates error-free.
• Observability: logs, metrics and traces: the per-version metrics every canary judge needs.
• Reliability: SLIs, SLOs and error budgets: the budget that gates releases.
• Cell-based architecture: deployment waves cell by cell.
• Backend failure modes: bad pushes, poison inputs and the Cloudflare config outage.
• Migrating a legacy system: shifting traffic route by route for a whole rewrite.
• Design a CI/CD deployment system: build the pipeline that runs all of this.

References and further reading

1. Site Reliability Engineering, Chapter 1: Introduction (Google)sre.googleThe “roughly 70% of outages are due to changes in a live system” finding, the trio of progressive rollouts, fast detection and safe rollback, and error budgets that can halt releases.
2. The Site Reliability Workbook, Chapter 16: Canarying Releases (Google)sre.googleCanary versus control, the 5% × 20% error-budget arithmetic, why before/after evaluation is risky, choosing metrics, and blue-green and traffic teeing as relatives.
3. Automating safe, hands-off deployments (Amazon Builders’ Library)aws.amazon.comOne-box stages, rolling batches of a third, regional waves, bake times, aggregate rollback alarms, deployment time windows, and feature flags shipped through their own pipeline.
4. Ensuring rollback safety during deployments (Amazon Builders’ Library)aws.amazon.comWhy protocol and format changes break rollback, the prepare-then-activate two-phase deployment, and reader-before-writer ordering.
5. BlueGreenDeployment (Martin Fowler)martinfowler.comTwo identical environments behind a router, rollback by switching back, rehearsing failover on every release, and separating schema changes from application upgrades.
6. CanaryRelease (Danilo Sato, martinfowler.com)martinfowler.comRouting a subset of users to the new version, ways to choose them, canary versus A/B testing, and why installed and mobile software is hard to canary.
7. Feature Toggles (aka Feature Flags) (Pete Hodgson, martinfowler.com)martinfowler.comRelease, experiment, ops and permissioning toggles by longevity and dynamism, consistent cohorts, toggle configuration and the carrying cost of flags.
8. Kubernetes documentation: Deploymentskubernetes.ioRecreate and RollingUpdate strategies, the 25% maxSurge and maxUnavailable defaults and their rounding, rollout undo, and what happens when a rollout stalls.
9. Spinnaker documentation: Best practices for configuring canaryspinnaker.ioComparing a canary with a fresh baseline rather than production, at least 50 data points per metric, and starting score thresholds for Kayenta.
10. SEC order: In the Matter of Knight Capital Americas LLC (2013)sec.govThe primary account of August 1, 2012: a repurposed flag, a deployment that missed one of eight servers, and an attempted rollback that made it worse.