Autoscaling done right

Autoscaling adds and removes capacity automatically so the fleet follows demand: you pay for the quiet night rather than the dinner peak, and each instance stays below the load where its latency climbs. Done right, it is a control loop with a well-chosen signal, headroom for its own lag, and hard limits. Done wrong, it scales too late for the spike, too far into a struggling database, or without end on a bug.

Picture it first: the food court on final night

A food court beside a stadium runs eight cooks on a normal evening. On cricket-final night the manager watches the token queue and, when it passes 40, phones for extra cooks. They are twenty minutes away, take ten more to light their stations, and cook slowly at first. By the time they are at full speed, the innings-break rush has peaked and the queue is out of the door.

A better manager calls cooks in an hour early, because the break comes on a clock. She watches tokens waiting per cook, not how busy the cooks look: a cook waiting for the one shared tandoor looks idle but can’t take more orders. And she knows ten more cooks only lengthen the line at the tandoor.

Map it back: cooks are instances or pods, the manager is the autoscaler, tokens per cook is the scaling signal, travel and set-up time is the scaling lag, calling cooks early is scheduled scaling, and the tandoor is a stateful dependency such as the database. Sending cooks home whenever the queue dips, then calling them back, is flapping.

What autoscaling is for, and what it isn’t

Autoscaling has two jobs.

Cost. Most traffic has a daily shape. Say an API sees 1,500 requests/s at 4 am and 15,000 at the dinner peak, averaging 6,000, and one pod handles 250 requests/s at its target. Sized for the peak, you run 60 pods all day: 60 × 24 = 1,440 pod-hours. Following demand averages 6,000 ÷ 250 = 24 pods, about 576 pod-hours, 60% less.

Availability. Every instance has a load beyond which queues build and latency climbs steeply. Autoscaling keeps each instance below that knee as demand grows.

It cannot:

• Absorb a spike shorter than its own lag. If new capacity takes four minutes, a surge that peaks in one lands entirely on the instances you already have.
• Scale a stateful bottleneck. More app servers only add connections and queries to a database at its limit.
• Fix a retry storm or a bug. It buys machines for the wrong work. Google’s SRE workbook describes a CPU-burning bug that grew a job “until all available quota is wasted”.
• Scale past its limits: your max, your account quota and the region’s spare machines, often all on the day you need them most.

Choosing the scaling signal

A good scaling signal has three properties:

1. It measures load per instance and falls in proportion as you add instances. AWS makes this a hard rule for target tracking: total requests at the load balancer don’t change when you add instances, so a loop steering on them never settles.
2. It moves early, before users feel the problem.
3. It is measured every 10–60 seconds, not every five minutes.

CPU utilisation

CPU is the default everywhere because it is free to measure. It works for compute-bound services (image resizing, serialisation, TLS, CPU inference) where each request costs roughly the same CPU. It misleads when:

• The service is I/O-bound. By Little’s law, in-flight requests = arrival rate × latency. At 400 requests/s and 50 ms that is 20 in flight per pod; if the database slows to 200 ms it is 80, and a 64-thread pool is exhausted while CPU reads 25%.
• Requests are wrong. The HPA measures utilisation as a percentage of each container’s CPU request, and with no request set it takes no action on CPU at all.
• The work isn’t serving users: garbage collection, JIT warm-up, a busy loop in a bad release, retries.

Requests or in-flight requests per instance

Requests per second per instance (AWS’s ALBRequestCountPerTarget, or a per-pod rate through a custom-metrics adapter) is a leading signal: it rises the moment traffic does. Set the target from a load test: if one pod holds its SLO up to 350 requests/s, target 250. It assumes every request costs the same. If the mix can shift towards expensive calls, use in-flight requests per pod, where slow requests count for longer.

Latency: a guardrail, not a target

Latency is what users feel, but a poor control signal:

• It isn’t proportional to capacity. It stays flat, then shoots up near saturation, so the loop learns of overload only once overloaded. AWS lists load-balancer latency among the metrics that don’t work for target tracking.
• It rises for reasons more instances don’t fix. A slow database raises latency on every replica, and scaling out adds callers to it.

Alert on p99; steer on load per instance.

Queue backlog per worker

For queue consumers, the signal is work waiting per worker, measured against how long a message may wait:

acceptable backlog per worker = acceptable latency × per-worker throughput

workers needed = backlog ÷ acceptable latency ÷ per-worker throughput

A notification service promises delivery within 30 seconds, and one worker sends 50 messages/s, so each worker can carry 30 × 50 = 1,500 waiting messages. With 36,000 queued and 12 workers (3,000 each), you need 36,000 ÷ 30 ÷ 50 = 24 workers, exactly what target tracking at 1,500 per worker computes: 12 × 3,000 ÷ 1,500 = 24. Raw queue length won’t do: it ignores how many workers are draining it.

Two refinements:

• Watch the arrival rate. At 2,000 messages/s you need 2,000 ÷ 50 = 40 workers just to keep pace, and backlog-based scaling gets there only after the backlog has grown. Take the larger answer.
• Respect the ceiling. A Kafka consumer group can’t keep more consumers busy than the topic has partitions, so KEDA’s Kafka scaler caps replicas at the partition count by default.

Policy types

Target tracking is a thermostat: pick a metric and a target (“CPU at 60%”, “250 requests/s per instance”) and the autoscaler computes the capacity that brings the metric back. It is AWS’s recommended default and the HPA’s model. With several target-tracking policies, an EC2 group scales out if any asks and in only if all agree, and it rounds up when adding (1.5 instances becomes 2).

Step scaling maps alarm bands to fixed adjustments: add 10% of the group at 70% CPU, 30% at 85%, remove 10% below 40%. You control the reaction to big breaches but tune the bands by hand. Its older form, simple scaling, waits out a 300-second cooldown after each change; AWS recommends against it.

Scheduled scaling sets capacity at a time you name, as a cron expression with a time zone. No metric and no lag, but it knows only what you tell it, and an EC2 scheduled action can start up to two minutes late.

Predictive scaling learns the schedule for you. On EC2 it needs 24 hours of history (two weeks is better), analyses up to 14 days, and forecasts hourly for the next 48 hours, refreshed every 6 hours. You run it forecast-only first. It only scales out, at the start of each hour unless you ask it to launch earlier, and does nothing for a surprise.

With several policies active, the group runs at the largest capacity any of them asks for. The usual stack: a schedule or forecast sets the floor, target tracking handles surprises, and a hard max caps both.

Kubernetes HPA: the formula and its defaults

The HorizontalPodAutoscaler (HPA) is target tracking for pods. Every 15 seconds by default, it averages the metric over the target’s pods and computes:

desiredReplicas = ceil(currentReplicas × currentMetricValue ÷ desiredMetricValue)

With 6 pods averaging 90% CPU against a 60% target: ceil(6 × 90 ÷ 60) = 9 pods, and the same work over 9 pods is 6 × 90 ÷ 9 = 60% each.

What matters in production:

• Tolerance. If the ratio is within 0.1 of 1.0, nothing happens: 64% against a 60% target is 1.07, so no change.
• New pods count conservatively. On CPU, not-yet-ready pods are set aside, and CPU from a pod’s first 5 minutes is ignored unless it was ready throughout, so a JVM’s start-up burst doesn’t trigger more scaling.
• With several metrics, the largest answer wins.
• Out fast, in slow. Up: no window, and every 15 seconds it may add 100% of current pods or 4 pods, whichever is more. Down: it acts on the highest recommendation of the last 300 seconds. The behavior field changes both.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: checkout-api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: checkout-api
  minReplicas: 6            # covers base load across three zones
  maxReplicas: 24           # set by the database connection budget
  metrics:
    - type: Resource
      resource:
        name: cpu
        target: { type: Utilization, averageUtilization: 60 }
    - type: Pods
      pods:
        metric: { name: http_requests_in_flight }
        target: { type: AverageValue, averageValue: "20" }
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - { type: Percent, value: 10, periodSeconds: 60 }

This adds an in-flight metric that catches I/O saturation, and removes at most 10% of pods a minute after the five-minute window. Recent Kubernetes versions also accept minReplicas: 0 when scaling on an object or external metric such as queue length.

Pods, nodes and the two-level problem

The HPA only creates pods. Whether they run depends on a node autoscaler; an optional VPA sizes them.

Vertical Pod Autoscaler

The VPA learns each container’s real CPU and memory use, including out-of-memory kills, and recommends requests. Off only recommends, Initial sets new pods, Recreate evicts pods to apply, and InPlaceOrRecreate resizes running pods where it can. Its value is correct requests, which both the HPA’s utilisation and node packing depend on.

Node autoscalers

When no node has room for a new pod’s requests, the pod is unschedulable, and a node autoscaler adds machines. Cluster Autoscaler grows pre-defined node groups, checks for unschedulable pods every 10 seconds, and removes a node after it has been unneeded for 10 minutes. Karpenter picks instance types for each batch of pending pods and launches VMs directly. Both decide from requests, not usage.

The two-level problem

Pod and node scaling are two loops in series. With room on a node, a new pod runs in seconds. Without it, the pod waits for a VM to boot, join and pull images: “usually about 5 minutes” in total per the Cluster Autoscaler FAQ, 3–4 of them the VM on GCE.

The fix is placeholder pods: pause containers that only reserve CPU and memory, at a priority below every real workload. A real pod that can’t fit evicts them and starts in seconds; the evicted placeholders go pending and trigger the node autoscaler, so the boot happens in the background. Give them a negative priority no lower than the Cluster Autoscaler’s cutoff of −10, or they won’t trigger scale-up. You pay for the buffer all day (four 2-vCPU placeholders are 8 idle vCPUs), so size it to a surge’s first five minutes.

KEDA: event-driven scaling and scale-to-zero

KEDA extends the HPA to event sources: queues, Kafka lag, Prometheus queries, cron. It handles activation (0 ↔ 1 replica) itself and feeds an HPA it creates for 1 ↔ N. By default it polls each trigger every 30 seconds and scales to zero after 300 idle seconds. Idle then costs nothing, but the first message after a quiet spell waits for a poll, a pod start and perhaps a node. Keep one replica on any path a user waits on; serverless architecture has the same trade-off.

The lag chain

Between load rising and new capacity serving it sits a chain of delays:

1. Metrics. The standard metrics-server install scrapes every 15 seconds. EC2 metrics arrive every 5 minutes, or every minute with detailed monitoring; target tracking accepts custom metrics down to 10 seconds.
2. Evaluation. The HPA runs every 15 seconds; a CloudWatch alarm needing three 1-minute breaches adds up to 3 minutes.
3. Provisioning. Seconds on a node with room, minutes for a new VM (AWS suggests assuming 300 seconds until you measure).
4. Image pull. A 1.5 GB image at an effective 50 MB/s takes 30 seconds; a 150 MB one, 3.
5. Start and warm-up. Seconds for a small Go binary, a minute or more for a large JVM service to reach full speed.
6. Readiness. A passing readiness probe (every 10 seconds by default) and the load balancer’s first health check.

Headroom: growth rate × lag

The fleet you already have must carry everything that arrives during the lag:

headroom = growth rate × total lag

Say an evening ramp adds 2,000 requests/s every minute on top of 20,000, and the lag is 4 minutes. When new capacity starts serving, load is 20,000 + 4 × 2,000 = 28,000, all on today’s fleet. So that fleet must run at 20,000 ÷ 28,000 ≈ 71% of its safe capacity, not 90%.

In general, with growth g per minute as a fraction of current load and lag L in minutes, the safe target is 1 ÷ (1 + g × L). Halve the lag and the same ramp allows 1 ÷ 1.2 ≈ 83%: a shorter lag is cheaper capacity.

A 10× spike in 30 seconds gives g × L = 9 and a 10% target. That is not an autoscaling problem: pre-scale if you know it’s coming, and cache, queue and shed load if you don’t. The SRE workbook adds a rule: autoscale before shedding starts, or you drop traffic you could have served.

Shortening the lag

Stability: flapping and safe scale-in

Feedback loops oscillate when they react to noise or to their own effects. Flapping means adding instances and removing them minutes later, again and again, paying for cold starts and dropped connections each time. The causes:

• Noise steering a loop with no dead band.
• Thresholds too close. Removing one of n instances raises load per instance by n ÷ (n − 1). With 3 instances, scale-out at 60% and scale-in at 45%: at 45% one goes, the other 2 run at 45 × 3 ÷ 2 = 67.5%, and it comes back. Scale-in must sit below scale-out × (n − 1) ÷ n, here 40%. Azure’s autoscaling guidance warns about exactly this; target tracking does the sum for you.
• Warm-up counted as load, and deciding again before the last action shows up.

The fixes: a dead band (the HPA’s 10% tolerance); a scale-in stabilisation window that acts on the highest recommendation of the last 5 minutes; warm-up exclusion (the HPA’s CPU initialisation period, EC2’s instance warm-up); rate limits on scale-in; out fast, in slow, because, as the SRE workbook puts it, failing to scale up drops traffic while scaling down late only costs money; and bounds: a min that survives losing a zone and a max your dependencies survive.

Scaling in without dropping work

• Drain first. Stop routing new requests, let in-flight ones finish, then stop. AWS load balancers wait 300 seconds for deregistration by default; health checks done right walks through the Kubernetes sequence.
• Protect long jobs. EC2 scale-in protection can be switched on around each unit of work; in Kubernetes, use a long termination grace period or run long work as jobs (KEDA’s ScaledJob).
• Make work resumable. Unfinished messages return to the queue, so checkpoint progress and make handlers idempotent.

Going deeper: how autoscaling goes wrong

Scaling into a saturated dependency

The classic self-inflicted outage. The database slows; pods hold requests longer; CPU and in-flight requests climb; the autoscaler adds pods; each opens its own connection pool. At 40 pods × 20 connections = 800 against a database that allows 500, new pods can’t connect and the database burns its remaining capacity on connection churn. The SRE workbook describes the same loop, where the extra load “can prevent the dependency from recovering”.

Retry storms inflate the metric

The autoscaler can’t tell retries from demand. Three layers making three attempts each turn one request into up to 3 × 3 × 3 = 27 calls at the bottom. In a 2016 Pokémon GO incident described in the SRE workbook, synchronised client retries produced spikes of 20× the previous global peak. Fix the source: see retry storm and timeouts, retries and backoff with jitter.

CPU limits hide the real demand

In Kubernetes a CPU limit is enforced by throttling, which caps what the HPA can see. With request = limit = 1 CPU and a 70% target, a pod that needs 2.5 CPUs reads 100%, so the HPA asks for ceil(n × 100 ÷ 70) ≈ 1.43n replicas when 2.5 ÷ 0.7 ≈ 3.6n are needed, and gets there over several lag-long rounds. Leave room between request and limit, and add a metric that throttling can’t flatten.

Warm-up and cold caches

A new JVM pod with empty caches may manage a fraction of a warm pod’s load for its first minute; give it a full share at once and it slows, hurts p99 and may fail readiness. Use a startup probe, warm-up requests before readiness, and the load balancer’s slow start. Beware capacity caches too: a burst of new pods is a burst of cache misses on the database (backend failure modes).

Quotas and a region out of machines

The cloud can say no: an account quota (EC2’s InstanceLimitExceeded) or a zone out of an instance type (InsufficientInstanceCapacity), most likely when everyone scales at once. Before a big event, raise quotas, reserve capacity in each zone, allow several instance types and zones, and pre-scale early. Services in one project share its quota, so one service’s scale-out can starve another’s.

Cost runaway

Without a max, a metric bug or junk traffic scales you to the edge of your quota. Set min and max, keep a kill switch that on-call can use to freeze autoscaling (the SRE workbook wants it easy and documented), and alert when a group sits at max. Cost-aware design covers budgets.

Stateful services

Adding a database replica, a Kafka broker or a cache node means moving data on machines that are already overloaded. Scale stateful tiers ahead of need on a plan, and use vertical resizing or read replicas for short-term relief (vertical vs horizontal scaling, database scaling).

Pre-scaling for known events

An innings break in a final, a noon flash sale, the 10:00 tatkal window on an IRCTC-style booking site: each arrives on a clock, faster than any reactive loop.

1. Load-test one instance with its real dependencies until latency breaks the SLO. Say a pod holds p99 under 300 ms up to 300 requests/s.
2. Size the forecast peak with headroom. 60,000 requests/s at a 70% target is 60,000 ÷ (300 × 0.7) ≈ 286 pods, plus nodes to hold them.
3. Check every dependency at that size. A payment gateway’s quota doesn’t autoscale.
4. Schedule it 30–60 minutes early, with target tracking on top for surprises.
5. Plan for being wrong. Niantic load-tested Pokémon GO to 5× its most optimistic estimate; launch traffic was nearly 50× that estimate.
6. Scale down slowly afterwards. Events come in waves.

Cheat sheet

How to say it in the interview

“I’d autoscale the stateless API tier on requests per pod, with CPU as a second metric. A load test shows one pod holds our SLO at about 350 requests/s, so I target 250. Queue workers scale on backlog per worker: the delay we promise times one worker’s throughput. The loop takes about 90 seconds when a node has room and about 5 minutes when it needs a new one, so I size the target from growth rate times lag and keep some placeholder pods. For the 10 am rush I raise the minimum on a schedule an hour ahead. Out fast, in slowly with a five-minute window, a minimum spread over three zones, and a max set by the database’s connection budget, because autoscaling can’t scale the database.”

What metric would you scale a queue worker on? Backlog per worker, targeting acceptable latency × per-worker throughput: a 30-second promise at 50 messages/s means 1,500 per worker. Cross-check arrival rate ÷ throughput, and for Kafka remember partitions cap the consumers.

How do you handle a 10× spike in 30 seconds? Not with reactive autoscaling: the lag is minutes, and absorbing 10× reactively means running at 10% utilisation. If it’s predictable, pre-scale. If not, serve what you can from caches and the CDN, queue what can wait, rate-limit and shed the rest with fast errors, and let autoscaling catch up.

Why not scale on latency? It isn’t proportional to capacity: flat until saturation, then explosive, so the loop reacts late and overshoots. It also rises when a dependency is slow, where more instances only add load. I alert on p99 and steer on load per instance.

How do you stop flapping? A dead band (the HPA’s 10% tolerance), a five-minute scale-in window, ignoring warming instances, rate-limited scale-in, and a scale-in threshold below scale-out × (n − 1) ÷ n.

Should the database autoscale too? Not reactively, because adding nodes moves data under load. Scale it on a plan, and cap the app tier’s replicas so their connection pools fit what the database accepts.

Where to go next

• Scalability: what scaling means and where it stops.
• Vertical vs horizontal scaling: bigger machines or more of them.
• Backpressure and load shedding: what to do with load you can’t serve yet.
• Health checks done right: readiness, slow start and draining.
• Cost-aware design: budgets, unit cost and choosing the max.
• Serverless architecture: platform-managed scaling and cold starts.
• Connection pooling: keeping a growing fleet from exhausting the database.

References and further reading

1. Kubernetes documentation: Horizontal Pod Autoscalingkubernetes.ioThe replica formula, the 15-second sync period, the 0.1 tolerance, the 300-second scale-down window, default scale-up and scale-down policies, how not-ready pods are handled, and scale-to-zero on external metrics.
2. Kubernetes documentation: Node Autoscalingkubernetes.ioHow node autoscalers provision for unschedulable pods from resource requests rather than usage, consolidation, and the differences between Cluster Autoscaler and Karpenter.
3. Kubernetes documentation: Vertical Pod Autoscalingkubernetes.ioThe recommender, updater and admission controller, and the Off, Initial, Recreate and in-place update modes.
4. Cluster Autoscaler FAQ (kubernetes/autoscaler)github.comThe 10-second scan interval, scale-up latency targets, 3–4 minute node provisioning on GCE, the 10-minute scale-down delay, and overprovisioning with low-priority pause pods.
5. KEDA documentation: Scaling Deployments, StatefulSets and custom resourceskeda.shActivation (0 to 1) versus scaling (1 to N through the HPA), scale-to-zero, and long-running work; the ScaledObject reference gives the 30-second polling and 300-second cooldown defaults.
6. Amazon EC2 Auto Scaling: Target tracking scaling policiesdocs.aws.amazon.comWhich metrics work for target tracking and which do not (total requests, latency, raw queue length), metric intervals, instance warm-up, and scale-out-if-any, scale-in-if-all behaviour.
7. Amazon EC2 Auto Scaling: How predictive scaling worksdocs.aws.amazon.comThe 24-hour minimum history, 14-day analysis, 48-hour hourly forecast refreshed every 6 hours, forecast-only mode, scale-out only, and pre-launching instances.
8. Amazon EC2 Auto Scaling: Scaling policy based on Amazon SQSdocs.aws.amazon.comBacklog per instance as the scaling metric, the acceptable-backlog calculation from latency and processing time, and scale-in protection for long tasks.
9. Managing Load (Google SRE Workbook, chapter 11)sre.googleAutoscaling failure modes (unhealthy instances, CPU-burning bugs, failing dependencies), min and max bounds, the kill switch, dependency analysis, and the Pokémon GO launch figures.
10. Autoscaling guidance (Azure Architecture Center)learn.microsoft.comScheduled plus metric-based rules, flapping from thresholds set too close, finding the bottleneck before adding compute, and bounding the maximum by quotas and dependencies.