Scalability

Scalability is a system’s ability to handle more load by adding resources, getting back roughly proportional capacity while latency stays within its target and the cost of each request stays flat. It is never a yes-or-no label. You say which load grows (requests a second, data size, fan-out), which percentile you must hold, and what each extra unit of capacity costs.

Picture it first: a cloud kitchen on IPL final night

You run a cloud kitchen. One cook handles 40 orders an hour; on IPL final night, 400 arrive. Your options:

• A bigger stove and a faster cook, until there is no bigger stove.
• Nine more cooks who each do everything. This works only if the order slip carries everything a cook needs. If cooks keep half-finished orders in their heads, a new cook can’t help.
• Stations: tandoor, biryani, desserts, with extra cooks at the busiest one.
• A second kitchen for the other half of the city.

Two things still cap output. If a head chef must taste every plate, the kitchen never ships faster than one person tastes. And if cooks must keep shouting updates about the one shared tray of paneer, each new cook adds shouting, and past some point the kitchen gets slower.

Map it back: the bigger stove is scaling up; identical cooks are cloned stateless servers, and the slip is a request carrying its own state. Stations are a split by function, the second kitchen a split by data. The head chef is Amdahl’s serial fraction; the shouting is the USL’s coherency cost.

What scalability means, precisely

Werner Vogels, Amazon’s CTO, gave a crisp definition in 2006: a service is scalable if adding resources increases its performance in proportion to the resources added. He added a second test that is easy to forget: if you add resources for redundancy, performance must not drop. That gives three things to pin down: the load, the performance you promise, and the cost of each step.

Describe the load with load parameters

“More users” is too vague to design for. Name the numbers that stress your system:

• Requests per second, by type: average and peak, for each expensive operation.
• Read-to-write ratio: 100:1 calls for caches and replicas; 1:1 calls for write partitioning.
• Concurrency: open connections and in-flight requests, which cost memory even when idle.
• Data size: total bytes and the hot working set.
• Fan-out: how much work one request triggers downstream.
• Skew: how uneven load is across keys. The celebrity, the flash-sale product and the tatkal train break systems, not the average user.

To turn a forecast into these numbers, see back-of-envelope estimation.

Describe performance with percentiles

For an online service, performance is response time as percentiles, because the slow tail is what users feel; for batch jobs it is throughput. A scalability target pairs load with a percentile: “p99 under 200 ms at 20,000 requests a second.” Percentiles and tail latency are covered in latency vs throughput vs bandwidth.

Scalability is not performance

A performance problem shows up with one user: a page takes, say, 2 seconds on an idle system because of an unindexed query. You fix it by profiling and optimising.

A scalability problem shows up only under load: the page takes, say, 80 ms at 500 requests a second but 3 seconds at 5,000, because something is queueing or contending. You fix it by removing the bottleneck, adding capacity or partitioning.

The two interact. Twitter reported in 2013 that its Rails hosts served 200–300 requests a second each, while the JVM services that replaced them handled 10,000–20,000. Doing 50 times more per host means about 50 times fewer hosts, and every distributed problem is easier at a fiftieth of the size.

Cost is part of the definition

If 10× the traffic needs 10× the servers, cost per request is flat. If it needs 20× because efficiency collapsed, the system technically “scales” but your margins don’t.

Load parameters in practice: Twitter’s home timeline

Twitter’s 2012 numbers, from Raffi Krikorian’s “Timelines at Scale” talk at QCon San Francisco, show why picking the right load parameter matters. About 400 million tweets a day arrived, roughly 5,000 a second on average and over 12,000 during big events. That write rate is modest; the hard part was what each tweet triggered.

The slides frame the choice as two complexity classes. Fan-out on write (push) inserts each tweet ID into every follower’s cached home timeline: an O(n) write, an O(1) read. Fan-out on read (pull) stores the tweet once and merges followed accounts’ tweets at read time: an O(1) write, an O(n) read. With reads outnumbering tweets 60 to 1, Twitter pushed, serving timelines from Redis lists at 1 ms p50 and 4 ms p99.

Skew breaks the averages. A tweet from an account with 31 million followers needs 31 million inserts: at the system’s average delivery rate, 31,000,000 ÷ 300,000 ≈ 100 seconds of fan-out capacity for one tweet. The slides sketch the hybrid that follows: for heavily followed accounts, the home timeline query merges in that account’s own timeline at read time instead.

“5,000 writes a second” would have led you to a small design. The load parameters that mattered were fan-out per write, its distribution, and the read rate. The full design is in design a news feed.

Scale up, scale out, and where the state goes

Scale up (vertical): move to a bigger machine. You gain simplicity: no code changes, and transactions and joins keep working. You pay with a hard ceiling, prices that climb faster than capacity near the top, one failure domain, and often a restart to resize. In 2020 Figma ran a single Postgres database on AWS’s largest physical instance, a sound choice until there is no bigger box.

Scale out (horizontal): add more machines. You gain near-unlimited room, small increments and redundancy; AWS’s Well-Architected reliability pillar recommends several small resources over one large one so a single failure takes out less. You pay with load balancing, partial failures, consistency questions and more to operate.

Most systems do both: stateless tiers scale out from day one, and stateful stores scale up as far as is affordable before being split. The full comparison is in vertical vs horizontal scaling.

Why stateless tiers scale out trivially

A server is stateless if it keeps nothing between requests that a later request depends on. The Twelve-Factor App makes it a rule: processes are stateless and share-nothing, anything that must persist lives in a backing service, and memory and local disk are only a scratchpad for the current request.

Then scaling out is arithmetic. Any request can go to any server, so a load balancer spreads traffic evenly, an autoscaler adds or removes servers freely, and a crash loses only requests in flight. As the Twelve-Factor concurrency page notes, one machine can only grow so large, so the app must span many processes on many machines.

Where the state goes

Making a tier stateless doesn’t delete the state; it moves it somewhere you scale and protect on purpose:

• Sessions go to a shared store with expiry such as Redis or Memcached, or into a signed token the client sends each time. Twelve-Factor rules out sticky sessions entirely.
• Caches become a shared or distributed cache; a small in-process cache is fine if losing it only costs a miss.
• Durable data goes to databases; files go to blob storage, never local disk.
• Work in progress goes to a queue, so any worker can pick it up.
• Long-lived connections (WebSockets) are stateful by nature: keep that tier thin and drain it gently on deploys.

Stateful vs stateless design covers when keeping state local is worth it.

The scale cube: three ways to split

AKF Partners introduced the scale cube in 2007 to name the three ways to divide work. Large systems use all three.

• X axis: clone. N identical copies behind a load balancer, each able to see all the data.
• Y axis: split by function. Search, cart and payments become services, ideally each with its own data.
• Z axis: split by data. The same code on separate slices of data, by customer, region or key hash: shards, pods and cells.

AKF’s write-up adds practical detail. X is easiest to build, but every copy needs the whole data set and caches weaken as data grows. Y gives each service a smaller data set, and it is the only axis that also scales the organisation, because teams can own services. Z keeps caches small and hot, isolates failures to one slice, and can keep EU customers’ data in the EU.

Scaling reads vs scaling writes

Reads and writes scale in opposite ways.

Reads scale by adding copies. Any copy can answer a read, so you put copies closer to users and in faster media:

• A cache turns most reads into memory lookups.
• Read replicas multiply read capacity, at the cost of replication lag.
• A CDN serves static files and media from the edge.
• Denormalised, precomputed views do the work once at write time; Twitter’s cached timelines are one.

Writes scale by dividing ownership. Every copy must apply every write, so copies add no write capacity. Instead you:

• Shard: give each key one home, so N shards take roughly N times the writes (see database sharding).
• Queue: accept the write into a durable queue, answer quickly, and let workers drain it steadily, flattening spikes.
• Batch: group small writes into one flush or bulk insert, trading a little latency for a lot of throughput.
• Append, don’t update: sequential log appends (Kafka, LSM-tree databases) are far cheaper than random in-place updates.

The patterns are covered in scaling reads, scaling writes and database scaling.

The limits: Amdahl’s law and the Universal Scalability Law

If scaling out were perfect, N servers would give N times the throughput. Two models explain why it doesn’t, and why it can even go backwards.

Amdahl’s law: the serial fraction sets a ceiling

In 1967 Gene Amdahl observed that about 40% of executed instructions went to sequential “data management housekeeping” that parallel hardware could not speed up, and argued that this overhead alone capped the gain from parallel processors at five to seven times.

Write the serial fraction as σ. With N workers, the speed-up is:

S(N) = N ÷ (1 + σ(N − 1))

As N grows without limit, S(N) approaches 1 ÷ σ.

Take σ = 0.05: 5% of the work must happen one step at a time.

• The ceiling is 1 ÷ 0.05 = 20×, however many workers you add.
• With 32 workers: 32 ÷ (1 + 0.05 × 31) = 32 ÷ 2.55 ≈ 12.5×, or 39% efficiency per worker.

In distributed systems, the serial fraction is whatever every request passes through one at a time: a single primary database, a global lock, a single-threaded leader, a hot row.

The USL: contention plus coherency

Neil Gunther’s Universal Scalability Law adds a second penalty. Besides waiting for shared resources (contention, σ), nodes must exchange data to keep copies consistent (coherency, κ). N nodes form N(N − 1) ordered pairs that might need to talk, so this cost grows with the square of N:

C(N) = N ÷ (1 + σ(N − 1) + κN(N − 1))

C(N) is throughput relative to one node. With κ = 0 it reduces to Amdahl’s law. With any κ above zero, the curve has a peak at:

N* = √((1 − σ) ÷ κ)

Past N*, throughput is retrograde: every node you add makes the whole system slower.

Worked example with σ = 0.05 (the same contention as above) and an illustrative κ = 0.002:

• Peak: N* = √(0.95 ÷ 0.002) = √475 ≈ 22 nodes.
• At 22 nodes: 22 ÷ (1 + 0.05 × 21 + 0.002 × 22 × 21) = 22 ÷ 2.97 ≈ 7.4×.
• At 64 nodes: 64 ÷ (1 + 3.15 + 8.06) = 64 ÷ 12.21 ≈ 5.2×, less than 8 nodes deliver (5.5×).
• Efficiency per node falls from 68% at 8 nodes and 45% at 16 (7.2×) to 8% at 64.

Using the models on a real system

Don’t guess σ and κ; measure them. Gunther suggests load-testing at at least half a dozen load levels and fitting the curve to the throughput at each. A large σ points to a queue or lock that everything passes through. A large κ points to chatter between nodes: invalidation broadcasts, all-to-all heartbeats, distributed locks. Gunther’s worst case, with both terms above zero, is “tasks acting on shared-writable data”, such as online reservation systems: exactly the tatkal and flash-sale problem.

Coordination is the enemy

Both models say the same thing: scaling stops where nodes have to agree. The usual culprits, and the standard escapes:

• Locks and hot rows. Thousands of people booking the same train at 10 am all want one seat-inventory row, so they queue on one lock: σ in action. Split the inventory into blocks owned by different partitions, or funnel each train’s requests through one ordered queue. See contention and double booking.
• Consensus. Every write to a Raft or Paxos group waits for a majority, and a bigger group is slower, not faster: 5 members need 3 acknowledgements to 3 members’ 2, and the leader sends every write to 4 followers instead of 2. Systems scale consensus by running many small groups, one per shard. See consensus and leader election.
• Cross-shard transactions. Two-phase commit holds locks across shards for extra round trips. Figma chose not to support atomic cross-shard transactions, and grouped related tables under one sharding key so most transactions stay on one shard. See two-phase commit vs sagas.
• Global counters and sequences. A single “total likes” counter or an auto-increment ID makes every write touch one key. When Twitter hash-sharded its tweets, MySQL could no longer issue IDs, so it built Snowflake to generate unique IDs on many machines without a central counter. Sharded counters summed on read follow the same idea. See design a unique ID generator.
• Keeping copies in sync. Every cache invalidation, gossip message or membership update is κ; keep the set of nodes that must hear about a change small.

The rule that falls out: most requests should touch one partition, and no request on the hot path should wait for all of them. That is why the shard key matters so much: it decides which operations stay local.

Elasticity and cost per request

Elasticity is how quickly and automatically capacity follows load, in both directions. A system can be scalable but not elastic: you can add shards, but each might take days to fill and rebalance. AWS’s reliability guidance says “stop guessing capacity”: watch demand and add or remove resources automatically, so you are neither saturated nor paying for idle machines.

Stateless tiers are elastic: a new server boots and takes traffic in minutes. Stateful tiers are not, because new capacity must first be filled with data, and caches start cold. Autoscaling also reacts after load arrives, so for known spikes, such as a Big Billion Days-style sale or the 10 am tatkal window, add capacity in advance. More in autoscaling done right.

Cost per request should stay flat or fall. The USL example shows how quietly that breaks: going from 8 to 16 nodes doubles the bill but raises throughput only from 5.5× to 7.2×, a gain of 1.31×, so each request costs 2 ÷ 1.31 ≈ 1.5 times as much. Track cost per request next to p99. Coordination, fan-out, cross-zone data transfer and peak over-provisioning push unit cost up; better cache hit ratios, committed-use discounts and amortised fixed costs push it down. More in cost-aware design.

A scaling journey: what breaks first

Most products climb the same staircase. Each step adds moving parts, so take it only when a measured bottleneck forces you to.

Cells are the Z axis applied to the whole stack: each cell is a complete, independent copy serving its own slice of customers, so a bad deploy hurts one cell, not everyone. See cell-based architecture and multi-region and active-active.

Two real journeys

Twitter, 2010–2013. After outages during the 2010 World Cup, Twitter found a Rails monolith serving 200–300 requests a second per host, and a time-sharded tweet store whose single MySQL master could not keep up with spikes and suffered read and write hot spots. Admitting it had been “throwing machines at the problem”, it split the monolith into services (Y), hash-sharded tweets (Z) instead of filling one database at a time, built Snowflake for IDs, and moved to JVM services handling 10,000–20,000 requests a second per host. In 2013 it absorbed a one-second peak of 143,199 tweets, about 25 times its daily average, without a visible blip.

Figma, 2020–2024. Figma’s database stack grew almost 100× from 2020. It began on one Postgres instance, the largest AWS offered; by the end of 2022 it ran caching, read replicas and about a dozen vertically partitioned databases. Then single tables reached several terabytes and billions of rows: vacuums hurt reliability, and the busiest tables were heading past the maximum IOPS of Amazon RDS. Vertical partitioning can’t split one table, so Figma built horizontal sharding on Postgres. The first table took about nine months, and the team gave up atomic cross-shard transactions, database-enforced foreign keys and efficient range scans on shard keys.

Cheat sheet

How to say it in the interview

“Scalability means that when load grows, I add resources and get roughly proportional capacity back, at the same p99 and a flat cost per request. First I name the load parameters: say 20,000 reads and 2,000 writes a second at peak, 5 TB of data, and a fan-out of a few hundred per post, skewed towards popular accounts. The app tier is stateless behind a load balancer, with sessions in Redis, so it scales out by adding instances. Reads scale with copies: CDN, cache, read replicas. Writes scale by sharding on user ID and moving slow work onto queues. And I keep coordination off the hot path, because contention and cross-node chatter make throughput flatten, or even fall, as nodes are added.”

How would you scale this to 10× traffic? Find which load parameter grows 10× and which component saturates first, by load-testing and watching utilisation, queue depth and p99 per tier. Stateless tiers get more instances; reads get a higher cache hit ratio, more replicas and the CDN; if one primary can’t take 10× the writes, shard, batch or queue them. Check hidden multipliers such as database connections and fan-out, then re-test, because the bottleneck moves.

Vertical or horizontal first? Horizontal for stateless tiers from day one: it is cheap and gives redundancy. Vertical for the database first, because a bigger box is the cheapest way to postpone sharding. Shard when the runway on the biggest affordable box is shorter than a sharding project takes, or one database is too big a blast radius.

Why doesn’t adding servers always help? Amdahl’s law caps the gain at 1 ÷ σ, so 5% serial work means at most 20×. The USL adds cross-node coherency, which grows with N², so throughput peaks at √((1 − σ) ÷ κ) nodes and then falls. And often the bottleneck is elsewhere, such as the database, where more app servers only add connections.

What makes a service hard to scale out? Local state (in-memory sessions, local files, sticky connections), operations needing a global view (unique constraints, global counters, multi-entity transactions), hot keys, and strict global ordering. Each forces either coordination or a careful partitioning key.

How do you know which part to scale? Measure. Under a load test that ramps traffic, track each tier’s utilisation, saturation (queue lengths, pool waits) and p99, and find the first resource to saturate. Fitting the USL to throughput at half a dozen load levels shows whether contention or coherency dominates. Scale that, then measure again.

Where to go next

• Vertical vs horizontal scaling: the full trade-off.
• Stateful vs stateless design: when local state is worth it.
• Scaling reads and scaling writes: the patterns behind each half.
• Database scaling and database sharding: replicas, partitions and shard keys.
• Consistent hashing: spreading keys so a new shard moves little data.
• Autoscaling done right: elasticity in practice.
• Cell-based architecture: the Z axis for the whole stack.
• Cost-aware design: keeping cost per request flat.

References and further reading

1. A Word on Scalability (Werner Vogels, All Things Distributed, 2006)allthingsdistributed.comThe working definition: performance that grows in proportion to the resources added, and redundancy that costs no performance.
2. Validity of the single processor approach to achieving large scale computing capabilities (Amdahl, AFIPS 1967; IEEE SSCS News reprint, 2007)www3.cs.stonybrook.eduThe original argument that sequential housekeeping, about 40% of executed instructions, caps what parallel processors can gain.
3. A General Theory of Computational Scalability Based on Rational Functions (Gunther, arXiv, 2008)arxiv.orgAmdahl’s law and the USL written with σ and κ, the peak at √((1 − σ)/κ), and which workloads are contention- or coherency-limited.
4. How to Quantify Scalability: the Universal Scalability Law (Neil Gunther, Performance Dynamics)perfdynamics.comConcurrency, contention and coherency, retrograde throughput, and fitting the model from about half a dozen load-test points.
5. The Scale Cube (AKF Partners)akfpartners.comThe X (clone), Y (split by function) and Z (split by customer or data) axes, with the benefits and costs of each.
6. The Twelve-Factor App: VI. Processes and VIII. Concurrency12factor.netStateless, share-nothing processes, why sticky sessions are ruled out, and scaling out through the process model.
7. Timelines at Scale (Raffi Krikorian, QCon San Francisco 2012, slides)qconsf.comTwitter’s 2012 load parameters: tweets per second, timeline reads per second, deliveries per day, follower counts, and push versus pull.
8. New Tweets per second record, and how! (Twitter Engineering, 2013, archived copy)web.archive.orgA real scaling journey: monolith to services, temporal to hash sharding, Snowflake IDs, and requests per host before and after.
9. How Figma’s databases team lived to tell the scale (Figma, 2024)figma.comOne Postgres on the largest instance, then replicas, caching and vertical partitions, then horizontal sharding and what it cost.
10. AWS Well-Architected Framework, Reliability pillar: design principlesdocs.aws.amazon.comScale horizontally to shrink the impact of one failure, and stop guessing capacity by scaling on measured demand.