Vertical vs horizontal scaling

Vertical scaling (scaling up) makes one machine bigger: more vCPUs, more memory, faster disks and network. Horizontal scaling (scaling out) adds more machines and spreads the work across them, behind a load balancer or by splitting the data. Scaling up is simpler and keeps everything in one place until you reach the biggest box you can buy. Scaling out has no fixed ceiling and survives failures, but only for work that can be shared out.

Picture it first: one tawa or many counters

A dosa stall near a college is swamped every morning. The owner has two options.

He can buy a bigger tawa and hire a faster cook. Nothing else changes: same counter, same queue, one cash box, one bucket of chutney. But only so big a tawa fits in the stall, the stall shuts for a morning while it is installed, and if the cook falls sick, nobody eats.

Or he can open more counters, each with its own tawa and cook, and post a helper at the entrance who sends each customer to the shortest queue. Now there is no fixed limit (open a tenth counter in exam week), and one sick cook only slows things a little. But every counter needs batter, someone must keep the chutney the same everywhere, and a customer who pays at counter 3 and collects at counter 7 needs counter 7 to know.

Map it back: the tawa and cook are a server’s CPU and memory. The bigger tawa is vertical scaling and more counters is horizontal scaling. The helper at the entrance is a load balancer. The batter, chutney and payments are shared state (a database, sessions, a cache), and keeping them consistent across counters is where the hard problems of scaling out live.

Two ways to add capacity

Vertical scaling (scale up) moves the same workload to a machine with more of whatever is saturated: vCPUs, memory, disk IOPS or network bandwidth. In the cloud it is an instance-type change, say from m7i.xlarge (4 vCPU, 16 GiB) to m7i.48xlarge (192 vCPU, 768 GiB). The software does not notice, apart from being restarted.

Horizontal scaling (scale out) runs more copies and divides the work. Stateless services sit behind a load balancer that spreads requests over identical instances. Data is partitioned (sharded) so each node owns a slice, and each slice is replicated for reads and safety.

The real difference is not “one versus many”. It is where the state lives. Scaling up keeps all state in one memory and one disk, so a transaction, a join or a lock is local and cheap. Scaling out spreads state across a network, so the same operations need coordination. That is why almost every scalability question turns into a question about state; see stateful vs stateless design.

How far up can you go?

Further than most people assume. As of October 2026, AWS’s general purpose and memory optimized specification tables list:

• m7i.48xlarge: 192 vCPU, 768 GiB, 50 Gbps of network.
• m8i.96xlarge: 384 vCPU and 1,536 GiB, the largest general-purpose size.
• x8i.96xlarge: 384 vCPU and 6 TiB, for big in-memory working sets.
• u7in-32tb.224xlarge (896 vCPU) and u7inh-32tb.480xlarge (1,920 vCPU): 32 TiB of memory each, aimed at very large in-memory databases.

On-demand Linux rates in US East are flat per vCPU within a family: every m7i size costs $0.0504 per vCPU-hour, from m7i.large (2 vCPU, $0.1008 an hour) to m7i.48xlarge (192 vCPU, $9.6768 an hour, about $7,064 a month at 730 hours). The cloud charges no premium for “big” until the very top. In the x8i family every size up to 64xlarge costs $0.1094 per vCPU-hour, and the 96xlarge costs $0.1269, 16% more. The 32 TiB u7in costs $360.99 an hour, roughly $263,500 a month, and per GiB that is $360.99 ÷ 32,768 ≈ $0.0110 against $21.01 ÷ 3,072 ≈ $0.0068 for x8i.48xlarge: 1.6× the price per GiB. So in the cloud the penalty for going up is mostly a wall, not a slope.

Sizes roughly double, so scaling up buys capacity in big lumps: needing 100 vCPUs means paying for 192. And past 192 there is no bigger m7i; m8i reaches 384, and then you are done.

Real systems run a long way up that ladder:

• Stack Overflow (2016) served 209 million HTTP requests a day, about 209,420,973 ÷ 86,400 ≈ 2,424 a second on average, and 505 million SQL queries, about 5,843 a second. Each SQL Server cluster had one primary “taking almost all of the load”: 384 GB of RAM with 2 × 12 cores, or 768 GB with 2 × 8 cores. Its 11 web servers were deliberately overprovisioned: the sites needed only one, and the rest bought rolling builds, headroom and redundancy.
• Let’s Encrypt runs its certificate authority against a single MariaDB database “to minimize complexity”, with replicas taking some reads. Its late-2020 upgrade went from 24 cores and 1 TB of RAM to 64 cores, 2 TB and 24 NVMe drives. Median API time fell from about 90 ms to about 9 ms, and primary CPU from over 90% to about 25%. The team says it may eventually need to shard, but hardware advances have let it avoid that so far.

Both are the same hybrid: a scaled-out, stateless web tier in front of a scaled-up database.

Why scaling up is often the right first move

What you gain:

• No code changes. Same binary, same schema, same queries.
• No distributed-systems problems. No shard keys, cross-node transactions, rebalancing or split brain. ACID transactions, joins, unique constraints and strong consistency stay local and cheap.
• Speed. A resize takes minutes; a sharding project takes quarters.
• One big memory. One 768 GiB buffer pool holds more distinct data than 24 replicas with 32 GiB each, which all cache the same hot rows.
• Less to operate. One box to patch, monitor and back up.

What you pay:

• A hard ceiling. And you must leave before you hit it, because migrating to a sharded design needs spare capacity to copy data while serving traffic.
• A single point of failure. One box is one failure domain, so production needs a standby anyway (single point of failure, failover). A second m7i.48xlarge is another $9.68 an hour that mostly idles.
• Downtime to resize. EC2 requires you to stop an instance to change its type: a few minutes, plus your application’s start-up. Managed databases hide this with a standby, but on RDS an instance modification is a documented failover trigger, and failovers typically take 60–120 seconds.
• Blast radius. Everything on the box fails together, and one runaway query slows every tenant.
• Diminishing returns. Doubling cores does not double throughput if work queues on a lock, one hot row, the write-ahead log or memory bandwidth between CPU sockets.
• The top-end premium, where the per-vCPU or per-GiB price steps up.

What scaling out buys, and what it costs

What you gain:

• Near-linear capacity for stateless work. If instances share nothing, 40 of them serve about 40 times what one serves.
• Fault tolerance. AWS’s Well-Architected reliability pillar says to scale horizontally, replacing one large resource with many small ones so a single failure hurts less. Losing 1 of 24 boxes costs 1 ÷ 24 ≈ 4.2% of capacity, not 100%.
• Rolling deploys. Update a few instances at a time with no maintenance window (deployment strategies).
• Elasticity. Autoscaling adds commodity instances in minutes and removes them when the spike ends, so you pay for IPL-final capacity only during the final (autoscaling done right).
• Geography and bandwidth. Instances spread across zones and regions (multi-region), and each brings its own network card: 24 × m7i.2xlarge have 24 × 3.125 = 75 Gbps of baseline bandwidth, against 50 Gbps for one m7i.48xlarge.

What you pay:

• Statelessness or partitioning. Azure’s guidance is to avoid instance stickiness: any instance must handle any request, so in-memory sessions, local files and machine-specific keys must go, and data must be split by a key.
• Load balancing and health checks, another hop and another component (L4 vs L7 load balancing).
• Distributed-data problems. Shard keys, hot shards, rebalancing, cross-shard queries and transactions, and replication lag.
• Operations and the network. More instances to patch and deploy, partial failures (one slow node, timeouts, retries), and function calls that become remote calls costing hundreds of microseconds.
• Connection fan-in. 63 app instances × a pool of 20 = 1,260 database connections; you need connection pooling.
• Coordination: anything the nodes must agree on costs messages, as the next section shows.

Why more machines can make it slower

Amdahl’s law covers the first limit. If a fraction s of the work is serial, the speedup on N processors is 1 ÷ (s + (1 − s) ÷ N), and it can never exceed 1 ÷ s. With 95% of the work parallel (s = 0.05):

• 16 nodes: 1 ÷ (0.05 + 0.95 ÷ 16) = 1 ÷ 0.109 ≈ 9.1×
• 64 nodes: 1 ÷ (0.05 + 0.95 ÷ 64) = 1 ÷ 0.065 ≈ 15.4×
• any number of nodes: never more than 1 ÷ 0.05 = 20×

The Universal Scalability Law (Neil Gunther) adds a second limit. Relative capacity is C(N) = N ÷ (1 + α(N − 1) + βN(N − 1)):

• α, contention: waiting in a queue for something shared, such as a lock, a single primary or one hot row. With β = 0 this is Amdahl’s law, and capacity approaches 1 ÷ α.
• β, coherency: the cost of keeping copies consistent through point-to-point exchanges, such as cache invalidations, distributed locks, gossip or cross-shard transactions. It grows with N × (N − 1), roughly the number of pairs of nodes, so past a point it beats the gain from adding nodes.

When β is above zero, throughput peaks at N = √((1 − α) ÷ β) nodes and then goes down. Take α = 0.05 and β = 0.002:

• Peak: N = √(0.95 ÷ 0.002) = √475 ≈ 22 nodes
• C(22) = 22 ÷ (1 + 0.05 × 21 + 0.002 × 22 × 21) = 22 ÷ 2.974 ≈ 7.4×
• C(64) = 64 ÷ (1 + 0.05 × 63 + 0.002 × 64 × 63) = 64 ÷ 12.21 ≈ 5.2×: nearly three times the hardware for 30% less throughput.

Three lessons carry into every design:

1. It applies both ways. Cores in one box contend for locks and memory bandwidth; nodes in a cluster contend for the database.
2. Cut α by removing shared bottlenecks: split the hot counter, partition the lock, move work to a queue (contention and double booking).
3. Cut β by making nodes not talk: share-nothing designs where each request touches one node. That is why a stateless app tier scales almost linearly, and the data tier under it does not.

Gunther suggests fitting α and β to half a dozen load-test points to predict where the curve bends before production finds out.

Scaling tier by tier

“Vertical or horizontal?” has a different answer for each tier, and the deciding factor is how much state the tier owns. Azure’s scale-out guidance makes the same point bluntly: if the database is the bottleneck, more web servers will not help, so find the bottleneck first.

• App servers scale out from day one; keep sessions and uploads off the instance.
• Caches scale up until one node’s memory is tight, then out with consistent hashing (distributed caching).
• Message brokers scale out through partitions shared among a consumer group (message queues).
• Search clusters add shards for index size and replicas for query rate (Elasticsearch).
• The primary database climbs a ladder: bigger instance, replicas for reads, splitting by domain, then sharding for writes (database scaling).

Worked example: a sale-day API

You run the catalogue and cart API for a Big Billion Days-style sale, peaking at 60,000 requests a second for a few hours. Assume a load test shows one m7i.xlarge (4 vCPU) serves 1,500 requests a second at 60% CPU with p99 inside target; measure your own.

1. Raw need: 60,000 ÷ 1,500 = 40 instances (N).
2. Spares: during the sale an instance will die and a deploy will hold a batch out of service, so 42 must survive (N + 2).
3. Zone loss: spread over 3 availability zones, losing one leaves two thirds of the fleet. So the fleet is 42 × 3 ÷ 2 = 63 instances, 21 per zone.
4. Cost: 63 × $0.2016 = $12.70 an hour, against $8.06 for the bare 40. On a normal sale-day peak each instance serves 60,000 ÷ 63 ≈ 952 requests a second, 63% of its safe limit.
5. Off-peak: a normal day’s 6,000 requests a second needs 4, but keep 2 per zone (6). Scale out on a schedule before the sale: autoscaling reacts in minutes, and a midnight sale arrives as a step.

Could one big box do it? On paper, 40 × 4 = 160 vCPUs fits in one 192-vCPU m7i.48xlarge at 83%. In practice, three things break:

• Scaling inside the box. 160 vCPUs only behave like 40 small instances if α and β are near zero. If just 1% of each request’s work queues on something shared, Amdahl gives 192 ÷ (1 + 0.01 × 191) ≈ 66 times one vCPU’s throughput, and at 1,500 ÷ 4 = 375 requests a second per vCPU that is about 66 × 375 ≈ 25,000 requests a second, well short of 60,000.
• Redundancy costs more in big pieces. Surviving a zone loss needs a second 48xlarge in another zone: 2 × $9.68 = $19.35 an hour, more than the 63-instance fleet’s $12.70. With k equal failure domains you must over-provision by k ÷ (k − 1): 100% with two, 50% with three, 33% with four.
• Every deploy restarts the only box.

The database behind it is a different story. The database scaling article estimates one PostgreSQL primary at roughly 8,000 inserts a second when index pages miss the cache at 64,000 IOPS, and about 32,000 on the fastest volumes. So:

• Orders at 8,000 writes a second match the pessimistic figure but sit well inside the optimistic one, so with lean indexes and time-ordered keys they fit one big primary. Scale up and add replicas for reads.
• Clickstream events at 200,000 writes a second are six times the optimistic single-node figure, so no box will do. Shard: 32 shards take 200,000 ÷ 32 = 6,250 writes a second each, under even the pessimistic figure, or use a store that partitions itself (Cassandra and DynamoDB). Start with fewer machines hosting many logical shards (database sharding).

Comparing the bills

Same 192 vCPUs and 768 GiB, bought as one box or as 24 small ones, at October 2026 on-demand prices:

With identical list prices, decide on availability, operations and licensing, and licensing pushes the other way. Commercial databases licensed per core charge for every core on every replica and shard, including the idle headroom each node keeps, so fewer, bigger, busier boxes waste fewer licences. Stack Overflow chose Elasticsearch over SQL Server’s full-text search partly for this reason; as Craver put it, “SQL CPUs are comparatively very expensive, Elastic is cheap”. See cost-aware design for the wider view.

Choosing: a decision guide

Ask these questions, in order:

1. What saturates first: CPU, memory, disk I/O, network, locks or connections?
2. Is the component stateless? If not, can its state be split by a key such as user, order or city?
3. How much runway is left before the biggest box, against how long a scale-out migration takes?
4. What does one failure cost: the share of users hit, and the price of a standby?
5. Does the load spike or grow? Spikes favour elastic scale-out; steady growth gives time to scale up.
6. Can you take the resize blip, and is it licensed per core?

Scale up when:

• An early-stage database with room left in its instance family: one resize beats a quarter of sharding work.
• One ACID database is a design goal, as at Let’s Encrypt, for security and reliability.
• Per-core licences make every extra node expensive.
• State that must be processed in one sequence at microsecond latency, like an order book: split by instrument, never within one book.
• A working set that just outgrows RAM, such as a session store at 90% of memory, when the next size fits it.

Scale out when:

• The tier is stateless: web and API servers and workers, from day one.
• Traffic spikes: IPL finals, tatkal at 10 am, flash sales. Elastic capacity costs money only while it runs.
• Availability matters: zone failures, rolling deploys, no maintenance windows.
• Data or writes exceed one node: shard by key, or use a store that partitions itself.
• Work fans out naturally: cricket-score pushes to millions of phones, thumbnails, queries over search shards.

The hybrid most systems run. Scale out the stateless tiers, scale up the stateful core, and climb the database ladder only when measurements demand it. Within a scaled-out tier, first pick a sensible node size (big enough to amortise per-node overhead, small enough that losing one hurts little), then add nodes; this is sometimes called diagonal scaling. At the largest scale, cell-based architectures repeat the pattern by stamping out identical copies of the whole stack.

Going deeper: traps and misconceptions

• “Horizontal is always better.” For a database that fits one box, scaling out buys sharding complexity years early. Stack Overflow and Let’s Encrypt serve huge traffic from a few big primaries.
• Scaling the wrong tier. Adding web servers while the database sits at 95% CPU only adds connections to the bottleneck.
• Forgetting that scale-out needs statelessness. In-memory sessions with sticky routing look fine until a node dies and its users are logged out.
• Forgetting headroom and failure domains. N is the answer for a perfect day; real fleets need N + 2, times k ÷ (k − 1) across k zones.
• Assuming linear speedup. Contention and coherency cap and then reverse gains. Load-test at 1×, 2× and 4× before promising 10×.
• “Big boxes cost more per core” and “resizing is free”: neither is true.

Cheat sheet

How to say it in the interview

“Vertical scaling means a bigger machine, horizontal means more machines. I’d scale stateless tiers out from the start: they sit behind a load balancer across three zones, so I can lose a zone, deploy without downtime and autoscale for a sale. For 60,000 requests a second at 1,500 per instance that’s 40, plus 2 spares, times 1.5 for zone loss: 63 instances. For the primary database I’d scale up first, because one box keeps transactions simple and the biggest instances are huge. Then read replicas for reads, and sharding only for the write path that outgrows one node. The price per core is the same either way, so I decide on availability, state and operations.”

When would you choose vertical scaling over horizontal? When the component is stateful and still fits one machine (a primary database, per-core licensed software, an order book) and the next size is quick to reach. I keep a standby for failover and watch the distance to the largest instance.

How do you scale a stateful service horizontally? Either move the state out (sessions to a shared store, files to object storage) so the service becomes stateless, or partition the state by a key with consistent hashing or a shard map, replicate each partition, and route each request to the node that owns its key.

Why can adding servers make throughput worse? Contention and coherency. In the Universal Scalability Law the coherency cost grows with the number of node pairs, so throughput peaks at √((1 − α) ÷ β) nodes and then falls. With α = 0.05 and β = 0.002 the peak is 22 nodes, and 64 nodes deliver 30% less.

Isn’t one big machine cheaper than many small ones? Not on cloud list prices: one m7i.48xlarge and 24 m7i.2xlarge both cost $9.68 an hour. The big box is cheaper to operate and keeps data local; the small ones are cheaper to make redundant (+4.2% for one spare against +100% for a second big box).

Where to go next

• Scalability: the bigger picture.
• Stateful vs stateless design: the precondition for scaling out.
• Database scaling: every rung of the database ladder.
• Database sharding and data replication: scaling the data tier out.
• Scaling reads and scaling writes: the interview patterns.
• Load balancing: L4 vs L7 and consistent hashing: how work gets spread.
• Autoscaling done right and cost-aware design: running an elastic fleet.
• Single point of failure and availability and the nines: why one big box needs a twin.

References and further reading

1. Amazon EC2: General purpose instance specificationsdocs.aws.amazon.comvCPU, memory and network for every size, including m7i.48xlarge (192 vCPU, 768 GiB, 50 Gbps) and m8i.96xlarge (384 vCPU, 1,536 GiB).
2. Amazon EC2: Memory optimized instance specificationsdocs.aws.amazon.comThe largest boxes: x8i.96xlarge (6 TiB), u7in-32tb.224xlarge (896 vCPU, 32 TiB) and u7inh-32tb.480xlarge (1,920 vCPU, 32 TiB).
3. Amazon EC2 On-Demand Pricingaws.amazon.comLinux on-demand rates in US East (N. Virginia), October 2026: m7i at $0.0504 per vCPU-hour at every size, x8i and u7in prices.
4. Amazon EC2 User Guide: Change the instance typedocs.aws.amazon.comYou must stop an instance to change its type, so plan for downtime of a few minutes plus your application’s start-up.
5. Amazon RDS User Guide: Failing over a Multi-AZ DB instancedocs.aws.amazon.comFailovers typically take 60–120 seconds, and modifying the instance is one of the documented triggers.
6. Stack Overflow: The Architecture, 2016 Edition (Nick Craver)nickcraver.comDaily traffic, a few big SQL Server primaries, 11 web servers when 1 would do, and why SQL CPUs are expensive.
7. The Next Gen Database Servers Powering Let’s Encrypt (2021)letsencrypt.orgOne MariaDB database by design, upgraded from 24 to 64 cores and 1 to 2 TB of RAM; median API time from about 90 ms to 9 ms.
8. How to Quantify Scalability: the Universal Scalability Law (Neil Gunther)perfdynamics.comThe USL with contention (α) and coherency (β), its reduction to Amdahl’s law when β = 0, and the peak in throughput when β > 0.
9. AWS Well-Architected Framework, Reliability pillar: Design principlesdocs.aws.amazon.com“Scale horizontally to increase aggregate workload availability”: replace one large resource with many small ones.
10. Azure Architecture Center: Design to scale outlearn.microsoft.comAvoid instance stickiness, find the bottleneck first (more web servers don’t help a database bottleneck), scale across zones, and model contention and coherency.