Concurrency vs parallelism

Concurrency is about structure: a program is split into tasks that can all be in progress at the same time, so it can deal with many things at once, even on one core. Parallelism is about execution: computations literally run at the same instant on several cores or machines, so more work gets done per second. Servers that mostly wait on databases and networks need concurrency, CPU-heavy work needs parallelism, and most real systems use both.

Picture it first: one cook, four pans

A dhaba cook has four pans going: dal, rice, paneer and rotis. He stirs the dal, flips a roti, checks the rice and comes back to the dal. Only one pair of hands moves at any moment, yet all four dishes are in progress, and because each spends most of its time simmering, one cook keeps four burners busy. That is concurrency.

Now the owner hires three more cooks, one per burner. Four pairs of hands move at the same instant. That is parallelism, and it only helps when there is hands-on work to do. If every dish is just simmering, four cooks stand around watching pots.

A UPI app’s help desk works the same way. One agent keeps ten chats open, because each customer takes a minute to type; she answers whichever chat pings. Ten agents only beat one when every chat needs continuous attention.

Map it back: the cook or agent is a CPU core, the dishes or chats are requests, simmering is waiting on I/O (a database query, a call to a bank) and hands-on work is CPU time. Concurrency is how you organise the work; parallelism is how many hands you have.

The two ideas, precisely

Rob Pike drew the line in his 2012 talk “Concurrency is not parallelism”. In the Go blog’s summary, concurrency is the composition of independently executing processes, and parallelism is the simultaneous execution of (possibly related) computations: concurrency is about dealing with lots of things at once, parallelism about doing lots of things at once.

• Concurrency is a property of the design. Tasks can be in progress during overlapping periods, and their steps may interleave in any order. On one core, the OS or a runtime switches between them.
• Parallelism is a property of the execution. At some instant, more than one computation is actually running, on more than one core, GPU lane or machine.

The two are independent, so there are four combinations:

• Neither: a script that does one thing at a time.
• Concurrent, not parallel: Redis serves every command from one thread, juggling thousands of client connections.
• Parallel, with little concurrency: one job cut into equal chunks, one per core, that never talk until the end. SIMD and GPUs do this at a finer grain.
• Both: a Go server whose goroutines run on every core, or nginx with one event-loop worker per core.

Effective Go calls Go “a concurrent language, not a parallel one”, and the Go FAQ adds that concurrency only enables parallelism when the problem itself is parallel.

Waiting versus computing

First ask where a request spends its time. Take an order-history API that needs 5 ms of CPU per request (auth, logic, JSON) and spends 95 ms waiting on the database and a pricing service.

• One request at a time per core: 1,000 ms ÷ 100 ms = 10 requests/s, with the core idle 95% of the time.
• Overlap the waits: while request 1 waits, the core runs request 2’s 5 ms, then request 3’s. Busy every millisecond, it serves 1,000 ms ÷ 5 ms = 200 requests/s per core, twenty times more from the same hardware.
• How many in flight? Little’s law: items in a system = arrival rate × time inside, L = λ × W. So 200/s × 0.1 s = 20 in flight per core, and an 8-core box needs about 160 to reach 1,600 requests/s.

That service is I/O-bound: its throughput comes from concurrency, not from more cores. Now take a CPU-bound job: a thumbnail that needs 200 ms of pure CPU. A core finishes 1,000 ÷ 200 = 5 per second no matter how many are in flight, so 8 cores top out at 40/s. Put 1,000 thumbnail requests in flight and throughput stays at 40/s while Little’s law says each one waits 1,000 ÷ 40 = 25 s. Only parallelism (more cores, a GPU, more machines) raises that ceiling; extra concurrency only adds queueing. JEP 444 says Java’s virtual threads are “not faster threads”: they exist “to provide scale (higher throughput), not speed (lower latency)”.

Most services are mixed: TLS, JSON parsing, compression, image resizing and regex validation are CPU work hiding inside “I/O-bound” services. Measure CPU time against wall-clock time per request before you choose.

How a server juggles many requests

Every model answers one question: when a task waits, what holds its place, and what does that cost?

OS threads, one per request. Each connection gets a thread that blocks on reads while the kernel runs another. You gain straight-line code, real stack traces and every core used. You pay a stack per thread: 1 MB by default for Java on Linux x64 (-Xss) and commonly 8 MB for a Linux pthread (pthread_create). That is reserved address space, not RAM in use, but each thread also costs kernel bookkeeping and a context switch whenever it blocks.

Bounded thread pools. A fixed set of threads takes work from a queue. You gain capped memory and a built-in load limit. You pay with pool exhaustion: if one dependency slows from 50 ms to 5 s, every thread ends up waiting on it. Hence one pool per dependency, the bulkhead pattern.

Event loops with non-blocking I/O. Dan Kegel’s “The C10K problem”, begun in 1999, asked how one server could serve 10,000 clients at once. Thread-per-client ran out of memory first: with 2 MB stacks, a 32-bit machine with 1 GB of user address space fits 2³⁰ ÷ 2²¹ = 512 threads. The answer that won was many clients per thread: non-blocking sockets plus a kernel call that reports which are ready (epoll on Linux, kqueue on BSD and macOS). nginx runs one loop per worker process and suggests one worker per core. Node.js runs JavaScript on one loop and sends file I/O, DNS and some crypto work to a libuv pool of 4 threads by default. You gain cheap idle connections: a waiting socket costs a state object and buffers, not a thread. You pay with cooperative scheduling: a callback that computes for a second stalls every client on that loop.

async/await. asyncio, tokio, C# and JavaScript let a function suspend at each await, so event-loop code reads top to bottom. You pay with function colouring (async spreads to every caller), and one synchronous call freezes the loop: Python’s docs note that a 1-second CPU calculation delays every other task by 1 second.

Goroutines and virtual threads. The runtime multiplexes many lightweight tasks onto a few OS threads (M:N scheduling), parking a task when it blocks. Goroutine stacks have started at 2 KB since Go 1.4 and grow as needed; GOMAXPROCS, by default the core count, sets how many run in parallel. Java 21’s virtual threads (JEP 444) unmount from their carrier OS thread on blocking I/O, with one carrier per core. You gain blocking-style code at event-loop scale, on every core. You pay with runtime traps: in Java 21 a virtual thread blocking inside synchronized stays pinned to its carrier (JEP 491 fixed most cases in JDK 24), and cheap tasks make it easy to fire 100,000 queries at a database with 32 connections.

Actors and channels. In Erlang, Elixir or Akka each actor owns its state and handles its mailbox one message at a time. Go’s channels follow Effective Go’s slogan: “Do not communicate by sharing memory; instead, share memory by communicating.” You gain lock-free state and a natural single writer per entity. You pay with hot mailboxes, message copying, and request-reply turned into choreography.

Worked example: 200 requests versus 50,000 sockets

A Swiggy-style menu API takes 2,000 requests/s at 100 ms, so 2,000 × 0.1 = 200 in flight. Thread per request reserves 200 × 1 MB = 200 MB of stack, which is trivial, so an async rewrite would buy almost nothing.

A delivery-tracking gateway holds 50,000 WebSockets, each idle except for a location update every few seconds. One thread each reserves 50,000 × 1 MB ≈ 50 GB of stack (about 400 GB of address space at the 8 MB pthread default) and hands the scheduler 50,000 sleeping threads. Eight event loops hold the same sockets, and 50,000 goroutines start at 50,000 × 2 KB ≈ 100 MB.

Count concurrent tasks before you pick a model. Below a few thousand, plain threads are fine; JEP 444 says virtual threads pay off above “a few thousand” concurrent tasks that are not CPU-bound.

How work runs in parallel

Threads in one process share memory, so handing data over is cheap, but every shared variable needs synchronisation. Separate processes share nothing: safer, but data must be copied. In CPython the global interpreter lock (GIL) lets only one thread run Python bytecode at a time. It is released during I/O, so threads help I/O-bound Python, but CPU-bound Python has needed multiprocessing or native code that releases the lock. PEP 703 made the GIL optional: since 3.13 a free-threaded build runs threads in parallel, at a single-thread cost the docs put at about 1% (macOS on ARM) to 8% (x86-64 Linux), and C extensions not marked safe switch the GIL back on.

Beyond threads versus processes, parallel work comes in three shapes:

• Data parallelism: the same operation on different pieces of data: SIMD, GPUs, and across machines MapReduce or Spark. It scales best because the pieces don’t talk.
• Task parallelism: different tasks at once, such as a gateway calling five services simultaneously or a build compiling files on every core.
• Pipeline parallelism: stages working on different items at the same time, such as decoding video frame n + 1 while encoding frame n. Throughput is set by the slowest stage.

Going deeper: why parallel speedup runs out

Amdahl’s law caps speedup by the serial fraction s: on N cores, speedup = 1 ÷ (s + (1 − s) ÷ N), never more than 1 ÷ s. Say 10% of a report job is serial, merging results under one lock:

• 16 cores: 1 ÷ (0.1 + 0.9 ÷ 16) = 1 ÷ 0.156 ≈ 6.4×
• 64 cores: 1 ÷ (0.1 + 0.9 ÷ 64) = 1 ÷ 0.114 ≈ 8.8×
• any number of cores: at most 10×

The vertical vs horizontal scaling article charts the same law for nodes and adds the Universal Scalability Law, where coordination makes throughput fall past a peak.

John Gustafson’s 1988 reply was that real users grow the problem with the machine. If the run time is fixed and 10% of it is serial, 16 cores do 16 − 15 × 0.1 = 14.5× the work of one. Amdahl describes finishing a fixed job sooner, like one user-facing request; Gustafson describes a bigger job in the same time, like nightly analytics.

Four costs eat into speedup in practice:

• Lock contention: a hot lock is the serial fraction in disguise.
• Memory bandwidth: cores sharing one memory bus starve on data-heavy work.
• False sharing: two cores writing different variables on the same cache line keep stealing the line from each other (the Linux kernel docs walk through it).
• Coordination: the Go FAQ warns that programs spending more time synchronising or communicating than computing can get slower with more CPUs.

When tasks share state: races, deadlocks and fixes

Concurrency’s price is that steps from different tasks can interleave in any order, and a race condition is a bug that appears only under some interleavings. The classic is the lost update: two tasks read the same value, and the second write silently erases the first.

The other hazards come from the fixes themselves:

• Deadlock: tasks wait for each other forever. It needs all four of the conditions Edward Coffman described in 1971: mutual exclusion, hold and wait, no preemption, and a circular wait. Break any one; the usual move is to always lock in a fixed order, such as the lower account ID first in a transfer.
• Livelock: tasks keep reacting to each other without progress, like two people stepping aside in the same direction. Random jitter on retries breaks the symmetry.
• Starvation: a task never gets its turn, such as a writer behind a stream of readers. Fair locks and queues fix it.
• Priority inversion: a low-priority task holds a lock a high-priority task needs while medium-priority work runs.
• Visibility: without a lock or atomic, one core’s write may not be seen promptly by another; the language’s memory model says when it must be.

The toolkit, from most to least shared:

• Locks: simplest; you pay with contention and deadlock risk.
• Atomics and compare-and-set: lock-free counters and conditional writes; you pay with retries when contention is high.
• Immutability: data nobody can change is safe to share; you pay with copying.
• Message passing: hand data over instead of sharing it.
• Single writer, partitioned by key: one owner per wallet, per show’s seat map, per train’s tatkal quota. You pay when one key gets hot, because its owner is a queue.

The same hazards reappear between machines, where locks become distributed locks and atomics become conditional writes in the database (optimistic vs pessimistic locking, MVCC). For “one seat, many buyers”, see contention and double booking.

Where it shows up in a design

Concurrency is capped by the narrowest stage. Request threads, goroutines or coroutines are cheap, but the database behind them has a fixed number of cores. A pool of 20 connections caps database concurrency at 20 however many request threads you run. The HikariCP wiki explains why the pool should be small: a core runs one thread at a time, so past the core count pure CPU work only gets slower. Its starting point is (core_count × 2) + effective_spindle_count connections, with spindles at zero when the data is cached.

The connection pooling article sizes the pool with Little’s law in detail. Four more places the trade-off shows up:

• Bulkheads. Separate pools for the payment provider, the SMS gateway and report queries, so one slow dependency exhausts only its own share.
• Long-lived connections. Chat, live order tracking and real-time updates mean tens of thousands of mostly idle sockets: event loops, goroutines or virtual threads, never a thread each.
• CPU worker pools. Transcoding and thumbnail workers take one worker per core. In containers, count the cores you are allowed and check what your runtime sees: before Go 1.25, GOMAXPROCS counted every host core; now, on Linux, it follows the container’s CPU limit.
• GPU inference. A model server batches many requests into one pass on the GPU (parallelism) and keeps enough requests in flight to fill those batches (concurrency); see LLM inference serving.

Redis chose a single thread on purpose. Its latency guide says one thread serves all requests by multiplexing, much like Node.js. You gain atomic commands with no locks. You pay when one O(N) command, such as a huge SUNION, makes every client wait. For more cores, shard. I/O threads parallelise socket reads, writes and protocol parsing (the config suggests 3 on a 4-core box), while commands still run one at a time. See Redis.

Choosing: the decision guide

Ask these questions for each workload, not once for the whole system:

1. Where does a request spend its time? Mostly waiting needs concurrency; mostly computing needs parallelism.
2. How many tasks are in flight? Rate × time. Hundreds suit threads; tens of thousands want event loops or M:N tasks.
3. Which stage is the real cap? Database pool, downstream limit, cores or GPU.
4. What can block? One synchronous library undoes an async design.
5. Who writes shared state? Pick an atomic update, a lock or a single writer.
6. How many cores do you really have? The container’s limit, not the host’s.

Lean on concurrency for a UPI payment-status check waiting on a bank, a notification service calling SMS providers, live rider-location pings, an API gateway and cricket-score fan-out to millions of connections. Lean on parallelism for video transcoding, image resizing, month-end reports, model inference, password hashing and nightly ETL.

The hybrid most systems use has both: an event loop or M:N runtime spread across every core in front, a CPU worker pool or job queue for heavy work, a small fixed database pool behind, and one owner or conditional writes for each piece of shared state. nginx with a worker per core and a Go service with goroutines on every core both follow this shape.

Going deeper: traps interviewers listen for

“More threads make it faster.” Only while threads are waiting. Past the core count, CPU work slows down from switching and contention, and past the database pool, extra threads just queue.

“Async makes the code faster.” It raises how many requests a core holds in flight, not the speed of any one. CPU-bound work gains nothing.

Blocking inside the loop. A synchronous HTTP client, time.sleep, a big JSON parse or a catastrophic regex freezes every client on that loop.

Python threads for CPU work. With the GIL, eight threads of pure-Python CPU work use about one core. Use processes, native libraries that release the GIL, or the free-threaded build.

Unbounded fan-out. Cheap goroutines and virtual threads make it trivial to launch 100,000 calls at once. JEP 444 says to guard a scarce resource with a semaphore, not a pool of virtual threads; downstream, see backpressure.

Ignoring the serial fraction. “64 cores, so 64× faster” fails as soon as 5% of the work holds one lock: Amdahl caps it at 20×.

Cheat sheet

How to say it in the interview

“Concurrency is how I structure work so many requests are in progress at once; parallelism is running them on several cores at the same instant. This API spends 5 ms of CPU and 95 ms waiting, so it’s I/O-bound: one core can keep 20 requests in flight and serve 200 a second. I’d use an event loop or goroutines, which also spread across every core, and cap database concurrency with a 32-connection pool, about twice the database’s cores. Image resizing is CPU-bound, so it goes to a queue with one worker per core. Hot state like a wallet balance gets conditional writes or a single writer, so concurrent debits can’t lose updates.”

Does switching to async make each request faster? No. Each request needs the same CPU time and waits as long on the database. Async lets one core hold more requests at once, so throughput per core rises.

How many threads should a pool have? Start from cores × (1 + wait ÷ compute): 8 × 20 = 160 for the I/O-bound API, 8 for thumbnails. Then cap by the narrowest downstream, such as the database pool, and load-test around it.

Why is Redis fast if it’s single-threaded? Commands are short in-memory operations, the loop never blocks on a socket, and there are no locks. The price is that one slow O(N) command stalls everyone. Scale out by sharding; I/O threads take the network work.

A Node.js service’s p99 jumped after someone added a CSV export. Why? The export runs on the event loop, so every request waits behind it. Move it to a worker thread or a job queue, and return a link when the file is ready.

Where to go next

• Synchronous vs asynchronous communication: the “async” between services, not inside one.
• Vertical vs horizontal scaling: Amdahl and the Universal Scalability Law across nodes.
• Connection pooling: sizing the pool that caps database concurrency.
• Concurrency control and the lost update problem: the same races inside a database.
• Distributed locking: mutual exclusion across machines.
• Backpressure and load shedding and message queues: bounding concurrency and sizing workers with Little’s law.
• Bulkheads: separate pools so one slow dependency can’t take everything down.
• Synchronous I/O and busy front end: what stalls event loops.
• Latency vs throughput vs bandwidth: why concurrency raises throughput, not speed.

References and further reading

1. Concurrency is not parallelism (Go blog, Rob Pike’s 2012 Waza talk)go.devConcurrency is the composition of independently executing processes, parallelism the simultaneous execution of computations: “dealing with” versus “doing” lots of things at once.
2. Effective Go: Concurrencygo.dev“Share memory by communicating”, goroutines multiplexed onto OS threads, one worker per CPU for parallel loops, and “a concurrent language, not a parallel one”.
3. Go FAQ: goroutines, GOMAXPROCS and more CPUsgo.devGoroutine stacks of a few kilobytes, hundreds of thousands per process, GOMAXPROCS defaulting to the core count, and why more CPUs can make a program slower.
4. Node.js: Don’t Block the Event Loop (or the Worker Pool)nodejs.orgOne event loop plus a libuv worker pool, epoll and kqueue, the 50 MB JSON.parse that takes 1.3 s, ReDoS, and partitioning versus offloading CPU work.
5. JEP 444: Virtual Threads (OpenJDK, delivered in JDK 21)openjdk.orgLittle’s law for servers, M:N scheduling on a ForkJoinPool sized to the cores, pinning, “not faster threads”, and semaphores instead of pooled virtual threads.
6. The C10K problem (Dan Kegel)kegel.comServing 10,000 clients at once: thread-per-client versus nonblocking I/O with readiness notification, and the 512-thread virtual memory limit with 2 MB stacks.
7. Redis docs: Diagnosing latency issuesredis.ioRedis serves all requests from one thread using multiplexing, so one slow O(N) command makes every other client wait.
8. HikariCP wiki: About Pool Sizinggithub.comTime-slicing on one core, why threads beyond the core count slow CPU work down, and the (core_count × 2) + effective_spindle_count starting point.
9. Python docs: Python support for free threadingdocs.python.orgSince 3.13 a CPython build without the GIL can run threads in parallel; about 1–8% single-thread overhead; extensions not marked safe re-enable the GIL.
10. Reevaluating Amdahl’s Law (John Gustafson, 1988)johngustafson.netAmdahl’s 1967 fixed-size speedup formula and its 1 ÷ s ceiling, and the scaled-speedup alternative when the problem grows with the processors.