Stateful vs stateless design

A stateless service keeps nothing in its own memory or disk that a later request depends on, so any instance can serve any request. A stateful service keeps that kind of state locally (a login session, an open WebSocket, a chess match, a running count), so each request must reach the one instance that holds it. The choice decides how easily you scale, deploy and survive crashes, and what every request pays in latency.

Picture it first: the bank teller and the family doctor

At a big bank branch you take a token and go to whichever teller is free. None of them has met you, and it doesn’t matter: the central ledger holds everything they need. If a teller goes for lunch, the next one serves you. On salary day the manager opens two more counters, useful within minutes.

Your family doctor is different. She remembers your history and what she said last month, so each visit is quick. But you must see her. When she is on leave you wait or start again with a stranger, and a second doctor can’t take half her queue without her files.

Map it back. The tellers are stateless instances, the ledger is the shared database or session store, and the token machine is a round-robin load balancer. The doctor is a stateful instance: the state lives in her head (process memory), so routing, absence and hand-over all become problems. And the bank isn’t stateless at all. The ledger is pure state, kept in one place built for it rather than in the tellers.

Stateful and stateless, precisely

State, here, means data that a later request depends on: who is logged in, what is in the cart, which socket belongs to which user, the board in a chess match, this minute’s order count.

• A stateless service keeps none of that between requests. Each request carries what it needs (a token, an ID, the input), or the service fetches it from a shared store. Instances are interchangeable.
• A stateful service keeps it in its own memory or disk, and later requests must find it there. Each instance owns some users, connections, rooms or keys.

The Twelve-Factor App states the rule bluntly. Processes are stateless and share-nothing; anything that must persist goes to a stateful backing service such as a database, and local memory or disk serve only as a cache within one transaction. It calls sticky sessions a violation and suggests an expiring store such as Redis or Memcached for session data.

Three clarifications prevent most confusion:

1. The system is never stateless. Taking state out of a service means putting it somewhere else: a database, a cache, the client. “Stateless” describes one tier.
2. Not all memory is state. A connection pool or a warm cache is soft state: lose it and you pay latency while it rebuilds, but nobody gets a wrong answer. Hard state is data whose loss changes behaviour: a logged-out user, a lost game, a wrong count.
3. The kill test. Kill one instance at noon and send its next request to a sibling. If no user can tell, the service is stateless; if someone is logged out, disconnected or loses progress, it is stateful.

What stateless buys, and what it costs

When instances are interchangeable, operations get simpler.

• Scale out by adding copies. Ten identical instances serve roughly ten times the traffic, if the shared stores keep up (vertical vs horizontal scaling).
• Simple, even routing. Round robin or least outstanding requests, with no memory of who went where (L4 vs L7 load balancing).
• Autoscaling both ways. A new instance takes its full share at once; a removed one takes nothing with it. That makes autoscaling safe for a Big Billion Days spike.
• Painless deploys. Rolling, blue-green and canary releases just swap instances (deployment strategies).
• Cheap failure. A crash loses only in-flight requests, which are retried elsewhere, safely if the API is idempotent.
• Serverless fits. AWS’s Lambda guidance is to reuse the environment for SDK clients and static assets, but not to keep user data there (serverless architecture).

What you pay:

• A fetch on every request. Reading from main memory takes about 100 ns. The Redis docs put a typical round trip on a 1 Gbit/s network at about 200 µs, so 200,000 ÷ 100 = 2,000 times slower. A session lookup adds a few hundred microseconds to a millisecond; a database query adds more.
• A shared store on the hot path. Azure’s guidance warns that stateful parts are the likeliest bottlenecks. If the session store is down, every instance is effectively down, so replicate it and size it for peak, or it is a single point of failure.
• Bigger requests, when the client carries the state instead.

Worked numbers: sessions for 2 million users

A ticketing app has 2M concurrent users at a tatkal-style peak. Each session is 2 KB (user ID, roles, a small cart, a CSRF token), and each user makes one request every 10 seconds.

• Memory: 2M × 2 KB = 4 GB, plus per-key overhead. That fits on one cache node; a replica for failover doubles it to 8 GB.
• Load: 2M ÷ 10 s = 200,000 session reads per second. The Redis benchmark page shows one node doing about 180,000 SETs per second without pipelining, so you would split the keys over three or four shards (a distributed cache) to keep headroom.
• Latency: one extra round trip, roughly 0.2–0.5 ms inside a zone. Against a 200 ms page budget, that is noise.

Externalising sessions is cheap per request, but the store becomes real infrastructure to size, replicate and watch.

What stateful buys, and what it costs

Keeping state in the process is not a mistake to fix. For some jobs it is the only sane design.

• Memory-speed access. When a computation touches the same data thousands of times a second (a game tick, a running total, a model’s weights), 100 ns beats 200 µs by three orders of magnitude.
• Long-lived connections. A WebSocket is state, a connection bound to one server; AWS’s load-balancer docs call WebSockets inherently sticky. Chat, live scores and presence need servers that hold connections (real-time connections).
• One owner, no locks. If one process owns a chess match, it applies moves in order and pushes the board out with no distributed lock per move (design an online chess game).
• Local state for streams. Flink partitions keyed state with the stream, so every update is local and, in its docs’ words, consistent “without transaction overhead”. Counting live orders per city per minute needs no database write per event.
• Stores themselves. Databases, caches and brokers exist to be stateful well.

What you pay:

• Routing. Something must know which instance owns what: a sticky cookie, consistent hashing or a registry (service discovery).
• Rebalancing. Adding or removing an instance moves ownership, and perhaps data, while traffic flows.
• Loss on crash, unless the state is replicated or checkpointed.
• Slower deploys. You drain an instance, hand its state over or make clients reconnect.
• Uneven load. One huge group chat or one India–Pakistan match room lands on a single owner.
• Harder scale-in. Removing an instance removes the users or data on it.

Discord’s write-up on scaling Elixir to five million concurrent users shows what recovery costs at a stateful edge. A session server held up to 500,000 live sessions, an average user was in 5 guilds (Discord servers), and finding each guild’s owner on the consistent-hashing ring took about 12 µs. A restart therefore cost 500,000 × 5 × 12 µs ≈ 30 seconds of lookups alone; cutting the lookup to 0.3 µs brought it to 750 ms. With stateful tiers, recovery speed is a design problem of its own.

Moving state out of the request tier

To make an app tier stateless, list every kind of state it holds and give each a better home.

Sessions: a shared store or a signed token

Server-side sessions in a shared store. The cookie holds only an opaque, random ID; the session lives in Redis or a database under that ID, with a TTL. OWASP asks for at least 64 bits of entropy from a secure random generator and an ID that means nothing: the meaning stays on the server. You gain control: logout, a ban or a role change is one DEL or update, and idle timeouts are enforced centrally (OWASP’s common ranges: 2–5 minutes for high-value apps, 15–30 for low-risk ones). You pay a lookup per request and a store to run.

Client-side signed tokens. A JSON Web Token (RFC 7519) is a compact, URL-safe set of claims, such as user ID, roles and exp (expiry), that is signed or MAC-protected. Any instance holding the key verifies it locally, with no lookup. You pay twice:

• Size. Base64url encoding adds a third. An RS256 signature alone is 256 bytes, which becomes 342 characters; with a header and a dozen claims a token easily reaches 0.5–1 KB, on every request. At 50,000 requests per second, 1 KB × 50,000 = 50 MB/s, about 400 Mbit/s of header bytes, against 1.6 MB/s for a 32-byte session ID.
• Revocation. The token stays valid until exp. A stolen token or a revoked role can’t be cut off unless every instance checks a denylist keyed by the jti (token ID) claim, which is a shared-store lookup again.

The usual compromise is a short-lived access token (5–15 minutes) checked locally, plus a long-lived refresh token checked against a server-side store. Revocation takes effect within one access-token lifetime, and the store sees one lookup per refresh instead of one per request. More in authentication: sessions, tokens and JWT.

Everything else an instance quietly keeps

• Uploads and generated files go to object storage, ideally uploaded directly with a presigned URL.
• Caches that must agree across instances move to a shared cache such as Redis; per-instance caches can stay as soft state.
• Multi-step flows such as checkout or KYC persist each step in the database, keyed by an ID the client sends back.
• Retries need idempotency keys in a shared store, so a retry on another instance is recognised.
• Timers and background work go on a queue, so another instance picks up a dying one’s work.
• Machine-specific keys, such as cookie-encryption keys generated at boot, move to shared key management; Azure names them as a hidden cause of stickiness.

Running stateful services well

Sticky sessions: the cheapest and the weakest

A load balancer can pin a client to one instance. With duration-based stickiness on an AWS Application Load Balancer, the balancer sets an AWSALB cookie that encodes the chosen target, with a lifetime of 1 second to 7 days (default 1 day). If that target is unhealthy or deregistered, the balancer quietly picks a new one and rewrites the cookie, so the user arrives with no state.

What you pay:

• Uneven load. As Azure’s guidance notes, traffic from one heavy client can’t be spread, and long-lived users pile up on older instances.
• Lost state on death. Every session on a failed instance is gone, as in Fig. 2.
• Weak autoscaling. New instances get only new users, so a spike from existing users doesn’t spread, and removing an instance logs out its users.
• Slow deploys. Replacing an instance without kicking users out means waiting for its sessions to expire.

Stickiness is fine when the pinned state is soft: a per-user cache that speeds up repeat requests, or an in-progress upload that can restart. Use it to make things faster, never to make them correct.

Ownership by key, not by cookie

Serious stateful services route by a key: user ID, room ID or partition number. A router or client library maps each key to its owner with consistent hashing, or a coordinator keeps an assignment table. Kafka Streams, for example, creates a fixed set of tasks from the input partitions and spreads them over the running instances; when an instance dies, its tasks restart on the survivors. How keys map to owners decides how much moves when the fleet changes.

Replicate or checkpoint the state

Ownership says where the state is. You also need it to outlive its owner.

• Checkpoint and replay. Flink snapshots operator state with the input offsets to durable storage; after a crash it restores the latest checkpoint and rewinds the source (Kafka, say) to those offsets. Frequent checkpoints cost overhead, rare ones mean more replay. Checkpointing is off by default.
• Changelog and standbys. Kafka Streams logs every change to a local state store in a compacted, replicated changelog topic. Optional standby replicas, warm copies on other instances, let a failed task move to an instance that is already caught up.
• Replication. Databases and caches copy writes to followers and promote one when the leader dies; see data replication and consensus and leader election.

Platform help: StatefulSets and draining

Kubernetes runs stateless pods as a Deployment and stateful ones as a StatefulSet. Each pod gets a stable name (web-0, web-1, …), a DNS entry and its own persistent volume that follows it when rescheduled. Pods start and scale down in order, rolling updates go one pod at a time from the highest ordinal, and volumes survive scale-down. It doesn’t replicate anything: your software must still copy the data.

Draining makes stateful deploys tolerable. A target deregistering from an ALB gets no new requests, and in-flight ones get up to the deregistration delay to finish: 300 seconds by default, configurable from 0 to 3,600. For WebSocket gateways, draining means asking clients to reconnect elsewhere, a slice at a time, with jitter.

Side by side

The hybrid most systems use

Real systems are stateless where they can be and stateful where they must be.

• Stateless request tier, stateful data tier. API servers hold nothing; databases, caches, queues and object stores hold everything. The default for any CRUD product, from a food-delivery menu to a UPI collect request.
• Stateful edge, stateless core. Connection gateways hold sockets and little else. Logic runs in stateless services, history sits in the database, and a registry maps each user to a gateway so any service can reach any user via pub/sub.
• Single owners for hot entities. When one entity needs fast, ordered updates (a match, a shared document, a live auction), give it one owner chosen by key and checkpoint its state so another owner can take over.

Sizing that edge for 10M concurrent users: say each gateway holds 50,000 connections, far below Discord’s 500,000 sessions per Erlang VM, leaving room for TLS buffers and bursts. That is 10M ÷ 50,000 = 200 gateways, plus spares. At an assumed 20–50 KB per connection (measure yours), each holds 1–2.5 GB of connection state. Losing one disconnects 50,000 ÷ 10M = 0.5% of users: fine with jittered reconnects, a stampede if all retry in the same second. More in design WhatsApp.

Choosing: a decision guide

Ask these questions per tier, in order:

1. Does a request depend on anything a previous request left behind? If not, it is stateless already.
2. Can that state live in a shared store for about a millisecond per request? Usually yes: a stateless app plus a store.
3. Is the state a long-lived connection, touched thousands of times a second, or in need of a single ordered writer? Then make it stateful, with ownership by key.
4. If an owner dies, what is lost, and how fast must it return? Choose between replication, checkpoints, or letting it go and rebuilding.
5. How will you scale in and deploy? If the answer is “wait for sessions to expire”, rethink.

Traps interviewers listen for

• “Stateless means no database.” It means no state in the instance; the data moved to a store.
• “We’ll scale with sticky sessions.” Stickiness spreads new users, not load, and turns crashes and deploys into logouts.
• Forgetting the shared store. Every request still reads it, and every new app server opens more connections to it. Size it, and decide what happens when it is down.
• “JWTs make logout free.” They make lookups free. Logout, bans and role changes now wait for expiry unless you add a denylist.
• “A StatefulSet makes it highly available.” It gives stable identity and storage; replication is still your job.
• “WebSocket servers can’t scale out.” They can, if they own only connections and route messages through a registry and pub/sub.

Going deeper: failure modes at scale

Reconnect storms. When a gateway holding 50,000 sockets dies, its clients reconnect at once and hit authentication, the registry and history sync together. Spread them with jittered exponential backoff, and make reconnecting cheap.

Hot owners. A single owner is a single bottleneck. Discord’s guilds with up to 30,000 concurrent users made one process fan out every event, and publishing took 900 ms to 2.1 s at peak until they spread the sending across nodes. Split hot entities, such as a stadium-sized live-score room, into sub-owners or move fan-out to its own tier.

Two owners at once. During a rebalance the old owner may not know it lost a key and keep writing. Leases and fencing tokens reject stale owners (consensus and leader election).

Warm-up. A fresh instance with a cold cache or unloaded model is slow; ALB’s slow start mode ramps a new target’s share up linearly.

Cheat sheet

How to say it in the interview

“I’ll keep the API tier stateless: sessions go to Redis with a TTL, uploads go straight to object storage, and any instance serves any request behind a round-robin balancer. That lets me autoscale and deploy freely. I pay up to a millisecond per request for the session lookup and run Redis with a replica, which is fine at this scale. The stateful parts are deliberate: the database, the cache and the WebSocket gateways, which own connections but no business data. A registry maps users to gateways; if a gateway dies, its clients reconnect with jitter to another.”

Why not just use sticky sessions? Load gets uneven, a crash logs those users out, new instances only take new users and deploys wait for sessions to drain. I’d use stickiness only to raise cache hit rates, never for correctness.

JWT or server-side sessions? JWTs remove the lookup but can’t be revoked before exp and add up to 1 KB per request. Server sessions cost one lookup but revoke instantly. My default: a 10-minute access JWT plus a server-side refresh token, so revocation lands within minutes and the store sees one hit per refresh.

Your session store is now a single point of failure. What then? Replicate it with automatic failover, shard it when throughput demands, and choose the failure mode per feature: shoppers can browse as guests while it is down; payments fail closed.

How do you scale a stateful WebSocket tier? Gateways own only sockets; messages route through a user-to-gateway registry and pub/sub, so new gateways simply take new connections. To deploy or scale in, ask a gateway’s clients to reconnect elsewhere in jittered slices.

How does a stateful stream processor survive a crash? It checkpoints state with the input offsets (Flink) or logs every update to a changelog topic (Kafka Streams). Another instance restores the state and replays from the saved offsets, so counts are neither lost nor double-counted.

Where to go next

• Vertical vs horizontal scaling: why statelessness makes scaling out easy.
• Load balancing: L4 vs L7: routing algorithms and cookie affinity.
• Authentication: sessions, tokens and JWT: sessions versus tokens in full.
• Distributed caching and Redis: where externalised state lands.
• Consistent hashing: routing keys to owners with little movement.
• Autoscaling done right and deployment strategies.
• Design WhatsApp, an online chess game and a distributed cache: stateful tiers in full designs.

References and further reading

1. The Twelve-Factor App: VI. Processes12factor.netProcesses are stateless and share-nothing, local memory is only a single-transaction cache, sticky sessions are a violation, and sessions belong in an expiring store.
2. Azure Architecture Center: Design to scale outlearn.microsoft.comAvoid instance stickiness (in-memory sessions and machine-specific keys cause it), stateful parts are the likeliest bottlenecks, and design for scale-in.
3. AWS Elastic Load Balancing: Edit target group attributes for your Application Load Balancerdocs.aws.amazon.comSticky-session cookies (1 second to 7 days, default 1 day), re-routing when the target is unhealthy, inherently sticky WebSockets, the 300-second deregistration delay and slow start.
4. Kubernetes documentation: StatefulSetskubernetes.ioStable pod names and DNS, per-pod persistent volumes, ordered start-up, scale-down and rolling updates, and volumes kept on scale-down.
5. RFC 7519: JSON Web Token (JWT)rfc-editor.orgJWT as a compact, URL-safe, signed or MAC-protected set of claims, with the exp (expiry) and jti (token ID) claims.
6. OWASP Session Management Cheat Sheetcheatsheetseries.owasp.orgSession IDs need at least 64 bits of entropy and no meaning, session data lives on the server, and typical idle timeouts by risk level.
7. Apache Flink documentation: Stateful stream processingnightlies.apache.orgKeyed state partitioned with the stream so updates are local, checkpoints plus replay from source offsets, the checkpoint-interval trade-off, and checkpointing being off by default.
8. Apache Kafka documentation: Kafka Streams architecturekafka.apache.orgTasks fixed per input partition and reassigned on failure, local state stores backed by compacted changelog topics, and standby replicas for fast failover.
9. How Discord Scaled Elixir to 5,000,000 Concurrent Users (Discord engineering blog)discord.comSession servers holding up to 500,000 live sessions, consistent-hashing lookups that made a restart cost about 30 seconds, and hot guilds overwhelming a single owner.
10. Redis documentation: Diagnosing latency issuesredis.ioA typical round trip on a 1 Gbit/s network is about 200 µs, the cost every stateless request pays to fetch shared state.