ZooKeeper and etcd

ZooKeeper and etcd are small, strongly consistent stores that a fleet of machines uses to agree on facts: who leads, who holds a lock, which servers are alive, what the configuration says. Each runs on three or five servers that replicate every write through consensus (Zab for ZooKeeper, Raft for etcd), keep everything in memory, and let clients watch keys and tie them to their own liveness. In an interview they answer “how do these nodes agree without split brain?”, and are the wrong answer for anything that looks like application data.

This page tracks ZooKeeper 3.9 (3.9.6 current, 3.8.7 stable) and etcd 3.6 and 3.7 (latest patches 3.6.16 and 3.7.3), as of October 2026.

Picture it first: the housing society’s register

A housing society keeps one register at the office: who holds the clubhouse key, which guards are on duty, the water-tank timings. Five committee members keep identical copies. A change counts only once the secretary has written it and at least three of the five have copied it, so the register works with two members away and no two residents are told different key-holders.

Residents can read any member’s copy: quick, but maybe a page behind, so when it matters they ask the member to check with the secretary. A guard who misses the check-in calls is struck off the duty list. People wanting the clubhouse take a numbered token and watch only the person just ahead. The key tag carries a number too: the caretaker turns away any tag older than the last he saw.

Map it back:

• Register → the coordination store: small and critical, not where the bookings live.
• Five members, three copies, one secretary → five voters, a majority quorum and a leader that orders every write.
• Reading a member’s copy → a local read (ZooKeeper’s default, etcd’s serializable); checking with the secretary → sync() or etcd’s default linearizable read.
• Check-in calls, struck off → session heartbeats or lease keep-alives; an ephemeral znode or leased key deleted on expiry.
• Tokens, watching the person ahead → sequential nodes, each waiter watching its predecessor; the number on the key tag → a fencing token.

Data model

ZooKeeper: a tree of znodes

ZooKeeper looks like a small file system in which every node, a znode, holds data and children. Reads return all of a znode’s bytes and writes replace them. Data should be kilobytes: the client rejects anything over jute.maxbuffer, just under 1 MB by default. A znode’s type is fixed at creation:

• Persistent: stays until deleted.
• Ephemeral: deleted when the session that created it ends; it cannot have children.
• Sequential (persistent or ephemeral): the server appends a 10-digit counter kept by the parent, as in lock-0000000007.
• Container (since 3.5.3): removed by the server after its last child goes; the natural parent for lock and election nodes.
• TTL (since 3.5.3, off by default): a persistent node removed once unmodified and childless for the TTL.

Each znode has a stat: version, cversion (child changes), czxid and mzxid (the transactions that created and last modified it) and ephemeralOwner. setData(path, data, version) fails if the version has moved: compare-and-set without a lock.

A session is the client’s proof of life: its timeout is negotiated between 2× and 20× the server’s tickTime, and the client pings when idle and can fail over to another server without losing it. If the ensemble hears nothing for the whole timeout, it expires the session and deletes its ephemeral nodes.

A watch is a one-shot notification set by getData, exists or getChildren: two changes before you re-register give one event, so you re-read. Since 3.6, addWatch sets persistent, optionally recursive, watches.

etcd: flat keys with history

etcd has a flat key space sorted by bytes, with no directories: get /app/ --prefix is a range query over [/app/, /app0), because 0 is the byte after /. Every change bumps one cluster-wide 64-bit revision, starting at 1; a transaction gets one revision however many keys it touches. Each key records:

• create_revision and mod_revision: the store revisions at which it was created and last modified;
• version: how many times it has changed since creation (a delete resets it to 0);
• lease: the lease it is attached to, if any.

The store is multi-version (MVCC): older values stay readable (get --rev=1150) until compaction discards history before a chosen revision. A lease is etcd’s session: grant one with a TTL in seconds, attach any number of keys, refresh it over a keep-alive stream; when it expires or is revoked, every attached key is deleted. A watch streams changes to a key or prefix from any revision still in history. A transaction is if (compare value, version or revisions) then (operations) else (operations), applied atomically.

etcdctl lease grant 10                    # lease 694d7a2c1f0e3b58 granted with TTL(10s)
etcdctl put /svc/payments/10.0.4.7 '{"port":8443,"zone":"ap-south-1a"}' --lease=694d7a2c1f0e3b58
etcdctl lease keep-alive 694d7a2c1f0e3b58 &   # stop refreshing and the key goes in ~10 s
etcdctl get /svc/payments/ --prefix       # every live payments instance
etcdctl watch /svc/payments/ --prefix --rev=1207   # replay from 1207, then follow live

Consensus underneath: Zab and Raft

Both protocols elect one leader and commit by majority; at interview depth the write path is the same:

1. A client sends a write to any server; followers forward it to the leader.
2. The leader assigns the next position (a ZooKeeper zxid, a Raft log index) and sends the proposal to every follower.
3. Each follower fsyncs it to its log and acknowledges.
4. When a majority, counting the leader, has it on disk, the leader commits, applies and replies; followers apply on hearing the commit.

Zab (Junqueira, Reed and Serafini, DSN 2011) was built for primary-backup with many writes in flight. Its key property is primary order: a leader’s changes are delivered in the order sent, after everything committed under earlier leaders. A zxid is 64 bits: a 32-bit leader epoch and a 32-bit counter. The new leader must have seen the highest zxid; its discovery and synchronization phases bring a quorum up to date and discard uncommitted proposals from old epochs, then broadcast resumes at (epoch + 1, 0). Observers receive commits and serve clients but never vote.

Raft (Ongaro and Ousterhout, 2014) numbers leadership terms. A follower that hears no heartbeat within a randomized election timeout (150–300 ms in the paper’s example) stands for election; servers vote once per term, only for a candidate whose log is at least as up to date as theirs, so a new leader holds every committed entry. etcd’s learners (since 3.4) are non-voting members promoted once caught up, so a new, empty node never enlarges the quorum.

Reads are where the two differ.

Guarantees, precisely

ZooKeeper: linearizable writes, local reads

The paper promises linearizable writes (all updates totally ordered by zxid, consistent with real time) and FIFO client order (one client’s requests run in the order sent, even when pipelined).

Reads are not linearizable: any server answers from memory, so a lagging follower returns old data. They are sequentially consistent, and a client never goes back in time: a server it reconnects to waits until it has caught up with the last zxid the client saw. Two tools close the gap:

• sync() then read. The server catches up with the leader first. The internals documentation warns that sync is not a quorum operation, so a deposed leader that doesn’t know it yet could serve it; for a strict guarantee, write first.
• Watch ordering. A client sees a change’s watch event before it can see the changed data. The paper’s ready znode pattern uses this: a new leader deletes ready, rewrites 5,000 config znodes asynchronously, then recreates ready, so nobody reading through ready sees half a configuration. Pipelined, those writes take under a second instead of 10 s at 2 ms each.

A write is acknowledged only once a quorum has logged it, and nothing a client has seen is rolled back. An expired session’s ephemeral nodes go at once; the owner learns of it only on reconnecting.

etcd: strict serializability by default

etcd promises, and Jepsen’s analysis of 3.4.3 (2020) confirmed under partitions, pauses, clock skew and membership changes, strict serializability for key-value operations: each get, put or transaction behaves as if it ran alone, in real-time order. A default read checks in: the member asks the leader for its commit index, the leader confirms it still leads with a heartbeat round to a majority, and the member answers once it has applied that index (the ReadIndex). With --consistency=s (serializable) any member answers locally: faster, available without a quorum, possibly stale.

Watches deliver events in revision order, once each, with no gaps within history, and resume from the last revision seen. They are not linearizable, so compare revisions when mixing them with reads. Healthy clusters deliver within about 10 ms.

In both systems a write that times out may still have committed: retry with a conditional write or an idempotency key, not blindly.

How it scales, and why writes don’t

Quorum arithmetic

n voters need a majority, ⌊n/2⌋ + 1, to commit or elect, so they survive n minus that many failures: three tolerate 1, five 2, seven 3. Four need 3 and still tolerate only 1: no better than three, with one more machine to fail. Odd sizes also leave at most one side of a split with a majority. Three is the minimum; five survives a failure during maintenance. The etcd FAQ suggests at most seven; Kubernetes recommends a static five.

Writes: one leader, a majority of fsyncs

Every write passes through one leader and waits on a majority of disks, so voters add no write capacity: each proposal goes to more followers, and the quorum grows. The ZooKeeper paper measured it in 2010: pure writes fell from 21k/s on 3 servers to 8k/s on 13, while pure reads rose from 87k/s to 460k/s, because every server answers reads from memory.

The latency floor is a round trip to the slower half of the cluster plus an fsync: about 10 ms on spinning disks and under 1 ms on SSDs, says etcd’s performance guide, hence a dedicated SSD for the log. Batching recovers throughput: etcd 3.2 on three 8-vCPU machines did 583 writes/s at 1.6 ms with one client, and 44k–50k writes/s at about 20 ms with 1,000.

To scale writes, partition: separate ensembles for separate jobs, as the paper advises, or Kubernetes events on their own etcd via --etcd-servers-overrides.

Reads: add non-voters, or accept staleness

• ZooKeeper: followers and observers serve reads locally; observers add capacity without joining the quorum and suit a second data centre.
• etcd: a linearizable read costs a leader round trip and a serializable one doesn’t (141k against 186k reads/s in the same benchmark); learners answer serializable reads too, but not watches. Wide fan-out is handled above etcd: the Kubernetes API server’s watch cache spares etcd one watch per controller.

Keep the data small

Everything sits in memory on every member. etcd’s default quota is 2 GiB, with 8 GiB the suggested maximum (system limits), and its hardware guide rates the cluster behind a 3,000-node Kubernetes cluster “xLarge” at just over 1 GB and 10,000 requests/s. Gigabytes, not terabytes.

Limits and numbers

Versions as of October 2026; quote benchmark rows as orders of magnitude for their year and hardware.

Recipes you should be able to sketch

Leader election

In ZooKeeper, each candidate creates an ephemeral sequential node such as /election/n-0000000012; the lowest number leads. Every other candidate watches only the node just before its own and, when it vanishes, re-lists the children to see whether it now leads. A crashed leader’s node goes with its session, waking exactly one successor. Apache Curator packages this as LeaderLatch.

In etcd, the Go concurrency package does the same with leases: NewSession grants and refreshes a lease, NewElection(session, "/election/scheduler").Campaign(ctx, "node-7") returns once its key has the lowest create revision, and Observe streams leadership changes. Either way, the old leader may still be running.

Locks and fencing tokens

A lock is the same recipe: the holder owns the lowest sequential node (ZooKeeper) or the lowest create revision under the prefix (etcd’s Mutex, which waits only on the key just before its own). Release is a delete; a crash releases on expiry.

For correctness, attach a fencing token to every write the holder makes, and have the resource reject tokens lower than the highest it has seen. Kleppmann suggests ZooKeeper’s zxid: use the lock node’s czxid, not its sequence suffix, which restarts if a container parent is deleted and recreated. In etcd, use the lock key’s create revision.

If the resource is etcd itself, put the check inside the write: a transaction comparing the lock key’s create revision (Mutex.IsOwner()). For a database, a conditional UPDATE … WHERE fence < $token works; the Redis page shows that SQL and the Redlock debate, and distributed locking the theory.

Membership and service discovery

Each instance registers an ephemeral node or a leased key such as /svc/payments/10.0.4.7:8443; clients list and watch the prefix. A crashed instance lingers until its session or lease expires, typically 10–30 s, so clients still need timeouts and retries. This is the “who is alive” half of service discovery, built on heartbeats; Consul adds health checks and DNS.

Configuration and feature flags

Keep a small config or flag set in one key or a few, and have services watch it: a change reaches every watcher in milliseconds, without a deploy (feature flags). For a multi-key change, use an etcd transaction, ZooKeeper’s multi(), or the ready-znode pattern.

Barriers yes, queues no

A barrier (start once N workers have joined) fits well. A work queue of sequential nodes is the classic misuse: every enqueue and dequeue is a consensus write, consumers race on getChildren of one huge parent, and a long child list can exceed jute.maxbuffer. Use Kafka or a real message queue.

When to pick it, and when not

Pick ZooKeeper when a system you run already depends on it (HBase master election, Solr’s cluster state, HDFS NameNode failover) or Curator’s recipes save you work. Kafka no longer needs it: Kafka 4.0 (March 2025) replaced it with KRaft, a Raft quorum of Kafka’s own controllers.

Pick etcd when you need a new coordination store or are on Kubernetes, and want gap-free watches, multi-key transactions and linearizable reads by default. Its maintainers recommend it over ZooKeeper; they are not neutral, but the advantages are real. Give your application its own cluster, not the one behind Kubernetes.

Pick neither when:

• The lock only prevents duplicate work, like a reminder SMS sent twice: a Redis SET NX PX is cheaper.
• The protected data lives in one database: SELECT … FOR UPDATE, an advisory lock or a conditional write, with no second system to fail (optimistic vs pessimistic locking).
• It is application data, large values or a high write rate: use a database, or a log for messages.
• It is a long multi-step process with retries and timers: use a workflow engine.

Common interview usage

Going deeper: failure modes

The paused leader

A JVM leader stalls in a long GC pause, its VM freezes, or its packets queue in a congested switch. Its session expires, a successor takes over, and the old leader wakes still believing it leads; HBase once had exactly this bug. Treat ZooKeeper’s Disconnected as “start nothing new” and Expired as “stop now”, but fix it at the resource with a fencing token or an epoch on every command (consensus and leader election).

Herd effect and watch storms

If 1,000 clients watch the children of /locks/job, each release wakes all of them into a getChildren stampede; watch the predecessor instead. A config change under 10,000 watchers, or an ensemble restart in which every client reconnects and re-registers, causes the same storm: spread clients across servers, add jitter, and cache in front of wide fan-out.

Treating it as a database or a queue

Big values, thousands of children under one parent and high write rates slow every request, because every member applies every write and snapshots everything. Keep bulk data elsewhere and only a pointer in the store.

Slow disks become leader elections

etcd treats disk latency as leader health: a leader that can’t fsync can’t commit, and late heartbeats trigger elections. Watch wal_fsync_duration_seconds and backend_commit_duration_seconds, and give the log a dedicated SSD. ZooKeeper’s administrator’s guide likewise demands a dedicated log device and no swapping.

etcd’s space quota: compaction and defrag

History grows with every write. Past --quota-backend-bytes, etcd raises a cluster-wide NOSPACE alarm and accepts only reads and deletes; in Kubernetes, nothing new can be scheduled. Recover by compacting old revisions, defragmenting each member (which blocks it meanwhile) and disarming the alarm; prevent it with auto-compaction, periodic defragmentation and a size alert. Compaction also costs watchers, as the figure shows.

Stretched and even-sized clusters

Spread three voters over three zones, not two: in a 2 + 1 layout, losing the zone with two kills the quorum. Across regions every write pays the round trip; set etcd’s heartbeat near the RTT and the election timeout to at least ten times it (tuning guide), and keep far copies as observers or learners (multi-region and active-active).

Losing quorum

With a majority permanently gone, the cluster can’t write, elect a leader or even change its membership; in Kubernetes, running pods carry on but nothing new is scheduled. Recovery means restoring a snapshot (etcdutl snapshot restore) into a new cluster and losing everything after it, so snapshot regularly (disaster recovery). When replacing a member, remove the dead one first, so a misconfigured newcomer can’t push the quorum out of reach.

Cheat sheet

How to say it in the interview

“For the scheduler’s leader election I’d use etcd: three members across three zones, five if we must survive maintenance plus a failure. Each scheduler holds a 10-second lease and campaigns on /scheduler/leader; the lowest create revision leads, the rest watch. Raft makes writes linearizable, so a minority partition can’t elect a second leader. A paused leader may still think it leads, so every dispatch carries its revision as a fencing token and the job store rejects older ones. etcd holds only this metadata, a few megabytes; jobs and run history live in Postgres.”

Why three or five nodes, and not four? Four need three for a majority, so they tolerate one failure, like three, with more machines to fail and a slower quorum. Five tolerate two, and odd sizes leave only one side of a split with a majority.

Can you read stale data from ZooKeeper? Yes. Writes are linearizable, but your server answers reads locally and may lag. Call sync() first, or write, for a strict guarantee. etcd reads are linearizable unless you choose serializable.

Why don’t writes get faster when you add nodes? Each write goes through one leader and waits for a majority to fsync, so more voters mean more messages and a bigger quorum: 21k writes/s on 3 ZooKeeper servers became 8k on 13. Scale reads with observers or serializable reads, and writes by splitting clusters.

What if the lock holder pauses for a minute? Its lease expires and someone else takes the lock while it still thinks it holds it. Only the resource can stop its late write: every write carries a fencing token (czxid or create revision), and anything below the highest seen is rejected.

ZooKeeper or etcd? Same job, same quorum maths. etcd adds linearizable reads by default, gap-free resumable watches, shared leases, transactions and gRPC, and already runs under Kubernetes. ZooKeeper fits where HBase, Solr or Hadoop already run it, or where Curator saves work.

Where to go next

• Consensus and leader election: Raft, Paxos and Zab in depth.
• Distributed locking and design a distributed lock: leases and fencing as a design problem.
• Service discovery, heartbeats and gossip: membership with and without a consensus store.
• Clocks, ordering and conflict resolution: why a zxid or revision beats a timestamp.
• CAP theorem and PACELC and data replication: where a CP store sits.
• Design a job scheduler and design a unique ID generator: where these recipes appear most.
• Kafka and Redis: the log that swapped ZooKeeper for KRaft, and the lock that is fine for efficiency only.

References and further reading

1. ZooKeeper: Wait-free coordination for Internet-scale systems (Hunt, Konar, Junqueira, Reed, USENIX ATC 2010)usenix.orgThe design: linearizable writes, FIFO client order, local reads and sync, the ready-znode pattern, herd-free locks, and the 2010 throughput table for 3 to 13 servers.
2. Zab: High-performance broadcast for primary-backup systems (Junqueira, Reed, Serafini, DSN 2011)marcoserafini.github.ioPrimary order, why plain Paxos did not fit many outstanding transactions, and the discovery, synchronization and broadcast phases.
3. ZooKeeper Programmer’s Guide (3.9)zookeeper.apache.orgZnode types (container and TTL since 3.5.3), the stat structure, sessions and the 2× to 20× tickTime bounds, one-shot and persistent recursive watches (3.6), jute.maxbuffer and the consistency guarantees.
4. ZooKeeper Internalszookeeper.apache.orgThe zxid as epoch plus counter, leader activation, proposal, ACK and COMMIT, and why sync followed by a read is not strictly linearizable.
5. In Search of an Understandable Consensus Algorithm (Ongaro and Ousterhout, USENIX ATC 2014, extended version)raft.github.ioTerms, randomized election timeouts, log replication and commitment, and the read-only path that confirms leadership with a majority before answering.
6. etcd API guarantees (v3.7)etcd.ioStrict serializability and linearizable reads by default, the serializable option, watch ordering and resumability, and leases.
7. etcd FAQ (v3.7)etcd.ioQuorum table and cluster size, disk latency as leader liveness, the WAL fsync and backend commit targets, and recovering from the space quota alarm.
8. Jepsen: etcd 3.4.3 (Kingsbury, 2020)jepsen.ioStrict-serializable key-value operations under faults, and etcd locks failing to provide mutual exclusion without fencing.
9. How to do distributed locking (Kleppmann, 2016)martin.kleppmann.comProcess pauses and delayed packets, fencing tokens checked by the storage, and the ZooKeeper zxid as a token.
10. Operating etcd clusters for Kuberneteskubernetes.ioAll Kubernetes objects live in etcd, a static five-member cluster is recommended, snapshot backup and restore, and what happens when quorum is lost.