Kafka

Apache Kafka is a distributed, replicated, append-only log. Producers append records to topics that are split into partitions, brokers keep those records for days rather than deleting them once read, and any number of consumer groups read them at their own pace by remembering a single number per partition, the offset. In an interview it is the default answer to “move lots of events between services, in order per key, with replay”, provided you can say exactly what it guarantees, what it costs and where it stops.

Picture it first: a courier hub’s register books

A courier hub records every parcel event (picked up, sorted, out for delivery) in 12 register books. The book is chosen by destination PIN code, so all events for one PIN code land in one book, in order. Lines are numbered and only ever added at the bottom.

Billing, SMS alerts and analytics each keep their own bookmark in every book, so slow analytics never holds up billing, and the SMS team can move its bookmark back to redo yesterday after a bug. Within a team, each book goes to one clerk, so a 13th clerk has nothing to read. Each book is photocopied to two other branches, and pages older than a week are shredded, read or not.

Map it back: the set of books is a topic, one book is a partition, a line number is an offset, the PIN code is the key, a team is a consumer group, a clerk is a consumer, a bookmark is a committed offset, the photocopies are replicas, and the shredding schedule is retention.

Data model: topics, partitions, offsets

A record has a key, a value, a timestamp and optional headers. The broker gives it an offset, its permanent position in one partition. A topic is a named stream split into partitions, each an independent, totally ordered log.

The producer picks the partition. With a key, the default partitioner computes murmur2(key) % partitions, so a key keeps its partition while the count stays the same. Without a key, it fills a batch on one partition before moving on (the “sticky” partitioner). Consumers pull batches from the partition leader, starting at their offset.

On disk, a partition is a directory of segment files named after their first offset. The broker appends to the active segment and rolls a new one at 1 GiB (segment.bytes). Retention deletes whole closed segments, so dropping old data means deleting files, not rewriting them. A production-ready topic:

kafka-topics.sh --bootstrap-server broker1:9092 --create --topic order-events \
  --partitions 12 --replication-factor 3 \
  --config min.insync.replicas=2 --config retention.ms=604800000

Say the numbers explicitly: by default a new topic gets one partition, replication factor 1 and min.insync.replicas of 1.

Retention or compaction

Reading a record does not delete it. Each topic sets a cleanup.policy:

• delete (default) drops segments older than retention.ms (7 days) or beyond retention.bytes. Good for standalone events: clicks, GPS pings, logs.
• compact keeps at least the latest record per key. A null value is a tombstone that deletes the key and is itself removed after delete.retention.ms (1 day). Good for current-state streams: prices, profiles, a table’s rows.

Two caveats: the active segment is never compacted, so a reader from offset 0 still sees some superseded values; and compaction is not an index. Kafka cannot answer “what is K2’s price?” without a scan, so consumers load the topic into a cache or database.

Guarantees, precisely

Durability: acks, the ISR and min.insync.replicas

Each partition has one leader, which takes all writes, and followers that fetch from it like consumers. The leader tracks the in-sync replicas (ISR); a follower that stays behind for replica.lag.time.max.ms (30 s), or whose broker stops heartbeating to the controller, is dropped. A record is committed once every ISR member has it, and consumers read only up to the high watermark, so they never see a record a failover could lose.

The producer’s acks chooses when it hears “ok”:

• acks=0: never waits. A broker failure silently drops records.
• acks=1: the leader has it. If the leader dies before followers copy it, an acknowledged record is lost.
• acks=all: every current ISR member has it. The client default since Kafka 3.0, together with enable.idempotence=true (KIP-679).

The trap: “all” means all replicas currently in the ISR, which can shrink to the leader alone. min.insync.replicas closes the hole: with acks=all, the leader rejects writes (NotEnoughReplicas) when the ISR is smaller than this. The standard setting is replication factor 3, min.insync.replicas=2, acks=all.

The documented guarantee: a committed record is not lost as long as one in-sync replica stays alive. With the standard setting you survive one broker failure with no loss and no write outage. Spread replicas across availability zones with rack awareness, or one zone outage takes two of them. Three fine points:

• No fsync per write. “Acknowledged” means “in the page cache of every ISR member”, not “on disk”; the design notes put an fsync on every write at two to three orders of magnitude of throughput. A correlated power loss across all replicas can lose the tail.
• Unclean leader election is off by default. If every ISR member is gone, the partition waits rather than promote a stale replica; turning it on trades data loss for availability.
• Current 4.x releases are stricter. The high watermark no longer advances while the ISR is below min.insync.replicas, and the controller tracks Eligible Leader Replicas that are safe to promote (ELR: preview in 4.0, default for new clusters from 4.1).

Ordering: per partition, and only per partition

Records in one partition are delivered in the order the leader appended them. There is no order across partitions, so “ordered per user” means “keyed by user”. Three things break it:

• Changing the partition count. % 12 and % 16 send a key to different partitions, splitting one user’s history across logs. Kafka does not move old data, and cannot reduce a partition count at all.
• Retries without idempotence. With several requests in flight, a retried batch can land behind a later one. The idempotent producer (default) prevents this with up to 5 in flight.
• Thread pools in the consumer, unless records are routed to workers by key.

The cost of strict ordering is covered in message ordering and the sequential convoy.

Delivery: at-least-once by default, exactly-once inside Kafka

A consumer’s progress is one committed offset per partition, stored in the internal __consumer_offsets topic. When you commit sets the semantics:

• At-most-once: commit, then process. A crash skips records.
• At-least-once: process, then commit. A crash replays records, so writes must be idempotent. The normal choice; the 2011 paper promised only this.
• Exactly-once (since 0.11, 2017): each record’s effect appears once, but only for Kafka → process → Kafka.

A trap in the defaults: enable.auto.commit=true commits, every 5 seconds, whatever poll() has returned. That is at-least-once if you finish each batch before polling again, and silently at-most-once if you hand records to another thread.

Exactly-once has two parts (Confluent, 2017):

1. The idempotent producer. Each producer gets an ID and numbers its batches per partition; the broker drops a batch it already has. The numbers live in the replicated log, so a new leader still spots a duplicate retry.
2. Transactions. A producer with a transactional.id writes to many partitions and commits the consumer’s offsets atomically; read_committed consumers see only committed transactions. An old instance returning from a long pause has a stale epoch and is fenced, so a “zombie” cannot commit.

The loop in Java (error handling trimmed; after an abort, rewind the consumer to its committed offsets):

producer.initTransactions();                       // fences older instances with the same id
while (running) {
  ConsumerRecords<String, Payment> batch = consumer.poll(Duration.ofMillis(200));
  producer.beginTransaction();
  for (var rec : batch) producer.send(toLedgerEntry(rec));
  producer.sendOffsetsToTransaction(nextOffsets(batch), consumer.groupMetadata());
  producer.commitTransaction();                    // or abortTransaction() and rewind
}

The cost is modest: in 2017, 1 KB records in 100 ms transactions lost about 3% of throughput against an in-order at-least-once producer. Kafka Streams wraps it all in processing.guarantee=exactly_once_v2.

How it scales

Partitions are the unit of everything

A partition is the unit of write parallelism (leaders spread across brokers), read parallelism (one consumer per partition per group) and placement (its local data lives on one broker). The 2011 paper chose this on purpose: sharing a partition inside a group would need locking and per-message state. To scale, add partitions and brokers; to rebalance, move partitions.

Why one broker is fast

A 2014 benchmark pushed 2,024,032 records/s (100-byte records, about 193 MB/s) into three brokers with three-way replication and leader-only acknowledgement, on machines with SATA disks and 1 Gb Ethernet. The reasons:

• Sequential I/O. Appends and in-order reads avoid seeks, so speed does not fall as data grows.
• The page cache, not the heap. Consumers that keep up read from memory, and the cache survives a broker restart.
• Zero-copy. sendfile moves bytes from page cache to socket without passing through the JVM. TLS disables this path.
• Batching and compression end to end. Producers batch per partition (batch.size 16 KiB, linger.ms 5 ms since 4.0) and compress whole batches, which brokers store and serve as is.
• No per-message bookkeeping. A consumer’s state is one integer per partition, which is why fan-out is cheap: ten groups cost ten reads, not ten copies.

Consumer groups, lag and rebalancing

A broker acting as group coordinator assigns partitions to a group’s consumers and reassigns them when one joins, leaves or stops heartbeating: a rebalance. Groups commit their own offsets and never block one another.

Lag is the number to watch. The CLI shows it per partition (trimmed):

$ kafka-consumer-groups.sh --bootstrap-server broker1:9092 --describe --group fraud-check
TOPIC        PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
ride-events  0          5180            5200            20
ride-events  2          3000            12400           9400

Rebalances used to hurt. In the classic protocol clients compute the assignment, and its original “eager” form makes every consumer drop every partition while the group re-forms; cooperative assignment and static membership (group.instance.id) softened that. The new protocol (KIP-848) computes assignments on the broker and is fully incremental. It is GA since 4.0 and on by default on brokers, but clients opt in with group.protocol=consumer until it becomes the client default, planned for 5.0.

The control plane: KRaft replaced ZooKeeper

Cluster metadata (topics, leaders, ISRs, configs) used to live in ZooKeeper. KIP-500 moved it into Kafka: a quorum of controllers runs Raft over a metadata log (__cluster_metadata), and brokers fetch changes the way consumers fetch records. The timeline:

• 3.3: KRaft production-ready for new clusters.
• 3.5: ZooKeeper mode deprecated.
• 3.9 (November 2024): the last release that can run with ZooKeeper; migrations happen on 3.x.
• 4.0 (March 2025): ZooKeeper removed; KRaft only.

Run 3 or 5 controllers; a majority must be up, so 3 tolerate one failure and 5 tolerate two.

Tiered storage and share groups

• Tiered storage (KIP-405, production-ready since 3.9) uploads closed segments to object storage such as S3. Brokers keep recent data locally (local.retention.ms) while retention.ms can be months, so retention no longer sets disk size and moving replicas copies less. Apache Kafka ships the interface, not an S3 plugin, and compacted topics are not supported.
• Share groups (KIP-932, “Queues for Kafka”, production-ready in 4.2) let many consumers share one partition. Each record is locked to one consumer (30 s by default), acknowledged individually and retried up to 5 times; 4.4 added dead-letter topics. You can run more consumers than partitions, but lose ordering; delivery is at-least-once.

Sizing partitions: the arithmetic

Jun Rao’s rule: measure one partition’s produce rate p and consume rate c; for target throughput t, use at least max(t/p, t/c) partitions. Take a Flipkart-style sale with 50,000 order events/s at peak, about 1 KB each:

• Producing: 50,000 × 1 KB ≈ 50 MB/s. At a measured 10 MB/s per partition, 50 / 10 = 5 partitions.
• Consuming: each event needs a 2 ms database write, so one consumer handles 1 s / 2 ms = 500 events/s, and 50,000 / 500 = 100 partitions.
• The consumer side wins. With headroom, about 120–150 partitions; batching the database writes would lower that.

Storage at an average of 10,000 events/s: 10 MB/s × 86,400 s ≈ 864 GB/day; × 3 replicas ≈ 2.6 TB/day; × 7 days ≈ 18 TB across the cluster, most of which tiered storage can move off broker disks.

Limits and numbers

Defaults are from the Kafka 4.4 configuration reference (October 2026). Benchmark figures are from a stated year and hardware: quote them as orders of magnitude.

When to pick it, and when not

Pick Kafka when several consumers need the same events, you need per-key order at high volume, you want to replay history after a bug or to build a new view, or one change stream feeds processors, search indexes and warehouses.

Pick something else when:

• It is a job queue, where each task needs its own retry, delay, priority or DLQ: SQS or RabbitMQ do this out of the box (or share groups, if you already run Kafka 4.2+).
• You need request/response. Use RPC.
• Payloads are large. Put them in blob storage and send a reference.
• You need lookups by key. Kafka is not a database.
• You want a queue per user. Partitions are heavyweight; key by user on a shared topic.
• Volume is small and the event must commit with a database write. A transactional outbox is enough, and is also how events usually get into Kafka without dual writes.

Common interview usage

Going deeper: failure modes

Hot partitions

One key is capped at what one partition and one consumer can do. A viral product in a sale, or one giant merchant in a UPI-style payments stream, can swamp a partition, as P2 in the consumer-group figure shows. More partitions do not help. Fixes, each with a price: salt the key (merchant_42#0 … #7) and lose strict order for that merchant; pre-aggregate before Kafka or in a first stage; or give the giant its own topic.

Slow consumers and rebalance storms

If a batch takes longer than max.poll.interval.ms (5 minutes), the consumer is evicted and its partitions move, often to an equally slow consumer, and the group churns. Keep batches small, use static membership so restarts do not reassign, and on 4.x use the KIP-848 protocol so one slow member does not pause the rest. Alert on lag that grows steadily; see backpressure and load shedding.

Poison pills

A record that always fails (bad schema, a bug) blocks its partition: the consumer cannot commit past it and retries forever. Consumer groups have no built-in DLQ, so you add a retry topic with a delay, then a dead-letter topic after N attempts, plus an alert and a replay tool. Share groups (4.4+), Kafka Connect and Kafka Streams (4.2+) have this built in.

Quiet data loss

Kafka rarely loses data by itself; configurations do:

• acks=1, or acks=all with min.insync.replicas=1, and a leader that dies before followers copy the tail.
• Replication factor 1, because nobody overrode the default.
• Unclean leader election turned on to end an outage.
• Auto-commit with asynchronous processing.
• Lag longer than retention. A consumer down for 8 days on a 7-day topic has no valid offset, and with auto.offset.reset=latest it silently jumps to the end. Alert on lag in time, not just records.

Duplicates, everywhere

Assume every consumer sees some records twice: after a crash before a commit, a rebalance, a retry with idempotence off, or a deliberate replay. Make consumers idempotent (upsert by key, a processed-IDs table, conditional writes); it is cheaper than exactly-once and covers external systems too. See delivery guarantees and DLQs.

Cheat sheet

How to say it in the interview

“I’d put a Kafka topic between the order service and everything downstream: 48 partitions keyed by order_id, so each order’s events stay in sequence while load spreads out. Replication factor 3, min.insync.replicas 2, acks=all: an acknowledged event survives a broker loss, and with two brokers down we stop taking writes rather than risk losing them. Billing, notifications and the search indexer are separate consumer groups, so a slow indexer never delays billing, and we can replay seven days after a bug. Consumers are at-least-once and idempotent; I’d use Kafka transactions only where we read and write Kafka, as in a stream processor.”

How do you keep a user’s events in order? Key by user ID: one key maps to one partition, which one group member consumes, and the idempotent producer keeps retries in order. Don’t add partitions later, and don’t fan records out to a thread pool unless you route by key.

Can Kafka lose an acknowledged message? With acks=all, min.insync.replicas=2 and unclean election off, only if every in-sync replica fails before flushing, say a simultaneous power loss, since Kafka does not fsync each write. With acks=1, yes: a leader can acknowledge and die before replicating.

How many partitions would you create? max(t/p, t/c). For 50 MB/s at 10 MB/s per partition, producing needs 5; if each consumer handles 500 events/s, 50,000 events/s needs 100. Then headroom, because the count can grow but never shrink, and growing remaps keys.

Is Kafka exactly-once? Inside Kafka, yes: the idempotent producer removes retry duplicates, and transactions commit outputs and input offsets atomically for read_committed readers. External side effects need idempotency keys or offsets stored in the same database transaction.

What happens if one message keeps failing? It blocks its partition, because the consumer can’t commit past it. Send it to a retry topic with back-off, then a dead-letter topic after a few attempts, alert, and replay once fixed.

Where to go next

• Design a distributed message queue (Kafka): designing the log, replication and controller yourself.
• Message queues and publish/subscribe: the patterns Kafka implements.
• Delivery guarantees: at-least-once, exactly-once and DLQs: the semantics in general.
• Message ordering: per-key queues and the sequential convoy: what strict order costs.
• Change data capture and transactional outbox: getting database changes into Kafka safely.
• Event sourcing and CQRS: when the log becomes the record.
• Flink and stream processing: windows, state and exactly-once sinks on top of Kafka.
• ZooKeeper and etcd and consensus and leader election: the coordination ideas behind KRaft.

References and further reading

1. Apache Kafka 4.4 documentation: Designkafka.apache.orgPersistence and page cache, batching and zero-copy, push vs pull, delivery semantics, ISR replication, unclean election, log compaction, quotas and share consumers.
2. Apache Kafka 4.4 documentation: configuration referencekafka.apache.orgCurrent defaults quoted in the article: retention, segment size, max batch size, min.insync.replicas, acks, idempotence, linger.ms, poll and session timeouts.
3. Apache Kafka 4.0.0 release announcement (March 2025)kafka.apache.orgFirst release without ZooKeeper; KIP-848 consumer protocol GA; Queues for Kafka in early access; Eligible Leader Replicas preview.
4. Apache Kafka 4.2.0 release announcement (February 2026)kafka.apache.orgShare groups (Queues for Kafka) declared production-ready.
5. KIP-500: Replace ZooKeeper with a self-managed metadata quorumcwiki.apache.orgWhy metadata moved into a Raft-replicated log inside Kafka, and how brokers fetch it.
6. KIP-932: Queues for Kafkacwiki.apache.orgShare groups: consumers beyond the partition count, per-record acknowledgement, delivery counts, at-least-once and no ordering guarantee.
7. Kafka: a Distributed Messaging System for Log Processing (Kreps, Narkhede, Rao, NetDB 2011)notes.stephenholiday.comThe original design: partition as the unit of parallelism, pull-based consumers, stateless brokers, at-least-once delivery.
8. Benchmarking Apache Kafka: 2 Million Writes Per Second (On Three Cheap Machines) (Kreps, LinkedIn, 2014)engineering.linkedin.comThe 2014 hardware, throughput and end-to-end latency figures used in the limits table.
9. Exactly-once Semantics Are Possible: Here’s How Kafka Does It (Confluent, 2017)confluent.ioIdempotent producer sequence numbers, transactions, read_committed, zombie fencing and the measured 3% throughput cost.
10. How to choose the number of topics/partitions in a Kafka cluster? (Rao, Confluent, 2015)confluent.ioThe max(t/p, t/c) partition formula and the ZooKeeper-era per-broker partition guidance.