Topic → partitions → offsets. The key picks the partition.
murmur2(key) % partitions keeps a key on one partition, in order, until the partition count changes. Order exists only within a partition.
Read it, and it’s gone?
Bad: Like a queue
Acknowledged messages are deleted: nothing left to replay.
Good: A log kept 7 days
Reads delete nothing; retention drops old segments. Any group can rewind and replay.
acks=all waits for the in-sync replicas, however few.
With min.insync.replicas=2 and three replicas, losing one broker loses nothing. Lose two and writes stop rather than live on one disk.
100
partitions, set by consumers, not bytes
A Flipkart-style sale peaks at 50,000 order events/s. At 10 MB/s per partition, the bytes need 5. A 2 ms database write per event needs 100.
One consumer per partition per group. Extras sit idle.
Groups keep separate offsets, so fraud-check lagging 9,400 on P2 never slows analytics. Scale reads by adding partitions.
Exactly-once is on. Your processor writes ledger entries, then sends an SMS.
The catch
The SMS can go twice. Transactions cover only Kafka topics and offsets; a replay repeats anything outside. Use idempotency keys.
Pick it, or not
- Pick: fan-out, per-key order, replay, high volume
- Not: job queues needing per-task retry, delay, DLQ
- Not: request/response, big payloads, lookups by key
- Not: low volume, atomic with a DB write
Say it like this
Pick Kafka when many consumers need the same events in per-key order, with replay. Key by order_id; RF 3, min.insync.replicas=2, acks=all survives a broker loss. Idempotent consumers. Not for job queues.