18 GB/s
out at peak, for 3 GB/s in
3M records a second at a festive-sale peak, 1 KB each. Followers and four consumer groups multiply every byte. Egress, not ingest, is the big number.
Where do messages live?
Bad: A table of messages
Insert, update, delete: random I/O per message. Stalls at a few thousand a second.
Good: An append-only log
Sequential appends; reading deletes nothing. Millions a second on ordinary disks.
When may the leader say “got it”?
With acks=all, once every in-sync replica has the batch. The high watermark is the lowest log end among them; consumers read only below it.
If the broker tracked each delivery, that’s 12M acknowledgements a second at peak.
The trick
Each group commits one offset per partition: 96,000 in all, 19,200 commits a second. Replay is moving the number back.
42 → 18
brokers, once old segments move to object storage
A week on three replicas is 1.8 PB, so disks set the fleet while NICs idle at 14%. Keep 12 hours local; tier the rest.
The ack is lost, so the producer retries. Written twice?
No. Each batch carries a producer ID and sequence numbers. The leader sees 0–4 again, skips the append and returns the original offsets.
Exactly once, honestly
- Idempotent producer: a retry is never appended twice
- Transactions commit outputs and input offsets together
read_committedreaders see only committed work- Beyond Kafka, store offset or event ID with the result
Remember
A partitioned, replicated, append-only log. Producers key and batch to leaders; acks=all waits for the in-sync replicas; consumer groups pull from their own offsets; a Raft quorum owns metadata.