1M/s
peak, when a cricket final’s ad break airs
Ten billion clicks a day average 116k a second. Size the job for 2× peak so it can catch up after an outage.
Kafka stalls. Must the redirect stall too?
Bad: Write to Kafka first
A Kafka incident becomes a redirect outage.
Good: Local spool, then ship
Group-commit to local SSD, return 302. Spools hold two days of clicks.
Exactly-once is a chain of four hops
Messages will arrive twice, so every hop must make a repeat harmless. A click affects the count once, however often it is delivered.
The job writes “+40 clicks”, crashes, and replays from its checkpoint. The sink gets “+40” again.
The trick
Write the running total under the window’s key. A replay overwrites the row: 40 stays 40, not 80.
Clicked at 12:01:50, arrived at 12:02:40. Which minute?
Count by when it happened, not when it arrived. 12:01 fires at 3 clicks, and this late click updates it to 4.
400k/s
clicks on one IPL sponsor’s ad
Every click for an ad goes to one task, sized for about 8k a second. Summing per second at each source first cuts that to about 1k records.
One click definition, run twice
- Live: provisional, about a minute old
- 02:00 IST: replay yesterday with fraud verdicts
- Fraud caught after invoicing becomes a credit
- Drift above 0.1% pages someone
Remember
Record fast, count each click once at every hop, by when it happened. Live numbers are provisional; billing replays the same code over the log.