A dataflow of parallel subtasks. keyBy brings each key together.
Every click for ad 42, and its counter, lands in one subtask: local state, no locks, no remote calls.
Which clock decides the window?
Bad: Processing time
The machine’s clock. Run ten minutes behind and clicks land in wrong windows.
Good: Event time
The click’s timestamp. Results depend only on the data; replays repeat.
When is 10:00–10:01 done? When the watermark passes 10:01.
Watermark = highest event time seen − 5 s. Clicks arriving behind it are late, and by default dropped.
Checkpoints save Kafka offsets and state at one logical point.
A barrier flows with the records. After a crash, Flink restores both and replays; replayed records meet state that never saw them.
Flink is exactly-once. After a crash, does each click run your code once?
The catch
No. Records since the last checkpoint re-run; only state counts them once. Sinks must be transactional or idempotent.
128
key groups by default: the rescaling ceiling
Keys hash into key groups; each subtask owns a range, 32 each at parallelism 4. Parallelism 200 is impossible, and changing the ceiling discards state.
Pick it, or not
- Pick: large state, event time, joined streams
- Not: Kafka to Kafka, modest state → Kafka Streams
- Not: stateless, or minutes of latency are fine
- Not: serving queries; write results to a store
Say it like this
Pick Flink for large keyed state and event-time windows. Watermarks close windows; checkpoints restore state with offsets. Exactly-once ends at the sink, so upsert by ad and minute. Not for stateless work.