The real difference is the input, bounded or unbounded.
Batch runs a job over a finished set, like yesterday’s orders, and stops. A stream job never ends, updating its answer seconds after each event.
80 min
stale, from an hourly job that runs 20
Staleness is interval plus runtime. A nightly 2 am job with a 3-hour run can be 27 hours behind; a stream, seconds.
Window clicks by when you see them?
Bad: Processing time
A 2-minute blip moves 12:00’s clicks into 12:02, billing both minutes wrong.
Good: Event time
Count when it happened; a watermark says when the window is done.
A watermark guesses when a window is complete.
It declares that nothing stamped before t should still arrive. Too fast and data lands late; too slow and every dashboard waits.
Exact day-long dedupe in a stream means remembering 86.4 billion click IDs.
The trick
That’s about 1.4 TB of state. A nightly batch job sorts or hashes the day once instead.
Stream for speed, batch for truth.
Flink counts clicks per ad per minute for dashboards and budget pacing in about 75 s. A nightly Spark job dedupes and bills. Each morning, compare.
Exactly-once stops at the engine’s edge
- Flink’s state counts each event exactly once
- An SMS sent from the job goes again on replay
- Give it an idempotency key from the input event
Remember
Pick by freshness. Batch is cheap, complete and easy to rerun; a stream answers in seconds and pays in state, event time and on-call. Often both, stream for speed and batch for truth.