Batch vs stream processing

Batch processing collects a bounded set of data, such as yesterday’s orders, and runs a job over all of it on a schedule; you get exact answers when the job finishes. Stream processing runs a job that never ends over an unbounded flow of events and updates its answers seconds after each one arrives. You trade freshness against cost, completeness and simplicity, and most real systems run both.

Picture it first: the monthly bill and the prepaid meter

There are two ways to charge a household for electricity.

The old way: a meter reader notes the dial once a month, and an overnight billing run turns thousands of readings into bills. It is cheap, exact and easy to fix: if a tariff was wrong, the office reruns the bills. But you discover the geyser was left on all month only when the bill arrives.

The new way: a prepaid smart meter deducts every unit as you use it, sends an SMS at ₹100 left and cuts supply at zero. You find out within minutes. But the meter must stay online, a backend must absorb millions of tiny updates around the clock, and when the network drops for an hour, that hour’s readings arrive late and must still land in the right hour.

Even the prepaid company sends a monthly statement that settles the exact amount: the live balance is for speed, the statement is the truth.

Map it back: the monthly run is batch (a bounded input, a schedule, output at the end). The meter is stream processing (an unbounded flow, a result per event). The delayed upload is late, out-of-order data, rerunning the bills is reprocessing, and the statement beside the live balance is the hybrid most real pipelines use.

Batch and stream, precisely

The real difference is the input. Tyler Akidau’s Streaming 101 calls data bounded (finite: yesterday’s orders, a month of invoices, a partner’s file) or unbounded (it never ends: clicks, payments, GPS pings), and keeps the word “streaming” for an engine designed for unbounded data.

Batch processing runs a job over a bounded input, publishes all its output at the end and stops: MapReduce, a Spark job, an INSERT … SELECT in a warehouse, or cron running a script at 2 am. For an unbounded source, it slices the input into chunks, one per hour or day.

Stream processing runs a job that never finishes. Each event updates some state (a count, a total, a fraud feature) and results flow out within seconds: Apache Flink, Kafka Streams, Spark Structured Streaming.

Micro-batching sits in between. By default, Spark Structured Streaming runs a stream as back-to-back small batch jobs, with end-to-end latency as low as about 100 ms and exactly-once guarantees; its experimental continuous mode reaches about 1 ms, but only at-least-once and only for map-like queries. Akidau counts micro-batch as streaming, and so should you: it keeps state between batches and waits on watermarks. From the other side, Spark’s available-now trigger processes whatever has arrived and stops, so a streaming query can run as a scheduled batch job.

The staleness arithmetic of a schedule

A batch result is stale by up to the interval plus the runtime:

• Hourly job, 20-minute runtime. An event at 10:00:01 waits for the 11:00 run to finish at 11:20: 80 minutes. One at 10:59 waits 21, so the average is about 30 + 20 = 50 minutes.
• Nightly job at 2 am, 3-hour runtime. An event at 2:01 am is published at 5 am the next day: about 27 hours.

A stream’s staleness is the time to log, process and write the event, typically a few seconds, plus any deliberate wait for late data.

What batch buys you, and what it costs

What you gain:

• Throughput and cost per record. Big sequential scans of columnar files such as Parquet keep every core busy, and the cluster exists only while the job runs, so cheap spot capacity works.
• Complete data. By 2 am almost all of yesterday has arrived, even the phone that left airplane mode at midnight, so joins, distinct counts and deduplication are exact.
• Simple fault tolerance. MapReduce re-executes a failed machine’s tasks and uses an atomic rename so each output file comes from exactly one attempt; with deterministic functions, the result equals one sequential run. The MapReduce paper describes a job that kept going while network maintenance cut off 80 machines at a time.
• Easy reprocessing and testing. Fix the bug, rerun on the same input, overwrite the output. Nathan Marz called this human fault-tolerance: raw data is immutable, so every view can be recomputed.

What you pay:

• Latency. Interval plus runtime; nothing inside the interval can be acted on.
• Spikes and late failures. The cluster idles most of the day, then every team’s nightly jobs fight over 2 am. A job that dies at 4:30 am after two and a half hours restarts, and the 9 am report is late. Stragglers hurt too: the paper’s sort took 44% longer without backup copies of the last slow tasks.
• Cliff edges. A session running 10:55–11:05 is split across two hourly runs.
• Growth. On a Big Billion Days–style sale day, 5× the input turns a 2-hour job into 10 hours, overlapping the next run.

What streaming buys you, and what it costs

What you gain:

• Low latency. Act while it still matters: block a fraudulent payment, page an engineer, raise delivery fees as the rain starts.
• Smooth resource use. Work is spread evenly over the day, as Akidau notes; there is no 2 am stampede.
• A natural fit for unbounded data. Sessions are built whole, with no cut at the top of the hour, and the output can be an action (an alert, a block) rather than a table.

What you pay:

• Time. Events arrive late and out of order, so you need event time, watermarks and a late-data policy, covered next.
• State. Counts, joins and deduplication keep per-key state while windows are open; it must be sized, checkpointed and carried across upgrades. See stateful vs stateless design.
• Exactly-once takes work: checkpoints, a replayable source and a transactional or idempotent sink.
• Harder reprocessing. You can replay only what the log still holds.
• Always on. An on-call rota, upgrades that carry state (Flink savepoints), backpressure when a sink slows, and a bill for every quiet hour at 4 am.

Event time, watermarks and late data

Two clocks matter. Event time is when something happened on the device; processing time is when your job sees it. The gap varies: a rider’s phone loses signal in a basement, a flight lands and a hundred phones leave airplane mode, a cross-region link congests.

Windowing by processing time is simple and never late, but a 2-minute network blip moves clicks from 12:00 into 12:02, and both minutes are billed wrongly. Anything about when things happened (billing, user behaviour, anomaly detection) needs event-time windows.

Then: when is the 12:00–12:01 window done? A watermark answers. In Flink’s words, a watermark carrying time t declares that no more events stamped at or before t should arrive. The usual recipe, and what Spark’s withWatermark does, is “largest event time seen, minus a bound”. When the watermark passes a window’s end, the window fires. An operator with several inputs takes the minimum of their watermarks, so the slowest input sets the pace.

Watermarks are guesses. The Dataflow paper notes they are sometimes too fast (data still arrives behind them) and sometimes too slow (one slow input holds everything back). A 10 s bound makes an event 30 s late late; a 10-minute bound puts every dashboard 10 minutes behind.

So you also need a late-data policy. Flink’s default allowed lateness is 0: late events are dropped. Raise it and the window keeps its state longer and re-fires with an updated result per late event, so the sink must upsert, not append. Anything later can go to a side output for a nightly batch to reconcile.

The Dataflow model frames this as four questions: what you compute, where in event time (windows), when in processing time you emit (triggers), and how later results relate to earlier ones. Its claim: correctness, latency and cost are knobs, and batch vs stream is a choice of where to set them.

Windows: tumbling, sliding and session

A stream never ends, so “count the clicks” must become “count the clicks in some window”. Three shapes cover almost every interview:

• Tumbling: fixed size, no overlap, so each event lands in exactly one. “Orders per city per minute.”
• Sliding (hopping): fixed size, a new window every slide, so each event lands in size ÷ slide windows. “Payments from this device in the last 10 minutes, every minute” puts each payment in 10. Flink’s docs warn that a 1-day window sliding every second copies each element into 86,400 windows.
• Session: per key, closed by a gap of inactivity, so the length follows the data: one viewer’s binge, one shopper’s visit. Batch cuts sessions at run boundaries; a stream builds them whole.

The price of every window is state, so keep aggregates, not events: Flink’s reduce and aggregate functions store one running value per window, while a process-window function buffers every element until it fires.

State, checkpoints and exactly-once

A batch job gets exactly-once almost free: rerun a dead task and atomically replace its output. A stream never ends, so it takes snapshots instead.

Flink uses asynchronous barrier snapshotting, a variant of the Chandy–Lamport algorithm: sources record their offsets and inject numbered barriers; each operator saves its state as a barrier passes, while copy-on-write lets processing continue. After a crash, Flink restores the latest checkpoint and rewinds the sources. Its docs are precise: each event affects Flink’s state exactly once, though events after the checkpoint are processed twice. End to end also needs a replayable source and a transactional or idempotent sink.

• Kafka Streams defaults to processing.guarantee=at_least_once; exactly_once_v2 wraps commits in Kafka transactions and cuts the commit interval from 30 s to 100 ms. The delivery guarantees article explains the transactions underneath.
• Spark Structured Streaming logs offset ranges in checkpoints and write-ahead logs and relies on idempotent sinks; continuous mode drops to at-least-once.

The price: a transactional sink reveals output only at commit, adding up to one checkpoint interval of latency. Often cheaper is an idempotent upsert keyed by (ad, minute), so a replay rewrites the same row.

Reprocessing: rerun the job or replay the log

Code changes and bugs happen. Jay Kreps calls reprocessing “a completely obvious but often ignored requirement”.

In batch, rerun the fixed job over the affected days, from raw input kept in blob storage by day.

In a stream, you replay only what is retained. Kreps’s recipe: keep, say, 30 days in Kafka; start version 2 of the job from the earliest offset into a new table, with extra parallelism; when it catches up, switch readers and delete the old job and table. You pay with double output storage for a while.

Do the arithmetic before promising it. At 1M events/s, 7 days is 1M × 604,800 s ≈ 605 billion events; replaying them in 12 hours needs 605B ÷ 43,200 s ≈ 14M events/s, fourteen times live capacity. So many teams backfill old days in batch from the archive and replay only recent hours, ideally running the same code in both modes on one engine.

Lambda and Kappa

Lambda, from Nathan Marz’s 2011 post, runs both. An immutable master dataset feeds a batch layer that recomputes views every few hours and a realtime (speed) layer that covers only the hours since; queries merge the two. Each batch run overwrites the speed layer’s output, so its mistakes are temporary.

The price is the same logic written twice. Kreps, who built such systems at LinkedIn, found keeping them in sync “exactly as painful as it seems like it would be”, and frameworks that compile to both offer only what the two share. The Dataflow paper adds a cautionary tale: a team ran weakly consistent streaming with a nightly MapReduce for truth, and their customers gradually stopped trusting the streaming numbers.

Kappa, Kreps’s 2014 alternative, keeps one streaming pipeline and fixes mistakes by replay. You gain one codebase and one system. You pay with long retention, spare replay capacity, temporary double storage, and the need for a stream engine with strongly consistent state.

The line keeps blurring: Spark lets you write a streaming query like a batch query, Akidau argues that a well-designed streaming engine can do everything batch does, and table formats such as Apache Iceberg and Delta Lake increasingly let streams write tables that batch jobs read.

Worked example: ad clicks, stream for speed and batch for truth

An ad network takes 1M clicks per second and needs per-minute counts per ad for dashboards and for billing. The full design is in design ad click aggregation; here we only split stream from batch.

Volume. 1M × 60 = 60M clicks a minute; 1M × 86,400 = 86.4 billion a day. At about 100 bytes a click that is 100 MB/s, 8.64 TB a day raw, perhaps 2 TB as compressed Parquet.

The stream path. Flink keys by ad ID and counts in 1-minute event-time tumbling windows with a 10 s watermark and 2 minutes of allowed lateness. Counting incrementally, state is 2M active ads × 16 bytes (8-byte ID, 8-byte count) = 32 MB per window, about 96 MB with three open, a few times that with overhead. Buffering raw clicks would hold 60M × 100 B = 6 GB per minute. Results are upserted by (ad, minute), so late firings and replays overwrite rows. A minute is complete about 60 + 10 + a few seconds ≈ 75 s after it starts. That speed is a business need: budget pacing must stop serving an ad soon after its daily budget runs out, not at 5 am tomorrow.

The batch path. Raw clicks are archived hourly to blob storage. A nightly Spark job dedupes click IDs across the day, removes bots using day-level patterns, joins campaign data and writes billable totals. Why not in the stream? Exact day-long dedupe means remembering every click ID for 24 hours: 86.4B × 16 bytes ≈ 1.4 TB of state, while a batch job sorts or hashes the day once. (Streams dedupe short windows exactly, or long ones approximately with a Bloom filter.)

Reconciliation. Each morning, compare totals per ad and alert if they differ by more than, say, 0.1%. Invoices use only the batch numbers.

The cost. Say the nightly job uses 50 spot nodes for 40 minutes: 50 × 0.67 h ≈ 33 node-hours a day. An always-on 12-node Flink cluster uses 12 × 24 = 288 node-hours, about 9× more before spot discounts. That buys 75-second dashboards and pacing; without those needs, nightly batch alone would win.

More scenarios on each side

Pick batch when the input is bounded and the answer must be exact:

• Monthly GST invoices per merchant: a calendar month, including refunds processed days later.
• Weekly restaurant payouts: commissions, refunds and penalties, where one wrong rupee is a support ticket.
• Payroll and end-of-day bank reconciliation against settlement files that arrive on a schedule anyway.
• Nightly model training on 90 days of history, index rebuilds and every backfill.

Pick stream when someone must act within seconds:

• UPI fraud scoring. The scorer sits in the payment flow with perhaps 100–200 ms, so it reads features a stream job keeps current, such as “new payees from this device in 10 minutes”. An attacker who takes over an account at 10 am and pays a new payee every 30 s makes 14 h × 120 = 1,680 payments by midnight if features are only computed nightly. With stream features, a “more than 5 new payees in 10 minutes” rule blocks the sixth, about 3 minutes in. The model itself is still trained nightly.
• A live orders-per-city dashboard for a Swiggy-style ops team moving riders when rain hits; its “same minute last week” baseline comes from batch.
• Alerting when the error rate stays above 2% for a minute; see metrics, monitoring and alerting.
• Surge pricing, ad budget pacing, and trending topics during a cricket final; see top-K trending.

Choosing: the decision guide

Ask these questions for each output, not once for the whole system:

1. How fresh must it be? Seconds, minutes, hours or tomorrow, and who acts on it?
2. Is the input bounded? A month, a file or a snapshot is batch-shaped.
3. Does late data matter? If results are about when things happened, you need event time and a late-data policy.
4. What does a wrong interim answer cost? If money depends on it, the final number comes from batch.
5. Can you replay? Know your retention and archive before promising reprocessing.
6. How big is the state? Keys × open windows × bytes per entry.
7. Who runs it at 3 am? A stream is a service with on-call; a batch is a scheduled task with retries.

The hybrid most real systems use is “stream for speed, batch for truth”: one durable log, often fed by change data capture, drives a stream for live views and an archive for nightly batches that produce the numbers of record, compared daily so a bug in either shows up as drift.

Going deeper: traps interviewers listen for

Saying “real-time” by reflex. Ask who acts on the number and how fast. A daily sales report doesn’t need Flink; a scheduled warehouse query costs a fraction of the node-hours. See cost-aware design.

Processing-time windows for anything billed. A network blip moves events between buckets. Bill on event time.

Ignoring late and out-of-order events. State your watermark bound, your allowed lateness and where later events go. See message ordering and clocks and ordering.

Claiming exactly-once end to end with a non-idempotent sink or an external API call. Exactly-once covers the engine’s state; the sink is your job.

No plan for reprocessing. Say how long the log is retained, where the archive lives, and whether you would replay or backfill.

“Just run the batch every minute.” Each run pays start-up cost, rereads input and writes small files, and an overrun delays the next. That is a micro-batch stream without the state handling, so use a streaming engine.

A stuck watermark. An operator’s event time is the minimum of its inputs, so one idle or lagging partition can stop every window firing. Watch watermark lag and consumer lag.

Cheat sheet

How to say it in the interview

“I decide per output by how fresh it must be. If someone acts on it within seconds, such as fraud features, alerts or budget pacing, it’s a Flink or Kafka Streams job with event-time windows, a watermark bound and allowed lateness, and results upserted by key so replays are harmless. If tomorrow is fine and the number must be exact, such as invoices or payouts, it’s a batch job over the archived input: cheaper, complete and easy to rerun. Most systems need both, so I keep one durable log, stream from it for speed, batch from its archive for truth, and compare the two daily.”

How do you handle events that arrive late? Window on event time, with a watermark of the largest event time seen minus a bound, say 10 seconds. Allowed lateness re-fires windows with corrected counts, which the sink upserts. Anything later goes to a side output the nightly batch reconciles.

Is your pipeline exactly-once? Flink checkpoints make each event affect the job’s state exactly once. End to end needs a replayable source (Kafka offsets) and a transactional or idempotent sink; I’d upsert by ad and minute. Outside calls carry idempotency keys, because no checkpoint covers them.

You found a bug in last week’s counts. How do you fix them? In batch, rerun the fixed job over the archived days. With Kappa, start version 2 a week back into a new table, scaled to about 14× live capacity to finish in 12 hours, then switch readers.

Why not run the batch job every minute instead? Start-up cost, rereads, small files and overrunning runs. At that frequency you need state carried between runs, which is exactly what a micro-batch or streaming engine provides.

Where to go next

• Flink and stream processing and Kafka: the engine and the log under most streaming designs.
• Stateful vs stateless design: keyed state, checkpoints and changelogs in more depth.
• Delivery guarantees and DLQs and idempotent consumers: exactly-once beyond the engine.
• Event sourcing and CQRS and change data capture: turning databases into replayable logs.
• OLTP vs OLAP and blob storage: where batch reads and writes.
• Count-min sketch and HyperLogLog: approximate counts that keep stream state small.
• Practice: ad click aggregation, web analytics, top-K trending and a job scheduler.
• Neighbouring trade-offs: push vs pull and strong vs eventual consistency.

References and further reading

1. The Dataflow Model (Akidau et al., VLDB 2015)vldb.orgCorrectness, latency and cost as the real axes; the what, where, when and how questions; why watermarks are sometimes too fast and sometimes too slow; and the team whose users stopped trusting weakly consistent streaming numbers.
2. The world beyond batch: Streaming 101 (Akidau, O’Reilly, 2015)oreilly.comBounded vs unbounded data, event time vs processing time, fixed, sliding and session windows, and how batch runs split sessions.
3. Questioning the Lambda Architecture (Kreps, O’Reilly, 2014)oreilly.comWhy keeping two codebases in sync is Lambda’s unfixable cost, and the Kappa alternative: retain the log, replay it into a second job and a new table, then switch.
4. How to beat the CAP theorem (Marz, 2011)nathanmarz.comThe original batch layer plus realtime layer design, views merged at query time, and recomputation from immutable data as human fault-tolerance.
5. MapReduce: Simplified Data Processing on Large Clusters (Dean and Ghemawat, OSDI 2004)static.googleusercontent.comBatch fault tolerance by re-executing tasks, atomic output renames, deterministic semantics, and backup tasks for stragglers (a sort took 44% longer without them).
6. Apache Flink documentation: Timely stream processingnightlies.apache.orgEvent time vs processing time, watermarks as Watermark(t), an operator’s event time as the minimum of its inputs, and late elements.
7. Apache Flink documentation: Windowsnightlies.apache.orgTumbling, sliding and session windows, allowed lateness (default 0), late firings, side outputs for late data, and state-size rules.
8. Apache Flink documentation: Fault tolerance via state snapshotsnightlies.apache.orgAsynchronous barrier snapshotting (a Chandy–Lamport variant), what exactly-once means for Flink state, and the replayable-source plus transactional-sink rule for end to end.
9. Apache Spark: Structured Streaming programming guidespark.apache.orgMicro-batch by default (latency down to about 100 ms, exactly-once), experimental continuous mode (about 1 ms, at-least-once), triggers including available-now, and watermarks.
10. Apache Kafka documentation: Configuring a Kafka Streams applicationkafka.apache.orgprocessing.guarantee with at_least_once as the default and exactly_once_v2 as the option, and the commit interval dropping from 30 s to 100 ms under exactly-once.