Streaming vs batch: what is the difference and when do you use each?

In this article

1. Before you start
2. Why interviewers ask this
3. What each one is
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

Part of the data engineering interview guide. If querying data is new to you, read the Structured Query Language (SQL) interview guide first.

Why interviewers ask this

Choosing between batch and streaming is one of the first design decisions in a data pipeline, so a data engineering round can open with it. The question separates a candidate who repeats "streaming is real time" from one who ties the choice to how fresh a result must be, what it costs to run and what can go wrong. A weak answer lists two definitions. A strong answer states the freshness requirement first, then the cost, then failure modes such as late data. A Staff-level answer adds that the two are a spectrum, and names micro-batch as the middle point.

What each one is

Batch processing

Batch processing is a way of running a computation over a bounded set of stored data in one job that starts on a schedule or a trigger and then finishes, so its results are only as fresh as the last completed run.

Stream processing

Stream processing is a way of running a computation continuously over an unbounded flow of events, updating results as new events arrive, so results stay current without waiting for a scheduled run.

Micro-batch processing

Micro-batch processing is a streaming technique that cuts the incoming flow into small groups and runs one small batch job on each group, so a stream is handled with batch machinery.

Batch in plain words

Bounded data is a set of records with a known end, such as yesterday's orders. A batch job reads that whole set, computes, writes the result and stops. The classic shape is extract, transform, load (ETL) on a schedule; the Apache Hadoop questions cover the older tools built around it. Batch jobs are usually steps in a data pipeline. Because the input is fixed, running the same job twice on the same data gives the same result, which makes testing and reruns straightforward.

Think of a bakery that counts the day's sales after closing. The count is complete and exact, but you learn about a slow morning only in the evening. The analogy holds for batch and breaks for streaming, as the next section shows.

Streaming in plain words

Unbounded data is a flow with no known end, such as clicks on a website. A streaming job never finishes: it keeps a result up to date as events arrive. Spark's guide models the input as an unbounded table that grows as data arrives, and a query over it is incremental.

Compare a till display that shows a running total. The analogy breaks in one place: the total is never final, because a late receipt can still arrive. Event time is the time an event happened, which can differ from the time the system processes it, and that gap is why late data is its own topic. The same guide has a section called "Handling Late Data and Watermarking".

Micro-batch: the middle ground

Micro-batch is where "streaming is not batch" stops being true. By default, Spark processes streaming queries with a micro-batch engine: a stream becomes a series of small batch jobs, with end-to-end latency as low as about 100 milliseconds and exactly-once fault tolerance. Exactly-once is a delivery promise that each input event affects the result one time, even if a failure forces a retry.

The guide says this is achieved through checkpointing and write-ahead logs. It makes the end-to-end promise conditional: it needs replayable sources and idempotent sinks, and its sink table lists the Kafka sink as at-least-once. In practice this means a team can reuse batch skills, such as partitioning and retries, for a stream, while accepting a delay of a fraction of a second or more between an event and its result.

Continuous Processing: faster, with limits

Continuous Processing is a low-latency Spark mode, introduced in Spark 2.3, with latency as low as about 1 millisecond but only at-least-once semantics. At-least-once is a weaker promise that each event is counted one or more times, so a retry can produce duplicates. The guide marks Continuous Processing as experimental and supports only map-like queries, such as projections and selections, with no aggregations. Faster costs you the stronger promise and much of the query power. Streaming and batch are a spectrum, and these figures are best cases, not typical ones. A candidate who quotes the 1 millisecond figure should add these limits in the same breath.

Kafka, per-event engines, watermarks and Lambda vs Kappa

Apache Kafka is a distributed event streaming platform that stores streams of events durably, so a consumer can read them again from an earlier position. That replayable log is what lets a team reprocess history after fixing a bug. Apache Flink is a stream processor that handles each event as it arrives, keeping state and taking checkpoints, which is the per-event alternative to micro-batch.

A watermark is a marker of how far event time has progressed, used to decide when a window of events is complete enough to emit and when a later event counts as late. Lambda architecture is a design that runs a batch layer and a streaming layer side by side and merges their results, at the price of keeping two code paths. Kappa architecture is a design with one streaming path that reprocesses history by replaying the log.

How they differ

Batch processes a bounded set of data on a schedule. Streaming processes an unbounded flow of events continuously. A batch result is as fresh as the data cutoff of the last completed run, so it ages until the next run finishes, while a streaming result stays close to current.

Replay matters when a bug is found. A batch job reruns over its stored input, while a streaming job can reprocess history only if its source keeps the events, as a Kafka log does.

A streaming job keeps a cluster running all day, while a batch job uses it only during the run. Late data in batch is handled by rerunning the job, while streaming needs event-time handling built into the job.

Text description of the diagram

• Batch processes a bounded set of data on a schedule.
• Streaming processes an unbounded flow of events continuously.
• A streaming job keeps a cluster running all day, while a batch job uses it only during the run.

Worked example

The numbers below are illustrative, not measurements. Data age is the time from an event arriving to its result being available. Events arrive at random through the day, and each design runs on four worker nodes.

A nightly batch job that runs for 60 minutes after each 24-hour window gives an average data age of 780 minutes and a worst case of 1,500 minutes.¹ An hourly batch job that runs for 6 minutes gives an average of 36 minutes and a worst case of 66 minutes.¹ A streaming job with a 60-second trigger and 5 seconds of processing gives an average of 35 seconds and a worst case of 65 seconds.¹

The node-hour column follows from the run time: a batch job holds four nodes only while it runs, while a streaming job holds them all day.¹ Freshness is bought with compute. If a result may be 36 minutes old, hourly batch may be enough.

When each wins

Take two illustrative cases. A daily revenue report read each morning may be hours old, so a scheduled batch job fits. A fraud alert that must reach a person within seconds cannot wait for the next run, so it needs streaming.

A result that may be hours old suits a scheduled batch job. A result needed in seconds to minutes suits micro-batch streaming. A result needed in milliseconds suits a record-at-a-time engine such as Flink, or Spark Continuous mode, which is experimental and at-least-once.

Text description of the diagram

• A result that may be hours old suits a scheduled batch job.
• A result needed in seconds to minutes suits micro-batch streaming.
• A result needed in milliseconds suits a record-at-a-time engine such as Flink, or Spark Continuous mode, which is experimental and at-least-once.

Trade-offs and edge cases

Streaming trades simplicity and cost for freshness. Batch is easier to rerun when something is wrong, because the input is a fixed set. Streaming must keep state across events, survive failures and cope with data that arrives late.

Output is a second choice. In the same guide, Complete Mode writes the entire updated result table, Append Mode writes only rows added since the last trigger, and Update Mode writes only rows that changed. The guide also describes using the same DataFrame operations (a DataFrame is a table of rows with named columns) in both batch and streaming, so the code can often be shared when the logic is the same.

Edge cases: a stream joined to static reference data needs the static side kept current; events with old event times can arrive after a result was emitted; and a design that needs a stronger delivery promise may have to accept higher latency. For modeling the tables these jobs write, see data modeling interview questions, and for the Spark application programming interface (API) itself see the PySpark tutorial, and for common Spark questions see the Spark interview questions.

How to answer it in an interview

The spoken structure: freshness first, then cost, then failure modes, then the spectrum.

Common mistakes

• "Streaming is always millisecond latency." (wrong) Spark's default is about 100 milliseconds at best, and its 1 millisecond mode drops to at-least-once. (right) "Streaming latency ranges from milliseconds to minutes depending on the mode."
• "Streaming and batch are different worlds." (wrong) Spark's default engine runs small batch jobs. (right) "Micro-batch sits between them."
• "Streaming cannot be exactly-once." (wrong) Spark documents end-to-end exactly-once in micro-batch mode, but only under conditions. (right) "Micro-batch can give end-to-end exactly-once when the source can be replayed and the sink is idempotent or transactional; continuous mode is at-least-once."
• "Always pick streaming, it is more modern." (wrong) It costs more compute and operational care. (right) "I pick the slowest design that meets the freshness need."

Follow-up questions

What are the batch vs streaming architecture tradeoffs?

Batch gives simple reruns and low compute cost but stale results. Streaming gives fresh results at the price of an always-on cluster, state and late-data handling. Lambda architecture keeps both a batch and a streaming path, while Kappa architecture keeps only streaming and reprocesses by replaying the log. In the worked example, node-hours rose from 4 to 96 as average data age fell from 780 minutes to 35 seconds.¹

How do batch, streaming and real-time inference differ?

In general, batch inference scores a stored set of records on a schedule, and streaming scores events as they flow through a pipeline. Real-time inference answers one request when a caller sends it, so it is a request-response service rather than a pipeline mode. The choice again follows how fresh the prediction must be, for machine learning (ML) as for any other result.

Is micro-batch streaming or batch?

Micro-batch processing is both: a streaming technique built from small batch jobs, which is how Spark's default engine works.

Can streaming be exactly-once?

Streaming can be exactly-once, conditionally: Spark's micro-batch engine gives end-to-end exactly-once with a replayable source (such as Kafka) and an idempotent sink; writing to Kafka itself is at-least-once.

When is a nightly batch job still the right choice?

When the result is read once a day, nobody acts on fresher data, and the cheaper cluster time matters more than freshness. The data engineering guide lists neighboring pipeline questions.

Sources

1. Structured Streaming Programming Guide (Spark 3.5.6) — Apache Spark project documentation, 29 May 2025
2. Apache Kafka documentation: Introduction — Apache Kafka project documentation, 18 Mar 2025
3. Apache Flink documentation: Stateful Stream Processing — Apache Flink project documentation, 24 Mar 2025