In this article
Before you start
This page is one step on the machine learning operations (MLOps) learning path.
Why interviewers ask this
The choice of serving mode is the first design decision after a model works, and it fixes the cost, the freshness and the failure behaviour of everything built on top. Interviewers ask it to see whether you start from the requirement, whether the input is known ahead and how fresh the answer must be, or from a favourite tool.
A weak answer says "real-time is better because it is faster." A strong answer asks who consumes the prediction, whether the input is known in advance, how stale it may be, how spiky the traffic is and what a late answer costs, and then says why the cheapest mode that meets those needs wins. Expect a follow-up that adds a constraint, such as a voice assistant that must answer immediately.
What each one is
- Streaming inference
- Streaming inference is a way of serving a model in which each event arriving on a stream is scored as it arrives and the score is written to a store or another stream, so predictions trail the event and no caller waits for them.
- Real-time inference
- Real-time inference is a way of serving a model in which an always-running endpoint computes a prediction for each request while the caller waits, so the answer reflects the request itself.
- Staleness
- Staleness is the age of a prediction when it is used, counted from the time of the data it was computed on.
Batch inference in plain words
Batch inference is a bakery that bakes overnight and sells in the morning. Everything is made in one scheduled run and stored; customers pick from the shelf. The analogy breaks in one place: a shelf can only hold what was baked, so a request for something new gets nothing.
The same limit applies to batch predictions: Chip Huyen notes that they work when the set of possible inputs is finite, such as users who already exist (Huyen, December 2020). Google's documentation describes the pattern as asynchronous: you send a job, no endpoint is needed, and it suits cases where no immediate response is required and you want to process accumulated data in a single request (Vertex AI documentation, as of October 2026).
Streaming inference in plain words
Streaming inference is a conveyor belt. Items arrive continuously, a worker scores each one as it passes and puts the result on an outgoing belt. Nobody stands at the belt waiting for one item. The analogy breaks where the belt backs up: if items arrive faster than the worker scores them, lag grows.
A typical job reads each event from a topic, scores it with the model and writes the score to a store that other systems read. Apache Beam's documentation says its RunInference application programming interface (API) supports batch and streaming pipelines (Apache Beam documentation, page updated October 2026). Chip Huyen's 2020 post names only batch and online prediction and treats streams as the way fresh features are computed, which the trade-offs section covers.
The word is overloaded. In this page it means scoring events from a stream. In large language model (LLM) products it also means sending the output to the user token by token; that is a delivery method for a real-time answer, not a separate serving mode.
Real-time inference in plain words
Real-time inference is a restaurant kitchen: each order is cooked when it arrives and the customer waits at the table. The kitchen must be staffed for the busiest hour, even if it is quiet at three in the afternoon, and a table that arrives during a rush waits longer or leaves. Chip Huyen's description is that an online prediction responds within milliseconds to seconds and can use the current request, as in fraud detection. Amazon's documentation says real-time inference is ideal for interactive workloads with low latency requirements (Amazon SageMaker documentation, as of October 2026). That is why the endpoint stays running and why its capacity is judged against the worst moment of the day, not the average one.
Why always-on real-time costs the most
An always-on endpoint is sized for peak traffic, but it is paid for around the clock, so most of its capacity sits idle. If peak traffic is five times the daily average and you keep about thirty per cent spare capacity, average utilisation falls to roughly one instance in seven. Batch is sized for total volume and runs only as long as the job takes, so almost nothing sits idle.1
Why a stream sized for average falls behind
Streaming sits in between, and the rule is easy to get wrong. A queue lets you size a consumer below peak only if your lag budget can hold the backlog: lag is roughly the backlog divided by the consumer's rate. Size for twice the average and, during a peak five times the average, the backlog grows every second; lag passes two seconds after 1.5 seconds of peak, reaches about 79 seconds after a one-minute peak and about 79 minutes after a one-hour peak.1
If you need seconds of lag at peak, size the consumer for peak and it costs as much as the real-time fleet. Autoscaling is the standard mitigation: ideally it follows the load, but minimum instances, scale-up delay and cold starts keep the real figure between the ideal and the fixed fleet.1 The price of the cheaper modes is staleness: batch results can be a day old and a lag-tolerant stream falls behind at peak. The table compares them.
| Dimension | Batch | Streaming | Real-time |
|---|---|---|---|
| Trigger | A schedule | Each event | Each request |
| Freshness | Up to a day old | Seconds off-peak; minutes behind at peak if sized below peak | Computed on request |
| Capacity sized for | Total volume | Average rate if lag may grow; peak rate if lag must stay seconds | Peak rate, or autoscaled |
| Failure shows as | A late job | Growing lag | Errors for callers |
Text description of the diagram
- The batch run in the example takes about 1.9 hours; its predictions then age until the next run.
- A stream sized for twice the average falls about 79 minutes behind after a one-hour peak and needs about 2.4 hours to drain.
- Real-time answers are computed on each request, so they have no age.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
Take one model that must score 10 million predictions a day. All numbers here are illustrative assumptions, and the script that computes them is saved with the page.1 The average rate is 10 million ÷ 86,400 seconds = 115.7 requests per second. Assume peak is five times that, 578.7 per second (illustrative), one instance serves 50 requests per second within a 200-millisecond latency target, and the peak lasts one hour (illustrative).
A fixed real-time endpoint with 30 per cent headroom needs ceil(578.7 × 1.3 ÷ 50) = 16 instances running all day: 384 instance-hours. With ideal autoscaling at the same headroom, instance-hours follow the load: 10 million ÷ 50 ÷ 3,600 × 1.3 = 72.2, about 9.5 times batch. An instance-hour is one instance running for one hour, the unit most clouds bill by. At peak, requests in flight equal rate times latency: 578.7 × 0.2 = 115.7 requests being handled at once.1
A streaming consumer sized for twice the average needs 5 instances, 120 instance-hours, at 46.3 per cent average utilisation. It keeps about 2 seconds of lag off-peak, but its capacity is 250 per second against a 578.7 peak, so the backlog grows by 328.7 events per second: lag passes 2 seconds after 1.5 seconds of peak, and a one-hour peak leaves 1.18 million events, about 79 minutes of lag, which takes about 2.4 hours to drain at average load.1
Sized for peak instead, the stream needs the same 16 instances and 384 instance-hours as real-time. A nightly batch that scores 400 predictions per second per instance, which is an assumed eightfold gain from batching, needs 6.94 compute hours plus start-up time for four workers: 7.61 instance-hours, finished in about 1.9 hours.1
| Quantity | Batch | Streaming (2x average) | Real-time (fixed) | Real-time (ideal autoscaling) |
|---|---|---|---|---|
| Instances | 4 workers, about 1.9 hours | 5 | 16 | follows load |
| Instance-hours per day | 7.61 | 120 | 384 | 72.2 |
| Average utilisation | not applicable | 46.3% | 14.5% | about 77% (1 ÷ 1.3) |
| Worst staleness | up to 24 hours | about 2 seconds off-peak; about 79 minutes after a one-hour peak | none | none |
The 50-fold gap depends on the assumed eightfold batch speed-up and on a fixed fleet; the autoscaled figure is an ideal, so minimum instances, scale-up delay and cold starts put the real ratio between 9.5 and 50. If batching gave no gain, the same job would need 56.2 instance-hours and real-time would cost 6.8 times as much.1 State the assumption you rely on whenever you quote a ratio.
Text description of the diagram
- Batch needs about 7.61 instance-hours a day in the illustrative sizing.
- A fixed real-time fleet needs 384 instance-hours a day, about fifty times batch.
- Ideal autoscaling needs 72.2, about 9.5 times batch.
Trade-offs and edge cases
The modes combine more often than they compete. Uber's 2017 platform post describes streaming feature computation, near-real-time features such as a one-hour average meal prep time built through Kafka and Samza, next to batch precomputation for slower features such as a seven-day average. Online models read both kinds of feature, with a P95 latency below 10 milliseconds for models that fetch features from its Cassandra store (Uber Engineering, September 2017). A recommendation system commonly precomputes candidates in batch and re-ranks them in real time.
Amazon's documentation lists other points on the spectrum. Serverless endpoints suit workloads with idle periods and tolerate cold starts. Asynchronous endpoints queue requests and suit large payloads, up to 1 GB, and processing times up to one hour, where results are needed soon but nobody holds a connection open (Amazon SageMaker documentation, as of October 2026).
Edge cases candidates miss:
- Cold start. A scale-to-zero endpoint is cheap and slow on the first request.
- Training-serving skew. If batch features and real-time features are computed by different code, the model sees different numbers in production than in training; a feature store, covered in the feature store design page that follows in this path, computes each feature once for both.
- Large models. For LLM serving the limit is often graphics processing unit (GPU) memory, not request rate; see the key-value (KV) cache and LLM inference pages.
- Monitoring. Each mode fails differently, so alerts differ: job lateness, stream lag, error rate. See LLM observability for the logging side.
Text description of the diagram
- If the input is known in advance and a day-old answer is fine, use batch and serve predictions by lookup.
- If the answer depends on the request itself while someone waits, use real-time.
- Otherwise score events from a stream.
How to answer it in an interview
When would you choose batch, streaming or real-time inference?
I start with two questions. Is the input known in advance and may the answer be a day old? Then I run batch and serve the stored predictions by lookup, even to a waiting user, and it is the cheapest. Does the answer depend on the request itself while someone waits? Then I use real-time, sized for peak or autoscaled. Otherwise I score events from a stream, and I size it by the lag I can tolerate: lag is backlog divided by consumer rate, so a consumer sized below peak falls behind during bursts. Often I combine them: precompute in batch, adjust in real time. I would quote the cost trade-off with my assumptions stated.
The spoken structure: name the two deciding questions, map each answer to a mode, mention that modes combine, then give the cost reason.
Common mistakes
- "Real-time is always better." (wrong) A fixed fleet costs the most and is idle off-peak. (right) "I use real-time only when the answer depends on the request itself."
- "A caller is waiting, so it must be real-time." (wrong) Precomputed batch predictions are routinely served to waiting callers by lookup. (right) "If the input is known ahead and a day-old answer is fine, I serve batch results by lookup."
- "Streaming means real-time." (wrong) A stream consumer has no caller waiting, and its lag grows whenever events arrive faster than it scores. (right) "Streaming scores events with a lag I must budget; real-time answers a waiting request."
- "Batch cannot work for new users." (wrong) It can, if you precompute a fallback and fill in real time when the user is new. (right) "Batch covers known inputs; real-time handles the rest."
- "I would size for the average load." (wrong) A queue only hides bursts while the lag budget can hold the backlog; lag is backlog divided by consumer rate. (right) "Real-time is sized for peak or autoscaled; a stream consumer is sized for the lag I can tolerate at peak."
Follow-up questions
What should be the difference between streaming and batching?
Batching groups many inputs and processes them together on a schedule; streaming processes each input as it arrives. Batch gives the best throughput per machine and the oldest results; streaming trades some efficiency for results that trail the event by seconds.
Batch vs real-time inference in machine learning?
Batch precomputes predictions and stores them, so a lookup is fast, including for a waiting caller, but the data is stale and only known inputs are covered; real-time computes on the request, so the answer is fresh but capacity must cover peak load. Google's documentation frames it as asynchronous jobs against synchronous requests (Vertex AI documentation, as of October 2026).
How do you ensure the voice response is immediate?
Treat it as real-time inference with a latency budget. Stream the audio in chunks, stream the response out token by token so playback starts early, keep the model warm, and place servers near users; every stage of the budget needs its own measurement.
When is batch inference not good enough?
When the input is not known in advance, when the right answer depends on what just happened, or when a wrong or late answer is costly. Huyen's example is fraud detection, where catching a transaction as it happens can prevent it (Huyen, December 2020).
How would you serve both streaming and batch uploads for a product?
Use one model package and one feature definition for both paths so they cannot drift. Send uploads that need an answer within seconds to a stream consumer, and send bulk back-fills to a scheduled job. For the surrounding system, the machine learning (ML) system design overview, classical machine learning hub and data engineering hub give the neighbouring pieces.
- Batch runs on a schedule, streaming on each event, real-time on each request.
- Start from whether the input is known ahead and how stale the answer may be, not from the tool.
- Real-time is sized for peak (or autoscaled) and idles off-peak; a stream is sized for the lag you can tolerate, and falls behind at peak if sized for average; batch for total volume.
- In the illustrative example a fixed real-time fleet cost about 50 times batch, about 9.5 times with ideal autoscaling, and 6.8 times if batching gave no gain.
- Production systems often combine precomputed batch features with real-time scoring.
Sources
- Deploy models for inference (Amazon SageMaker AI Developer Guide) — Amazon Web Services (date = read date), 24 Oct 2026
- Get online inferences (Vertex AI documentation) — Google Cloud, 22 Oct 2026
- Machine learning is going real-time — Chip Huyen, 27 Dec 2020
- About Beam ML (Apache Beam documentation, RunInference) — Apache Software Foundation (page last updated 2026-10-23), 23 Oct 2026
- Meet Michelangelo: Uber's Machine Learning Platform — Uber Engineering, 5 Sep 2017
