In this article
Before you start
This answer sits inside the machine learning (ML) system design interview round, where a latency limit is added after you present a first design.
Why interviewers ask this
A limit changes the design, not just the tuning, so "add a cache" is where a weak answer stops. Candidates report this question in senior ML design rounds, usually after a recommender or ranker has been drawn on the board.
It separates answers that treat speed as one number from answers that treat it as a budget. A weak answer promises to "make everything faster" and names no stage. A strong answer says which percentile the limit applies to, divides the time among the stages, and names what the system drops when it runs late.
A Staff-level answer adds the failure path: what the user sees when one dependency is slow, and what you measure to know. The limit also forces trade-offs, such as fewer candidates, a lighter model or an older cached result, and a good answer says which one you pick and why.
A latency budget from first principles
- Budget split
- A budget split is the allocation of a request's total time limit across its stages, such as network, feature fetch, candidate generation and scoring, with a margin left unspent, so every stage has a number and a defined fallback when it overruns.
- Stage fallback
- A stage fallback is the cheaper result that a stage returns when it overruns its slice of the latency budget, such as a cached list or a skipped optional step, so the whole request still meets its limit instead of failing.
In plain words
Picture a train journey with three connections. The railway promises you arrive within a set time. Each leg gets a slot in the timetable. One late connection makes the whole journey late.
A request works the same way. The limit covers the whole trip, every stage takes a slice, and the slowest part decides the result. The analogy breaks in one place: a train cannot skip a leg, but a recommender can drop an optional stage, or return a cached list, when time runs short.
In one precise sentence, a limit such as the illustrative 100 milliseconds is a promise about the percentile of whole-request time, and you meet it by budgeting stages and controlling the slowest call.
Step 1: Set the target as a percentile of the whole request
The first change is to say what the number means. A tenth of a second is about the limit at which a user feels a system reacts immediately, according to Nielsen Norman Group (excerpt of a 1993 book).2 The same source puts the limit for an uninterrupted flow of thought at one second. These are perceptual rules of thumb, not engineering contracts.
An engineering target needs a percentile. The 99th percentile, called p99, is the response time that 99 of every 100 requests beat. A mean of 60 milliseconds can hide a p99 of 300 (illustrative). Ask also where the clock runs. If it runs at the user's screen, network and rendering time come out of the same 100 milliseconds (illustrative budget); that is a design inference, not a sourced rule. You now have a target: a stated percentile, measured at a stated place.
Step 2: Fan-out makes the slowest backend set the latency
Fan-out turns rare slow calls into common slow requests. If a request calls N backends concurrently and waits for all, the chance that at least one is slow is 1 minus (1 minus p) to the power N, where p is the chance a single call is slow.
With p at 1%, that chance is 1% for one backend, 9.6% for 10, 39.5% for 50 and 63.4% for 100.1 At 300 backends it is 95.1%.1 Run backwards, for 99 of 100 requests to stay fast across 100 backends, each backend must itself be fast for about 99.99% of calls.1
This is the main idea of The Tail at Scale by Dean and Barroso (2013). Its remedies include the hedged request. A hedged request is a duplicate sent after a short delay, with whichever answer returns first used; it trades a little extra load for a shorter tail, as an InfoQ article explains. Fewer shards, per-call timeouts and fallbacks also shrink the tail.
Text description of the diagram
- With a 1% slow chance per backend, 63.4% of requests meet a slow one at 100 backends.
- At 300 backends the share is 95.1%.
Step 3: Split the budget across stages
A budget split gives each stage a slice and keeps a margin. Recommenders are staged, so the split follows the funnel. Google's course describes three stages: candidate generation, scoring and re-ranking. Candidate generation is the first stage, which cuts a huge catalogue down to hundreds or thousands of items. Scoring then ranks that subset down to about 10 items, and because the subset is small it can use a more precise model. Re-ranking applies extra rules such as freshness and diversity.
| Stage | Assumed time (milliseconds) | What it covers |
|---|---|---|
| Network and gateway | 10 | connection, authentication, routing |
| Feature fetch | 15 | user and item features |
| Candidate generation | 20 | search over the catalogue |
| Scoring | 30 | the heavier model on the short list |
| Re-rank and response | 10 | rules and serialising |
| Margin | 15 | unspent, for variance |
These numbers are illustrative, not a measured system. Their value is the habit: every stage has a number, the numbers add up, and a stage that overruns has a defined fallback. The search ranking design page applies the same staging to search.
Text description of the diagram
- Scoring gets 30 milliseconds, the largest slice.
- 15 milliseconds stay unspent as a margin.
- The six slices add up to 100 milliseconds.
Step 4: Count batching delay as spent time
Dynamic batching is a serving feature that groups requests arriving close together into one model call, which raises throughput but makes early requests wait. The Triton batcher documentation says the batcher does not delay when forming batches by default. A delay is set with max_queue_delay_microseconds, and the docs advise raising it only while the latency budget still holds.
Every millisecond of waiting comes out of the same limit. A window of 2 milliseconds costs 2% of a 100 millisecond budget and a window of 5 milliseconds costs 5%.1 At an illustrative 2,000 requests per second, those windows collect about 4 and 10 requests per batch.1 If the 5 millisecond window is charged to scoring, that stage keeps 25 of its 30 milliseconds.1 You now have the four decisions: the percentile, the tail, the split and the wait.
Step-by-step description
- The target is a percentile of the whole request.
- The slowest backend sets the latency.
- Each stage gets a slice of the budget.
- Batching delay is spent time.
- The target is a percentile of the whole request.
- Fan-out lets the slowest backend set the latency.
- Each stage gets a slice of the budget.
- Batching delay is spent from the same budget.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →The Staff-level answer
What changes if this has to return in 100 milliseconds?
First I pin down what the 100 milliseconds measures: the 99th percentile of the whole request, and where the clock stops. Then I split it across stages, say network, feature fetch, candidate generation, scoring and re-ranking, and I keep a margin. Because retrieval fans out to many shards, the slowest shard sets the latency, so I add per-stage timeouts, hedged requests and a fallback such as cached or popular items. Any batching delay is charged to the same budget. If the target still fails, I cut candidates or use a lighter model before I touch the percentile.
What makes it Staff-level is that it names the percentile, the fan-out tail and the fallback, and says what you would give up first.
The rubric
| Dimension | Weak answer | Strong answer | Staff answer |
|---|---|---|---|
| Correctness | Says "make it faster" and adds a cache. | Splits the time across named stages. | Adds the percentile, where the clock runs and a margin. |
| Depth | Averages only. | Mentions the slow tail. | Explains fan-out and how hedging cuts the tail. |
| Trade-offs | Names none. | Trades model size for speed. | States what is dropped first and what quality it costs. |
| Communication | Lists techniques. | Walks the request in order. | Gives the budget as numbers that add up. |
| Follow-up readiness | Stalls on "what if it is slower?" | Names timeouts. | Names fallbacks, load tests and the metric to alert on. |
Three variants of this question
What changes if the limit is 30 milliseconds on a vehicle graphics processing unit (GPU)?
A tighter limit removes whole stages. A multi-model pipeline on one graphics processing unit (GPU) needs a fixed time slice per model, models of a size that fits, and no waiting for batches. Candidates report this variant for perception systems; the same budget logic applies with smaller slices.
What if the service must handle 5,000 concurrent requests without losing any?
Concurrency equals request rate times latency, so 5,000 concurrent requests at 100 milliseconds each is 50,000 requests per second.1 To lose none, use a bounded queue, reject early with a clear error when full, and retry with a limit rather than dropping silently. Losing nothing is a statement about accounting, not about speed.
How would you cut a p95 from seconds to tens of milliseconds?
Measure first. Trace one slow request, find the stage with the largest share, and fix that stage before any other. Common fixes are caching or precomputing results, removing a serial call, shrinking the candidate set, and using a lighter model on the hot path. Report the percentile before and after.
Common mistakes
- "I will get the average under 100 milliseconds." (wrong) The tail is what breaks the limit. (right) "I target the 99th percentile of the whole request and alert on it."
- "I will add replicas and a bigger cache." (wrong) More copies do not remove a slow straggler behind a fan-out. (right) "I cap the straggler with timeouts, hedged requests and a fallback."
- "Dynamic batching always makes serving faster." (wrong) It raises throughput and can add waiting time. (right) "I set the batch delay from what the budget allows, a few milliseconds."
- "One big model in one step is simplest." (wrong) A heavy model over the whole catalogue cannot fit the limit. (right) "I use a cheap first stage and a precise model on a short list."
Follow-up questions
Why is the 99th percentile more important than the average for a 100 millisecond limit?
The average hides the slowest requests, and those are the ones users notice. With fan-out, the slowest backend decides each request, so a rare slow call becomes a common slow request, as the numbers above show.
What is a hedged request and when do you use one?
A hedged request sends a duplicate after a short wait and takes the first reply. Use it for calls that are safe to repeat and where a small rise in load is acceptable, as in the Tail at Scale technique.
Does dynamic batching make serving faster?
It raises throughput, but each request may wait for the batch to fill. Keep the wait small and charge it to the budget, as the Triton documentation advises.
How do you stop one slow dependency from breaking the budget?
Give every call a timeout shorter than its stage slice and a fallback result. Pass the remaining time down the call chain so later stages do not start work that cannot finish. The real-time fraud detection and estimated time of arrival (ETA) prediction designs show where fallbacks sit.
Does the 100 millisecond limit include the network and the browser?
It depends on where the clock runs, so ask. If it is measured at the user's screen, network and rendering time come out of the same limit. For a very low limit on a vehicle or a robot, see the computer vision system design questions; for a tight limit on every keystroke, see typeahead design.
- State the limit as a percentile of the whole request, and say where the clock runs.
- Fan-out lets the slowest backend set the latency, so the tail matters more than the mean.
- Split the time across stages in numbers that add up, and keep a margin.
- Charge any batching delay to the same budget.
- Say what you drop first, and what the fallback returns, when time runs out.
Sources
- Response Times: The 3 Important Limits — Nielsen Norman Group (Jakob Nielsen, excerpt of Usability Engineering, 1993); accessed 2026-10-26, 1 Jan 1993
- Recommendation systems overview: types of systems — Google for Developers, Machine Learning course; undated page, accessed 2026-10-26, 26 Oct 2026
- Batchers (Dynamic Batcher), Triton Inference Server user guide — NVIDIA; current documentation, accessed 2026-10-26, 26 Oct 2026
- The Tail at Scale — Communications of the ACM 56(2) (Dean and Barroso), 1 Feb 2013
- Adaptive hedged requests and p99 latency — InfoQ; undated, accessed 2026-10-26, 26 Oct 2026
