In this article
Before you start
This page is one question inside the machine learning (ML) system design interview path. You need two ideas.
Why interviewers ask this
An answer that works on a whiteboard for a small load can fail for reasons the small load never shows, and this question tests whether you know which reasons. Interviewers can use it to separate a candidate who lists components from one who foresees where a design will give way.
A weak answer says "the database becomes a bottleneck" and stops. A strong answer names a mechanism, such as fan-out, resource exhaustion, retries or overload, and says how to detect and contain it. A Staff-level answer adds arithmetic the candidate can do aloud and the design change that follows from it. The question also arrives in other wordings, covered under the variants below, so the mechanism matters more than any single example.
What breaks at scale from first principles
- Cascading failure
- A cascading failure is a failure that grows over time through positive feedback: one part of a system fails, which raises the load or failure chance of the other parts, which makes more of them fail in turn.
In plain words
Picture a group dinner where the table is served only when every dish is ready. Each dish is late one time in a hundred, which seems rare. With a hundred dishes, the table almost always waits for something. The analogy breaks in one place: servers also affect each other, because a slow dish blocks the counter that the next dish needs.
In one precise sentence, what breaks at scale is the assumption that rare problems in one component stay rare for a request, and the assumption that components fail independently. Each step below removes one of those assumptions.
Step 1: Fan-out turns a rare slow server into a common slow request
A request that waits for many servers is slow if any one of them is slow. Dean and Barroso (2013) write that even rare performance hiccups affect a significant fraction of all requests in large-scale distributed systems.
Input: the share of slow responses per server, and the fan-out. Output: the share of slow requests, which is one minus the chance that every server is fast. With a slow rate of one in a hundred per server, the share is 1.0% for one server, 9.6% for ten, 63.4% for a hundred, and five hundred servers make 99.3% of requests slow.1 A hundred servers matches the 63% that Colyer's summary quotes from the paper.
Even a rate of one in 10,000 makes 18.1% of requests slow across 2,000 servers.1 To keep a hundred-server request slow at most one time in a hundred, each server must be fast about 99.99% of the time.1 This arithmetic assumes servers are slow independently; if slowness is correlated, for example from a shared network fault or a shared overloaded dependency, the real share differs from the formula.
Text description of the diagram
- With a slow rate of one in a hundred per server, 63.4% of requests across a hundred servers are slow.
- Five hundred servers make 99.3% of requests slow.
Step 2: Slow requests hold resources until a server runs out
Slow requests hold threads and connections until a server runs out of them. Brooker (2019) puts it this way: when a number of requests hold on to resources for a long time, the server can run out of those resources, such as memory, threads or connections.
Input: requests that wait on a slow dependency. Output: a server that cannot accept new work. A server whose whole thread pool is waiting on one slow dependency has an idle processor and still turns requests away. A timeout is the maximum time a client waits for a request to complete, and without one the client holds its own resources too, so the slowness moves upstream.
A partial failure, where only a percentage of requests succeed, makes this worse: averages look acceptable while a growing queue builds behind them.
Step 3: Retries add load to a system that is already struggling
Retries mask transient failures but add load, and each layer that retries multiplies the calls that reach the bottom of the stack. Brooker also warns that it is not always safe to retry.
Input: one failing call and a retry rule. Output: more calls than the system received. With two retries per layer, each call makes up to three attempts, so the worst case at the bottom is 3 calls for one layer, 9 for two, 27 for three and 81 for four (illustrative).1
Idempotency is the property that repeating a call has the same effect as making it once. Calls with side effects, such as charging a card, are unsafe to retry unless they are idempotent. Backoff spaces retries out: exponential backoff doubles the wait after each failure. Jitter is random variation added to each retry delay so that clients that failed together do not retry together.
Step 4: One overloaded replica pushes its load onto the rest
One overloaded replica pushes its load onto the survivors. The Google book on cascading failures defines the pattern as "a failure that grows over time as a result of positive feedback," and gives the example of one replica failing from overload, which raises the load on the rest.
Input: replicas sharing a fixed load. Output: a domino effect. Take five replicas, each at 70% of capacity (illustrative). When one fails, each of the four survivors carries 87.5%. When a second fails, each of the three left would need 116.7%, which is more than they can serve, so they fail too.1 The cause is the loss of headroom, not simultaneous faults. Containing it means spare capacity, shedding excess requests early, and degrading answers instead of refusing them.
Step-by-step description
- Rare slow servers make slow requests common.
- Slow requests hold threads and connections.
- Retries multiply the load.
- Survivors overload and fail too.
- A request that waits for many servers is slow if any one of them is slow.
- Slow requests hold threads and connections until a server runs out of them.
- Retries multiply the calls that reach the bottom of the stack.
- One overloaded replica pushes its load onto the survivors.
Why can't the slow requests simply be removed?
Slow requests cannot be removed because they come from ordinary variability in shared environments. Dean and Barroso state that eliminating all sources of latency variability in large-scale systems is impractical, especially in shared environments. Background work, queueing and contention for shared hardware all produce occasional slow responses.
Their Google example shows the stake: the 99th-percentile latency for all requests to finish is 140 milliseconds, while the figure for the first 95% of the sub-requests finishing is 70 milliseconds, so waiting for the slowest 5% is responsible for half of the total.2 The remedy is to design for tolerance, because techniques that target the slow outliers can cut the overall 99th-percentile latency sharply (Dean and Barroso, 2013).
Why does one hot key or shard break before the rest?
Skew breaks a partitioned system because load is rarely uniform, so the busiest partition saturates while the average across partitions still looks healthy. A hot key is a single key, such as a celebrity's profile or a trending query, that draws a large share of traffic, and every request for it lands on the one shard that owns it. Adding shards does not help, because the hot key still lives on only one of them. A related failure is a cache stampede: a popular cached entry expires, many requests miss together and all hit the database to rebuild it. A store with a single writer or per-node state has the same shape, because one node limits the whole system.
The controls are splitting or salting a hot key across several shards, coalescing identical in-flight requests into one, keeping extra replicas of hot data, and spreading expiry times with jitter. Watch per-shard load, not the average.
Where do machine learning systems break differently?
Machine learning systems add failures that appear over weeks, not milliseconds. Sculley et al. (2015) argue that developing and deploying such systems is relatively fast and cheap, but maintaining them over time is difficult and expensive. They list boundary erosion, entanglement, hidden feedback loops, undeclared consumers, data dependencies, configuration issues and changes in the external world.
The serving mechanisms above apply directly: a retrieval stage that queries many index shards is fan-out, as in the multi-stage retrieval funnel, and a scoring path with a hard latency budget, such as real-time fraud detection, feels every slow outlier on every call. Training-serving skew is another quiet failure: a feature computed one way when the model was trained and another way in production makes predictions worse without any error appearing.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →The Staff-level answer
What breaks at scale?
Four things break, in the order a request meets them. First, the tail: if one request waits on a hundred servers that are each slow one time in a hundred, about two thirds of requests are slow, so I budget against the fan-out's tail, not the average. Second, slow requests hold threads and connections, so I set timeouts and bound queues. Third, retries multiply load; two retries at each of four layers can mean eighty-one calls at the bottom, so I retry at one layer with backoff, jitter and idempotent calls. Fourth, overload cascades, so I keep headroom and shed load early. In a machine learning system I add feedback loops and data dependencies, which break quietly.
What makes it Staff-level: each failure comes with arithmetic you can do aloud and the control that contains it, in the order the request meets them.
The rubric
| Dimension | Weak answer | Strong answer | Staff answer |
|---|---|---|---|
| Correctness | Says "the database is the bottleneck" with no mechanism. | Names fan-out, resource exhaustion, retries or cascades as mechanisms. | Orders the mechanisms by where a request meets them. |
| Depth | Lists components. | Explains why a rare slow response becomes common. | Computes the slow share for a stated fan-out aloud. |
| Trade-offs | Says "add more servers". | Notes retries help transient faults but add load. | Weighs headroom cost against cascade risk. |
| Communication | Rambles through a list. | Gives one cause per sentence. | States the cause, the number, the control. |
| Follow-up readiness | Stuck on "why". | Handles "what if one dependency is slow". | Extends to machine learning feedback loops. |
Three variants of this question
What are the bottlenecks when traffic grows by a factor of ten?
Name the resource each request holds, such as threads, connections or memory, and find which runs out first under load. Then add the fan-out view: more traffic often means more shards per request, which raises the slow share before any single resource is full.
How do you find what will fail before launch?
Load a staging copy until it saturates and watch tail latency, not the average. Remove a replica under load to see whether the survivors cope, and slow one dependency to see whether timeouts and queue bounds hold.
What happens when one dependency is slow rather than down?
A slow dependency is often worse than a dead one, because a dead one fails fast while a slow one holds threads and connections until they run out. Timeouts, bounded queues and fast failure turn the slow case into the dead case, which the system already handles.
Common mistakes
- "The tail latency of each service is fine, so the user's is fine." (wrong) Fan-out multiplies the tail. (right) "I compute the slow share across the fan-out, not per service."
- "Slow requests mean something is broken." (wrong) They come from ordinary variability that cannot be fully removed. (right) "I design to tolerate the slow few instead of hunting every cause."
- "Adding retries makes it reliable." (wrong) Retries add load and are unsafe for non-idempotent calls. (right) "I retry at one layer, with backoff, jitter and idempotent calls."
- "A cascade happens because many things fail at once." (wrong) One failure raises load on the survivors, which is positive feedback. (right) "I keep headroom so that losing one replica does not overload the rest."
- "Once the model ships, scaling is just infrastructure." (wrong) Maintaining machine learning systems over time is difficult and expensive (Sculley et al., 2015), and scaling does not remove that. (right) "I plan for feedback loops and data dependencies from the start."
Follow-up questions
How do you reduce tail latency in a fan-out system?
Send hedged requests: if the first replica has not answered by the 95th-percentile expected latency, send a second copy to another replica and use whichever answer arrives first. In a Google benchmark reading 1,000 keys spread over 100 servers, hedging after 10 milliseconds cut the 99.9th-percentile latency from 1,800 milliseconds to 74 milliseconds while sending about 2% more requests (Dean and Barroso, 2013). Also touch fewer servers per request where the design allows, set timeouts so slow answers do not hold resources, and return good-enough partial results when a few servers are late. Targeting the slow outliers can bring dramatic reductions in tail latency. Autocomplete, where each keystroke fans out, shows the pressure in typeahead system design.
When is it safe to retry a failed request?
A retry is safe when the call is idempotent, meaning repeating it has the same effect as making it once. Even then, retry at one layer with backoff and jitter so the retries do not become the next overload.
How much spare capacity do you need to survive one failure?
With five replicas sharing the load, each must run at or below 80% of capacity so that four survivors can carry the traffic after one failure.1 In the worked case above, 70% is within that limit for one failure but not for two.
What breaks first in a machine learning serving path?
Usually the tail, because serving chains retrieval, features and scoring, and every stage can fan out. Prediction tasks with tight budgets, such as estimated-time-of-arrival prediction and dynamic surge pricing, show why the tail is budgeted explicitly. Slower-moving failures, such as feedback loops, follow later.
- Rare slow responses become common once a request fans out to many servers.
- Slow requests hold threads and connections until a server runs out of them.
- Retries multiply load across layers, so retry once, with backoff, jitter and idempotent calls.
- Overload cascades through positive feedback, so keep headroom and shed load early.
- Machine learning systems add feedback loops and data dependencies that fail quietly.
Sources
- The Tail at Scale — Communications of the ACM (Dean and Barroso); accessed 2026-10-26, 1 Feb 2013
- The Tail at Scale (summary quoting the paper) — the morning paper (Adrian Colyer); accessed 2026-10-26, 15 Jan 2015
- Addressing Cascading Failures, Site Reliability Engineering chapter 22 — Google, O'Reilly; accessed 2026-10-26, 1 Apr 2016
- Timeouts, retries, and backoff with jitter — Amazon Builders' Library (Marc Brooker); accessed 2026-10-26, 5 Dec 2019
- Hidden Technical Debt in Machine Learning Systems — NeurIPS 2015 (Sculley et al.); accessed 2026-10-26, 1 Dec 2015
