What's the largest system you've worked on, and how did you scale it?

In this article

1. Before you start
2. Why interviewers ask this
3. A scaling answer from first principles
4. The Staff-level answer
5. The rubric
6. Three variants of this question
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This question usually arrives inside a machine learning (ML) system design interview, but it also appears in plain software rounds.

Why interviewers ask this

The question is a cheap way to find out whether a candidate has operated something real. The answer shows scale, ownership, judgment and honesty at once, and the interviewer's next questions are chosen from what you say. A vague answer gets vague follow-ups; a specific one gets a conversation about your actual decisions.

A weak answer lists a stack ("Kafka, Redis, Kubernetes") and a large user count, then stops. A strong answer says how much load the system carried, which part broke first, what you changed, what that change cost and what happened to the numbers afterwards. A Staff-level answer also names what you would do differently and which risk you left in place on purpose.

The question does not reward the biggest system. It rewards the system you can explain at the level of one bottleneck and one trade-off. A modest service you understand completely beats a giant one you only touched at the edges.

A scaling answer from first principles

Trade-off

A trade-off is a cost that you accept in exchange for a gain when you choose one design over another, such as extra memory for faster reads, and naming it shows the interviewer what your fix gave up.

In plain words

Think of a restaurant kitchen. Scaling the kitchen is not buying more stoves at random. You find the station that makes every order wait, perhaps the grill, fix that, and then discover the next slowest station. The analogy breaks in one place. In a restaurant each dish uses one grill; in a fan-out system each request uses every shard, so one slow shard delays the whole order.

In one precise sentence: scaling a system means measuring its load, finding the bottleneck, removing it with a change whose cost you can name, and measuring again. The answer tells that story about one system you worked on. It has five parts: the system and its scale, your role and the goal, the bottleneck, the fix and its trade-off, and the result. Each part carries one specific fact. This framework is the site's own method, not a sourced standard.

1 · System and scale2 · Role and goal3 · Bottleneck4 · Fix and trade-off5 · ResultReplay

Step-by-step description

1. Name the system and its measured load.
2. Say what you owned and why scaling was needed.
3. Name the one limit and the percentile you measured.
4. State the change and what it cost.
5. Report the numbers and what you would change.

• The answer has five parts: system and scale, role and goal, bottleneck, fix and trade-off, and result.
• Each part carries one specific fact.

Step 1: State the system and its scale in numbers

The first part of the answer names what the system does and how big it is, using two or three measured numbers. Queries per second (QPS) is the number of requests a system serves each second, and it is the usual headline figure; add data size, the number of machines or the p99 latency target if they matter.

Say "about 1,200 queries per second at peak, a product search service" (an illustrative figure) rather than "very high traffic". Round honestly and say whether the number is peak or average. If you do not remember a figure, give a range and say so; an invented precise number invites the follow-up that exposes it.

The numbers in this page's example system are illustrative, chosen so the arithmetic is easy to check, and are not typical sizes for any company.¹ You now have the system on the table; the next step says where you stood in it.

Step 2: Say your role and the goal that forced scaling

The second part says what you owned and why scaling was needed. "I owned the serving path" and "I was one of six engineers on the ranking service" are both fine, and honesty about your share is a strength, because the interviewer will probe the parts you claim.

Name the trigger, since scaling is always a response to a goal: a launch that would triple traffic, a latency target the product team set, or a cost ceiling. The trigger tells the interviewer what "better" meant, which makes your later choices judgeable. A scaling story with no goal sounds like tuning for its own sake.

Keep this part to two sentences. Its job is to set up the bottleneck, not to describe the org chart. In the example system, the goal is a p99 latency of a quarter of a second at a peak load that is 1,200 queries per second (illustrative).¹

Step 3: Name the bottleneck, and remember that fan-out widens the tail

The third part names the one thing that limited the system, backed by a measurement such as a profile, a queue length or a latency percentile. This is where scaling answers are won or lost, because "we added servers" is not a bottleneck.

Many large systems have fan-out, and fan-out changes the arithmetic. Dean and Barroso (2013) note that temporary high-latency episodes that are unimportant in moderate-size systems can dominate overall performance at large scale. If each backend is slow on 1 request in 100, the chance that a request calling n backends hits at least one slow one is one minus 0.99 raised to the power n, assuming slow events are independent across backends; correlated slowness from shared hosts or garbage-collection storms changes the figure.

That is 63.4% at 100 backends, matching the table in the ClickHouse tail-latency explainer (last updated July 2026).¹ The same source cites the paper's extreme case: with only 1 request in 10,000 slow per server, a fan-out of 2,000 servers still leaves almost one request in five slow; the arithmetic gives 18.1%.¹

So a system can look healthy on average and still fail its p99 target. Say which percentile you measured.

Text description of the diagram

• With 1% of calls slow per backend, 63.4% of requests that call 100 backends hit a slow one.
• At 200 backends the share is 86.6%.

Step 4: Say the fix and the trade-off

The fourth part states what you changed and what it cost, because every scaling fix buys one property with another. Caching buys speed with staleness. Sharding buys capacity with fan-out and operational work. Extra headroom buys a short tail with money. Say the trade-off in the same breath as the fix.

For a fan-out tail, the Dean and Barroso (2013) abstract describes tail-tolerant techniques that can reuse resources already deployed for fault tolerance and let utilization rise without lengthening the tail. A hedged request is a second copy of a slow request sent to another replica, using whichever answer returns first, and it is one such technique. With 1% of calls slow and independent copies (an illustrative assumption), two copies are both slow 0.01% of the time, so a fan-out of 100 leaves about 1.0% of requests slow instead of 63.4%.¹ The price is extra load on the replicas.

Another fix is headroom: running instances at 60% busy instead of 80% takes 14 instances instead of 10 in the example, 40% more (illustrative).¹

Step 5: Give the result and what you would change

The last part reports what happened, in the same units as Step 1. "p99 fell from 900 to 240 milliseconds at the same traffic, and we spent about 40% more on instances" is a complete result, because it shows the improvement and the price (illustrative numbers).¹ If the result was mixed, say so; a candid "it fixed the tail but the cache invalidation bugs cost us a quarter" is more credible than a clean win.

End with one sentence on what you would do differently, such as load-testing the tail before launch or adding a capacity alarm earlier. That sentence is what separates a senior answer from a Staff one, since it shows that the system taught you something general.

The Staff-level answer

The answer below is an illustrative composite, not a real company's system, and its numbers are illustrative inputs and the arithmetic computed from them.¹

The answer is Staff-level because it picks the right percentile, names a mechanism rather than a tool, states its cost and ends with a lesson. Say it in about a minute, and stop talking when it ends so the interviewer can choose the follow-up.

The rubric

Three variants of this question

What is the hardest scaling problem you have faced?

Use the same five parts, but start at the bottleneck: the hardest problem is the one where the first fix failed or created a new problem. Say what you tried first and why it did not work.

How would you scale your system by ten times?

Redo Step 3 with the new load: find the next bottleneck, check whether fan-out grows with the new size, and say which fix you would try first and what would make you stop. Name the cost of ten times in money and in complexity.

What if you have never worked on a large system?

Pick the system where you measured the most, even a small one, and describe it with the same five parts; being exact about a modest system beats being vague about a large one. A load test of your own project, with its numbers, is a legitimate answer if you say plainly that it was a test.

Common mistakes

• "We scaled by adding more servers." (wrong) That names no bottleneck, and in a fan-out system more servers can widen the tail. (right) "Profiling showed the shard stage was the limit, so I fixed that stage and then re-measured the p99."
• "Average latency was fine." (wrong) Averages hide the slow requests users notice. (right) "The average was fine but the p99 missed its target."
• "It handled huge traffic." (wrong) A vague size is not a measured load. (right) "About 1,200 queries per second at peak" (illustrative).
• "We did everything ourselves." (wrong) Interviewers probe your share and find the gap. (right) "I owned the serving path; a teammate owned the index build."
• "It was a complete success." (wrong) No trade-off sounds untrue. (right) "It cut p99, and it cost about 40% more in instances."

Follow-up questions

What number should I give for the scale of my system?

Give the number you measured and used to make decisions, usually requests per second at peak, and say whether it is peak or average. If you cannot recall it, say "roughly" and give a range.

What if I worked on only one part of the system?

Describe the part you owned as the system for this answer, and say once how it fits in the larger one. The interviewer wants to hear decisions you made, not the size of the company.

What if my company's numbers are confidential?

Give orders of magnitude, ratios or relative changes instead of exact figures: "tens of thousands of requests per second", or "p99 fell to roughly a quarter of what it was". Say once that the exact figures are confidential, then keep every other part of the story specific.

How do I describe the scale of a machine learning system?

Use the same measures plus the ones that are specific to it: requests per second at serving time, the latency percentile at the model call, the number of model replicas or graphics processing units (GPUs) and the size of the data you train on. A design such as a search ranking system or an ads ranking system shows how these numbers are used, and estimated time of arrival (ETA) prediction and anomaly detection show where scale numbers come from in other systems.

Why does adding servers sometimes make latency worse?

When one request calls many servers and waits for all of them, each extra server is another chance to be slow. The Dean and Barroso (2013) paper is the standard reference. The typeahead design and real-time fraud detection design are examples where tight latency targets make the tail the main constraint.

How long should the answer be?

About a minute, or 60 to 90 seconds, then stop and let the interviewer pick the thread. Practice the answer on your own system until each of the five parts is one or two sentences.

Sources

1. The Tail at Scale — Google Research (Dean and Barroso, Communications of the ACM 56), 1 Feb 2013
2. What is tail latency? Why p99 spikes when averages look fine — ClickHouse, 29 Jul 2026