ML system design interview: how the round works and how to run it

In this article

1. Start here: the learning path
2. How the round works and what it scores
3. Run the 45-minute clock
4. Operating the system: debt, drift and the 2am question
5. Questions interviewers ask on ML system design
6. All 36 guides in ML system design
7. Follow-up questions
8. Sources

Start here: the learning path

The five steps below are the order to run the round in. Say the order aloud in the first minute so the interviewer knows where you are going. The sections after the diagram take the steps one at a time.

ML system design interview

An ML system design interview is a round in which a candidate designs a machine-learning system for an open-ended product goal, judged on structure, trade-offs and operations rather than on code or on one correct model.

Candidate funnel

A candidate funnel is a pipeline of stages, usually retrieval and then one or more ranking stages, in which each stage runs a costlier model on fewer items than the stage before; many systems end with a re-ranking step that adjusts the short list.

Latency budget

A latency budget is the total time a request may take from arrival to answer, divided among the stages that handle it, so that a slow stage must be paid for by a faster one.

Text description of the diagram

• An answer has five steps: frame, data and labels, model and serving, evaluation, failure modes.
• Failure modes are the last step and are never skipped.
• Rule #2 of Zinkevich's Rules of Machine Learning supports the habit of instrumenting first.

An answer has five steps: frame, data and labels, model and serving, evaluation, failure modes.

1. Frame the problem. The decision, the user, the metric and the constraints.
2. Data and labels. What one training example is, where labels come from, and how late they arrive.
3. Model and serving. The simplest model that meets the latency budget, and the funnel around it.
4. Evaluation. Held-out metrics first, then an A/B test on live traffic that decides the launch, then a ramp-up.
5. Failure modes and operations. Drift, feedback loops (the model's own choices becoming its next training data), on-call, and what happens when a part fails.

Failure modes are the last step and are never skipped. The sections below take the steps in turn, then two worked designs; failure modes and operations get their own section after the sample clock.

1. Computer vision system design interview questions and answers
2. From business goal to ML task: how do you frame the problem?
3. Retrieval, ranking, re-ranking: how does the multi-stage funnel work?
4. How do you design a search ranking system?
5. How do you design a typeahead (query autocomplete) system?
6. What is one row of your training data?
7. Design real-time fraud detection
8. Design harmful-content detection
9. How do you design on-device and privacy-constrained ML?
10. Pre-deployment metrics looked great, the A/B test did not: why?
11. How do you design an ads ranking system?
12. How do you design an experimentation platform?
13. How do you design a text-classification system at scale?
14. Design ETA prediction
15. How do you design ML for healthcare?
16. What changes if this has to return in 100 milliseconds: the latency budget
17. Design dynamic and surge pricing for a marketplace
18. How do you design an autonomous-vehicle perception stack?
19. How do you design a multimodal embedding model and the service around it?
20. How do you design the AV data engine?
21. How do you design a forecasting system?
22. How do you design visual search, step by step?
23. How do you design an anomaly-detection system?
24. How do you design churn, LTV and credit-risk prediction?
25. Adversaries adapt: what it means for ML system design
26. How do you design abuse and access-control infrastructure?
27. How do you reduce tail latency with spiky traffic?
28. What's the largest system you've worked on, and how did you scale it?
29. How do token limits impact GenAI application design?
30. Can you identify flaws in ML systems and propose enhancements?
31. How do you design an LLM document-processing pipeline?
32. Design a feature store for machine learning
33. How would you build a model to detect fraudulent host listings?
34. What breaks at scale?
35. How do you scale an AI search assistant to 10M articles?
36. And how would you define that metric itself?
37. Are there some limitations to these metrics?
38. What metric are we using here to decide if we can roll out or not?
39. Offline vs online evaluation: why do you need both?

Frame the problem before choosing a model

Framing means stating the decision the system makes, who it affects, the metric you will watch and the constraints. A proxy metric is easy to measure and can be wrong. Google's Rules of Machine Learning (Zinkevich, updated August 2025) says so in Rule #13: "The ML objective should be something that is easy to measure and is a proxy for the 'true' objective". Rule #39 adds that launch decisions are a proxy for long-term product goals. Covington, Adams and Sargin (2016) give a concrete case: ranking by click-through rate often promotes videos that users do not finish, so their ranker predicts expected watch time instead.

Name the proxy, name the true goal, and say how you would check that they move together. Then turn the goal into a task: "reduce churn" becomes a prediction target, a decision threshold and the action that follows a high score, because a prediction nobody acts on has no value. Rule #2 supports the habit of instrumenting first: before building the model, track as much as possible in the current system. For the measurement foundations see probability and statistics.

What is one row of your training data?

One training example is the inputs known at decision time, the label that later says what happened, and the timestamp. The question exposes three traps at once: features that would not have existed at serving time, labels that arrive days later, and negative examples that are not really negative. A user who did not click has not necessarily rejected the item; they may never have seen it.

Define the row aloud, then state the label delay and how you handle it. For fraud the label may arrive weeks later when a dispute closes, so you train on older data and accept that the newest behaviour is unlabelled. Say which fields come from logs at the moment of the decision, which are looked up later, and which you must exclude to avoid leaking the answer into the inputs. A strong answer shows that the design could be built on a real data pipeline, so a clear definition of the row earns more credit than a longer list of features.

The funnel: retrieval, ranking and re-ranking

A candidate funnel exists because the best model is too expensive to run on every item. Covington, Adams and Sargin (2016) describe YouTube's recommender as the classic two-stage approach: a deep candidate generation model produces a manageable set, and a separate deep ranking model orders it.

The arithmetic shows why. Take an illustrative search index of 10 million documents: a cheap retrieval model keeps 1,000, a light ranking model keeps 100, and a heavy re-ranking model such as a cross-encoder keeps 10. The heavy model then scores 100 documents instead of 10 million, which is 100,000 times fewer; the funnel's total model time is 202 model-seconds against 100,000 for brute force ¹. These costs are assumptions, not measurements, and in practice retrieval uses an index so it costs less than scoring every item, but the ratio is the interview point.

In recommenders the heavy model usually sits at the ranking stage, and a final re-ranking step applies integrity filters and diversity or business rules (Meta's Instagram Explore write-up, August 2023).

Text description of the diagram

• A cheap retrieval model keeps 1,000 of 10 million documents.
• Light ranking keeps 100 and heavy re-ranking keeps 10.
• The heavy model scores 100 documents, which is 100,000 times fewer than brute force.

For the retrieval stage see embeddings and vector search, and for the last stage see re-ranking with a cross-encoder. Evaluate each stage on its own: retrieval by how often the right item survives into the short list, ranking and re-ranking by how well they order what reaches them.

What changes if this has to return in 100 milliseconds?

A latency budget turns a vague wish for speed into a design constraint. Fix the total, say an illustrative 100 milliseconds for the whole request, then divide it: a few milliseconds for the cheap first stage, more for ranking, and a hard reserve for the network and for feature lookups. Whatever a stage costs comes out of the next stage's share.

Under a tight (illustrative) budget the usual moves are fewer features, a smaller model, cached results for popular requests, and a cheaper first stage that sends fewer items onward. State the trade each time: a smaller model usually loses some accuracy, and a cache serves slightly stale answers. Walk the budget stage by stage and say which stage you would cut first.

Evaluate before and after launch

Evaluation has two halves. Before launch you measure the model on a held-out set, which is data the model never trained on, using a metric chosen in the framing step; this is cheap and fast but only a proxy. Then an A/B test on live traffic decides the launch: a random share of users gets the new system and the rest keep the old one, and you compare the metric you named in advance. A ramp-up sends the new system to a small share of traffic first, then raises it while the numbers hold.

The two halves can disagree. The held-out set was logged by the old system, so it shows only what that system chose to show, and the new model changes what users see. Say which metric would stop the launch and which would only be watched. When held-out gains fail online, check first for training-serving skew: a feature computed one way in training and another at serving.

Rule #37 of Google's Rules of Machine Learning says a gap between next-day and live performance likely signals an engineering error. Then check the held-out set's bias toward what the old system showed, and then whether the held-out metric tracks the online one. Feedback loops, where the model's own choices become its next training data, show up later and are watched after launch.

Design a search ranking system

A worked design shows the framework end to end. Query understanding turns text into a form the system can match, retrieval pulls a candidate set from an index, a learned ranking model orders it, and the metric must not reward clicks alone, for the proxy reason above. Pair clicks with a measure of whether the user found what they wanted, such as the absence of an immediate return to the results. Walk the five steps in order and spend the time on the data and the metric, because the model choice here is the least surprising part.

Design real-time fraud detection

Fraud detection stresses every step. Fraud is rare, so a model that always answers "not fraud" looks accurate while being useless; the classes are extremely imbalanced and you need metrics that separate the rare class. Labels arrive late, so the training row must say how old it is. And the adversary adapts: a rule that works this month is probed and avoided next month, so monitoring and retraining are part of the design, not an afterthought. Say how each of these three constraints reshapes your answer.

How the round works and what it scores

In our judgement, a design round rewards a connected structure at least as much as a clever model: the interviewer wants to see the parts joined into one flow, from the goal to the data to the model to how it is checked and kept running. This is the author's reading of how such rounds are run, not any employer's published rubric.

Google's Rules of Machine Learning supports two parts of this order: metrics before models (Rule #2) and a simple first model (Rule #4). Rule 1 is "Don't be afraid to launch a product without machine learning", which is a useful opening question: would a heuristic do? The guide's four phases run from "Before Machine Learning" through the first pipeline and feature engineering to slowed growth and complex models, and they describe how a real system matures, not the order of an interview answer. Naming the rule you rely on and applying it to the prompt shows that your design follows published practice.

Run the 45-minute clock

A design round needs a time budget, because a model-only answer leaves no time for the data, the evaluation or the failure story. The split below is an illustrative budget for a 45-minute round (it is the author's judgement, not a published standard; real rounds vary in length) ¹.

Text description of the diagram

• The illustrative split is 8, 7, 15, 10 and 5 minutes across a 45-minute round.
• Model and serving take the most time at 15 minutes.
• Failure modes and follow-ups keep a reserved block of 5 minutes.

Operating the system: debt, drift and the 2am question

A design is not finished until you say how it fails and who is called. Sculley et al. (2015) note that it is common to incur massive ongoing maintenance costs in real-world ML systems, and name risk factors including boundary erosion, entanglement, hidden feedback loops, undeclared consumers and data dependencies. Pick two that fit the prompt and say how you would limit them. Also name data drift, the change over time in live inputs that makes a trained model stale. The wider operations toolkit is in the machine learning operations (MLOps) hub and large language model (LLM) observability.

Questions interviewers ask on ML system design

Questions come in three families: open design prompts, deep-dive questions that disturb a design you have already given, and project walk-throughs of a system you built.

Typical prompts, each with a one-line answer:

• Design a video recommendation feed. Retrieve a few hundred candidates cheaply, rank them with a heavier model, then apply diversity and freshness rules; the label is watch time, not clicks.
• Predict ad click-through. Train a probability model and check calibration, because the auction uses the predicted probability itself, not only the ordering.
• Classify harmful content. Trade precision against recall by severity, and send borderline cases to human review rather than forcing a yes or no.
• Predict the estimated time of arrival (ETA) of a trip. A regression problem whose label is the actual arrival time, known only after the trip ends, so labels arrive late.
• Build query autocomplete under a latency budget. Use a cheap candidate source, a small ranker and a cache for popular prefixes, and say what accuracy each cut costs.
• Held-out metric up, A/B test flat: why? Check training-serving skew first, then logged-data bias, then metric mismatch.

The wider interview loops hub shows where this round sits, the system design hub covers the non-ML counterpart, the classical ML hub covers the models, and the AI engineer question list covers language-model application questions.

Follow-up questions

How would you make it operationally sustainable and reliable?

Monitor inputs and outputs for drift, keep a simple fallback that does not need the model, retrain on a schedule with a rollback, and name an owner for alerts.

What if this part fails at 2am and you are not on call?

Design for a safe default: the fallback serves a heuristic result, alerts page the owner, and nothing needs a human to restore basic service.

What's the largest system you've worked on, and how did you scale it?

Name the scale in numbers you can defend, the bottleneck you hit first, and the change that moved it, such as adding a funnel stage, caching or a smaller model. Then say what you would do differently now.

Does the ML coding round involve implementing algorithms from scratch?

It can: an ML coding round may ask you to implement a model such as logistic regression or k-means in plain Python or NumPy instead of calling a library. Some loops pair the system design round with a separate ML coding round; see the ML coding hub for the topics and the numerical traps.

Why did you pick that metric before the model?

Because a model chosen before the success measure cannot be defended. Rule #2 of Google's guide says to track the system first, and Rule #13 says the objective should be easy to measure and a proxy for the true goal.

Sources

1. Deep Neural Networks for YouTube Recommendations — Google Research (Covington, Adams, Sargin), RecSys 2016, 15 Sep 2016
2. Scaling the Instagram Explore recommendations system — Engineering at Meta, 9 Aug 2023
3. Hidden Technical Debt in Machine Learning Systems — NeurIPS 2015 (Sculley et al.), 7 Dec 2015
4. Rules of Machine Learning: Best Practices for ML Engineering — Google for Developers (Zinkevich), 25 Aug 2025