Can you identify flaws in ML systems and propose enhancements?

In this article

1. Before you start
2. Why interviewers ask this
3. Finding flaws in an ML system from first principles
4. The Staff-level answer
5. The rubric
6. Three variants of this question
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one question inside the ML system design interview guide, which explains how the whole round runs.

Why interviewers ask this

A candidate who only praises a design shows that they have not run one. Candidates report this prompt as an open critique in ML system design rounds, sometimes about their own design and sometimes about a system the interviewer describes.

The question separates people who know the model from people who know the system. A weak answer lists generic problems: "the data may be biased, the model may overfit." A strong answer picks one flaw, says where it lives, shows the number that proves it and names a fix with a cost. A Staff-level answer also says how the fix will be checked after launch, because ML systems fail quietly.

Finding flaws in an ML system from first principles

Hidden feedback loop

A hidden feedback loop is a path by which a model's outputs change the data it is later trained on, so the model shapes its own future training set without anyone having planned that effect.

In plain words

Think of a car inspection. A good mechanic does not admire the paint. They check the engine, brakes and tires one by one. They measure something and say what to replace first. The analogy breaks in one place. Roads stay the same, but the data an ML system sees keeps changing, so the inspection never ends.

In one precise sentence, finding a flaw means locating it in a layer, proving it with a metric that fits the problem, and proposing the smallest change that fixes it.

Step 1: Walk the system from data to feedback

Walk four layers in order: data, model, serving, and monitoring with feedback. Data holds label errors, bad joins and leakage of future information into training rows. The model layer holds the objective and the slices it serves badly. Serving holds skew and latency; a tight latency limit changes the whole design, as what changes when the answer must return in a hundred milliseconds explains. Monitoring holds drift and feedback loops.

Sculley et al. (2015) list ML-specific risk factors that include boundary erosion, entanglement, hidden feedback loops, undeclared consumers, data dependencies and changes in the external world. Google's Rules of Machine Learning (Rule 10) warns about silent failures: a table that stops updating makes the system decay gradually instead of crashing.

Input: the system diagram. Output: one suspect per layer. You now have candidates, but no proof yet.

Step 2: Prove the flaw with a number that fits the class balance

Pick the suspect you can measure and prove it with a metric that suits the data. Take an illustrative fraud model on 10,000 transactions with 100 frauds.¹ A rule that always says "not fraud" scores 0.99 accuracy and catches nothing, so accuracy is the wrong metric.¹ The toy model scores 0.986 accuracy, 0.375 precision and 0.6 recall.¹

Now cut the same results by slice. Domestic transactions (illustrative: 60 frauds) have recall 0.9, while new-country transactions (illustrative: 40 frauds) have recall 0.15.¹ The overall 0.6 hid a group where the system misses most fraud. The flaw is proved with a metric cut by slice.

Input: predictions and labels. Output: a per-slice table that names the flaw. You now know where the system fails, so the next step is the change.

Step 3: Propose one fix and its cost

Propose the smallest fix that moves the proven number, and state what it costs. Here a separate, lower threshold for the new-country slice makes new-country recall rise from 0.15 to 0.7 after the fix, while overall recall rises from 0.6 to 0.82 and domestic recall stays at 0.9. False alarms rise from 100 to 240 and precision falls from 0.375 to 0.2547.¹ The cost is more review work, and a good answer says so.

Prefer simple fixes first. Google's Rules of Machine Learning advise a simple first model (Rule 4) and note that interpretable models are easier to debug (Rule 14). Other fixes follow the flaw: fix the join for a data flaw, log serving-time features for skew, add a monitor for silent decay.

Text description of the diagram

• New-country recall rises from 0.15 to 0.7 after the fix.
• Overall recall rises from 0.6 to 0.82 and domestic recall stays at 0.9.

Step 4: Check the fix against the same slices, then keep watching

The fix is verified on the same slices, then the cause is monitored. Re-run the per-slice metrics from Step 2 and confirm the target slice moved and the others did not drop. For skew, check that the model gives the same score in the training environment and the serving environment on the same inputs, as Rule 5 of the Rules of Machine Learning advises.

For drift, the population stability index (PSI) is a number that compares how a feature's values are spread across bins in two periods. The formula is:

Here $e_i$ is the training share of bin $i$ and $a_i$ is the serving share. In words: for each bin, multiply the change in share by the log of the ratio of shares, then add the bins up. For illustrative bins with shares 0.25 each in training and 0.10, 0.20, 0.30 and 0.40 in serving, the four terms are 0.1374, 0.0112, 0.0091 and 0.0705, so the PSI is 0.2282; for identical shares it is 0.¹ Cutoffs for a "bad" PSI are industry convention, not a standard, so state your own alert level.

1 · Walk the layers2 · Prove the flaw3 · Propose one fix4 · VerifyReplay

Step-by-step description

1. List one suspect per layer.
2. Cut a fitting metric by slice.
3. State what the fix costs.
4. Re-run the slices and add a monitor.

• The audit walks four layers: data, model, serving and monitoring.
• The flaw is proved with a metric cut by slice.
• The fix is verified on the same slices.

The Staff-level answer

What makes it Staff-level: it names one flaw with a number, states the cost of the fix, and says how the fix will be checked and watched.

The rubric

Three variants of this question

How would you debug a model that scored well before launch but fails in production?

Suspect training-serving skew first: compare the features and scores the model sees in serving with those in training for the same inputs, then check for stale data sources, as Rule 10 of Rules of Machine Learning describes.

What is the first thing you would fix in a system you did not build?

Start with measurement: confirm the metric fits the problem and cut it by slice. A fix to an unmeasured system cannot be checked, so measurement comes before any change.

What if only a tiny share of cases are positive?

Then accuracy hides failure. In the example above, a rule that never flags anything scores 0.99 accuracy with zero recall.¹ Use precision, recall and per-slice recall, and size the threshold to the cost of a miss against the cost of a false alarm.

Common mistakes

• "The pre-deployment score is high, so the system is good." (wrong) A pre-deployment score can differ from production because of skew and slices. (right) "The aggregate looks fine, so I cut it by slice and compare serving with training."
• "I would retrain more often." (wrong) Retraining on data shaped by the model's own outputs can reinforce a hidden feedback loop, which Sculley et al. (2015) list as an ML-specific risk. (right) "I would first check what the retraining data contains."
• "The model is the whole system." (wrong) Data, serving and monitoring carry flaws too. (right) "I would walk data, model, serving and monitoring."
• "I would use a bigger model." (wrong) It is an untested fix with no cost stated. (right) "I would change one thing, state its cost and re-run the same metric."

Follow-up questions

How do you find flaws in an ML system you did not build?

Ask for the system diagram, the metrics and the data sources, then walk the four layers. Look first for what nobody monitors, since silent failures sit there.

How do you know your fix worked?

Re-run the same slice table on fresh data and check that the target slice moved without hurting the others. Then keep a monitor on the cause so a regression shows up quickly. Real-time fraud detection shows this pattern end to end.

What is training-serving skew and how do you catch it?

Training-serving skew is a gap between training and production data or code. Catch it by logging serving-time features and comparing the model's score in both environments on the same inputs. The anomaly-detection design page covers monitoring the same drift.

Does retraining more often fix a drifting model?

Not by itself. If the cause is a broken feature or a feedback loop, more retraining repeats the flaw; validate the data first.

Where do flaws show up in a multi-stage system?

At every stage boundary: a weak retriever caps what a ranker can do. Track a metric per stage, such as how often the retriever's list contains a good item and how well the ranker orders that list, so a flaw is traced to one stage. See retrieval, ranking and re-ranking, and harmful-content detection for how slice-level harm is measured.

Sources

1. Hidden Technical Debt in Machine Learning Systems — Advances in Neural Information Processing Systems 28 (Sculley et al.), 7 Dec 2015
2. Rules of Machine Learning: Best Practices for ML Engineering — Google for Developers (Martin Zinkevich), 26 Oct 2026