ML System Design

Are there some limitations to these metrics?

By Interview Clozo · Published 27 October 2026 · Last verified 26 October 2026 · 13 min read

Are there some limitations to these metrics? Yes: accuracy, precision, recall and the F1 score each summarize a model at one threshold on one dataset, so they hide class imbalance, the base rate and unequal error costs.¹ A common slip is quoting a number without its threshold, and the first step below shows why that fails.

In this article

Before you start
Why interviewers ask this
Metric limitations from first principles
Worked example
The Staff-level answer
The rubric
Three variants of this question
Common mistakes
Follow-up questions
Sources

Before you start

This page is one question inside the machine learning (ML) system design interview guide. "These metrics" here means the usual set for a classifier: accuracy, precision, recall, the F1 score and the precision-recall and receiver operating characteristic (ROC) curves.

Why interviewers ask this

In a system design round the metric is the first decision, because every later choice (data, loss, threshold, monitoring) is tuned to it. Interviewers ask about its limits to see whether a candidate treats a metric as a fact about the model or as a statement about a decision.

A weak answer lists the formulas and says "accuracy is bad when classes are imbalanced." A strong answer shows three separate ways a number misleads, with arithmetic. A Staff-level answer connects the choice to the decision: the scikit-learn guidance is to start from the ultimate goal of the prediction and to use the scoring function you are given, for example in a business context, when there is one. The same reasoning runs through designs such as real-time fraud detection and ML for healthcare, where the two kinds of error cost very different amounts.

Metric limitations from first principles

Operating point: An operating point is the specific threshold a deployed classifier uses, together with the precision and recall it produces there. Two models with the same curves can run at different operating points, and users only ever experience the operating point, not the curve.

In plain words

Imagine a smoke alarm. "It is right 99% of days" sounds excellent. But an alarm that never rings is right on almost every day, because there is almost never a fire. The number is true and still useless. It does not describe what you care about: does it ring when there is a fire, and how often does it ring for nothing?

The analogy breaks in one place. A smoke alarm has one sensitivity dial, while a model has a score for every example and you choose where to cut. Put precisely: every metric below is a function of a model's scores, a threshold and a labeled dataset. Each one throws part of that information away.

Precision is the number of true positives over the number of true positives plus false positives; recall is the number of true positives over true positives plus false negatives. Accuracy is the share of all predictions that are correct.

Step 1: Every metric describes one threshold on one dataset

A metric is a snapshot: the same model gives different numbers at different thresholds and on different data. The precision-recall curve shows the trade-off between precision and recall for different thresholds, so a single precision figure is meaningless until you say where the cut was made.

The default cut is a convention. scikit-learn predicts the positive class when the probability is greater than 0.5, a rule its documentation calls "most certainly not ideal for most use cases"; its insurance example reaches a cost-driven threshold of about 0.02. Curves are no escape either: the same page shows a default and a tuned classifier that share the same ROC and precision-recall curves because they share scores, even though their predicted labels differ. Curves ignore the operating point.

You now know the root limit: a number needs its threshold and its data. Next, the first way a number misleads.

Step 2: Accuracy hides class imbalance

Take 10,000 examples in which 1% are positive (illustrative). A model that always predicts negative scores 99% accuracy and zero recall: it is right on 9,900 cases and misses all 100 positives.¹ On the page that defines it, precision-recall is described as useful when the classes are very imbalanced for exactly this reason.

The effect appears on any task where the case you care about is rare: fraud, disease, abuse. scikit-learn notes that accuracy is the default score for classifiers, which is why it is so often reported without a second thought.

The fix is not a cleverer single number. Report recall for the rare class next to precision, and look per class, because a macro or weighted average across classes can also hide one class that fails completely.

You now know accuracy can look excellent with a useless model. Next: why precision falls even when the model is good.

Step 3: Precision falls with the base rate

Precision depends on how rare positives are, not only on how good the model is. At 1% prevalence, 90% recall and a 5% false-positive rate give a precision of about 15%: of 10,000 examples the model catches 90 of the 100 positives but also raises 495 false alarms.¹ Moving the same model to a population with 10% positives lifts precision to about 67%, with nothing changed in the model.¹

At a fixed recall and false-positive rate, precision follows the formula below, so it falls as the base rate falls. Here R is the recall, F is the false-positive rate and π is the base rate.

precision = \frac{R π}{R π + F (1 - π)}

Two consequences follow. First, a result measured on a balanced test set does not carry over to production. Second, when positives drift in frequency, a metric can fall without the model getting worse. For a detector in production, such as harmful-content detection or an anomaly-detection system, you therefore estimate the base rate first and compute the expected number of false alarms per day, which is the number the operations team feels.

Step 4: Averages like F1 hide which side is weak and assume equal costs

The F1 score collapses precision and recall into one number. Take the scikit-learn toy case of four labels where the model gets precision 1.0 and recall 0.5 (model evaluation documentation). Its F1 is 0.67, a respectable-looking figure that does not reveal that half the positives were missed.¹ Weighting changes the picture: on the same predictions the F-measure that favors precision (F0.5) is 0.83, and the one that favors recall (F2) is 0.56.¹

The deeper limit is cost. F1 weights a false alarm and a miss equally. The scikit-learn tumor-detection example argues the opposite: the cost of a missed cancer is much higher than the cost of further diagnostic tests, so physicians prioritize recall and accept more false positives. If the two errors have different prices, choose the metric and the threshold from those prices.

Step 5: Offline metrics are proxies for the online outcome

A held-out metric is a proxy for the outcome the business cares about, so it can improve while that outcome does not. The held-out set is a frozen sample of past data. Once the model is live, its actions change the data it later sees (a feedback loop), labels can arrive weeks late (label delay), and the population drifts (distribution shift). Each of these makes the offline number a stale or biased estimate.

The check is an online experiment, an A/B test, that compares the new model with the current one on the business metric. The scikit-learn guidance is to start from the ultimate goal of the prediction, and that goal is measured online. An offline metric is only a proxy, and an online experiment checks it: use the offline number to screen candidates and the experiment to decide.

Each common metric hides a different failure, so name the failure before you trust the number.CC BY 4.0 — source: Are there some limitations to these metrics?

Step-by-step description

A metric is a snapshot at one cut-off.
Accuracy can hit 99% with zero recall.
Rare positives lower precision.
F1 assumes equal error costs.
An online test checks the proxy.

A model that always predicts negative scores 99% accuracy and zero recall when 1% of examples are positive.
At 1% prevalence, 90% recall and a 5% false-positive rate give a precision of about 15%.
F1 averages precision and recall and assumes their errors cost the same.
An offline metric is only a proxy, and an online experiment checks it.

Knowing it is half the job. Saying it is the other half.

Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.

Start Free Trial →

Worked example

Twenty examples show the effect of the threshold. Six are positive with scores 0.95, 0.85, 0.70, 0.45, 0.32 and 0.12; fourteen are negative, the highest scoring 0.60, 0.40 and 0.35 and the other eleven 0.25 or less, one of them 0.11 (illustrative scores). A score counts as positive only when it is above the threshold.¹

At a threshold of 0.5, the model flags four examples, three of them truly positive: precision is 0.75, recall is 0.5 and accuracy is 0.8. At 0.3 it flags eight, five of them positive: precision is 0.63, recall is 0.83 and accuracy is still 0.8.¹ Accuracy is 0.8 at both 0.5 and 0.3, while recall rises from 0.5 to 0.83. At 0.1 it flags fourteen, catching all six positives: recall is 1.0, precision falls to 0.43 and accuracy drops to 0.6.¹

Accuracy was identical at the first two cut-offs, yet recall differed by a third, and the best F1 sat at the middle threshold, not at the default.
Threshold	Precision	Recall	Accuracy	F1
0.5 (the default)	0.75	0.5	0.8	0.6
0.3	0.625	0.8333	0.8	0.7143
0.1	0.4286	1.0	0.6	0.6
Always predict negative (baseline)	undefined	0.0	0.7	0.0

The always-negative baseline already reaches an accuracy of 0.7 here, so accuracy of 0.8 is only a small gain over doing nothing.¹ With a missed positive ten times as costly as a false alarm (illustrative), 0.1 is the cheapest of the three (cost 8 against 13 and 31); with equal costs, 0.5 and 0.3 tie at four errors each, which is what equal accuracy means.¹ The metric does not choose; the cost does.

Moving the threshold trades precision for recall, and accuracy stayed flat between the first two cut-offs while recall jumped.CC BY 4.0 — source: Are there some limitations to these metrics?

Text description of the diagram

Accuracy is 0.8 at both 0.5 and 0.3, while recall rises from 0.5 to 0.83.
At a threshold of 0.1 recall is 1.0, precision falls to 0.43 and accuracy drops to 0.6.