In this article
Before you start
The machine learning algorithms overview covers the models these metrics judge, and probability and statistics covers the maths underneath.
Why interviewers ask this
A wrong metric ships a wrong model. A team that tracks accuracy on rare-event data can celebrate a model that finds nothing. Interviewers ask the question to see whether you tie the metric to the task, the cost of each error and the class balance, which is what Google's Machine Learning Crash Course says decides which metrics are meaningful.
A weak answer recites a list: accuracy, precision, recall, F1. A strong answer says which metric fits which problem and what each one hides. A Staff-level answer also says where and how the number is measured: on held-out data (examples the model never trained on), at a stated threshold, with a cost attached, and watched for distribution shift after launch. The follow-up usually arrives within a minute, so expect to defend the choice, not just name it.
Model evaluation metrics from first principles
- Area under the ROC curve
- Area under the ROC curve is a metric that scores how well a classifier ranks positive examples above negative ones across all thresholds, so it summarises ranking quality without committing to one cutoff.
In plain words
Think of a smoke alarm. Precision asks: when it rings, was there a fire? Recall asks: of the real fires, how many did it ring for? An alarm that rings all day has perfect recall and poor precision. An alarm that never rings has no false alarms and no recall.
The analogy breaks in one place. A real alarm has one fixed sensitivity. A model has a dial, and you can turn it after training.
In one precise sentence: an evaluation metric turns a model's predictions and the true answers into one number, and the right number depends on the task and on what each kind of error costs.
Step 1: Count the four outcomes before you pick a metric
A confusion matrix comes first, because every classification metric reads from it. Accuracy is the share of all predictions that were correct: true positives plus true negatives, divided by all four counts (Google crash course).
Take the setup of a worked imbalanced-classification example: 100 real positives and 10,000 real negatives. The model finds 90 positives, raises 30 false alarms, misses 10 and labels 9,970 negatives correctly.1 Accuracy is 10,060 of 10,100, or 0.9960. A model that says negative for everything scores 0.9901 and has a recall of 0.1
Google notes that when a data set is imbalanced, or one error costs more than the other, you should optimize a different metric (crash course). You now have the four counts. Step 2 turns them into metrics that see the rare class.
Text description of the diagram
- Accuracy is 0.9960 for the model and 0.9901 for a model that predicts all negative.
- Recall is 0.9 for the model and 0 for the all-negative model.
Step 2: Precision, recall and F1 score are computed at one threshold
Precision and recall describe the model at one decision threshold. A threshold is the score cutoff above which a classifier says positive. The F1 score is the harmonic mean of precision and recall (an average that stays low unless both are high), 2 × precision × recall divided by their sum (Google glossary).
In the example, precision is 90 of 120 predicted positives, 0.75. Recall is 90 of 100, 0.9. The F1 score is 0.8182.1
Moving the threshold trades one for the other. Google warns that 0.5 is not a safe default when error costs differ or classes are imbalanced (thresholds page). The table uses four positives and six negatives with illustrative scores.1
| Threshold | True positives | False positives | Precision | Recall | F1 score |
|---|---|---|---|---|---|
| 0.75 | 2 | 0 | 1.0 | 0.5 | 0.6667 |
| 0.5 | 3 | 1 | 0.75 | 0.75 | 0.75 |
| 0.38 | 3 | 2 | 0.6 | 0.75 | 0.6667 |
| 0.25 | 4 | 3 | 0.5714 | 1.0 | 0.7273 |
Step 3: Area under the ROC curve ranks positives above negatives across all thresholds
Area under the ROC curve is the probability that a model scores a random positive above a random negative, measured across every threshold, and it is called area under the ROC curve (AUC) for short. ROC stands for receiver operating characteristic (ROC). The ROC curve plots the true positive rate (another name for recall) against the false positive rate (false alarms divided by all real negatives) as the threshold moves, and the area under it is the AUC (Google glossary).
A coin-flip model has an AUC of 0.5 (Google ROC page). In the toy scores, the model ranks the positive above the negative in 20 of 24 pairs, an AUC of 0.8333.1 That is not an accuracy.
AUC needs no threshold, so it differs from F1 score. Google says ROC and AUC suit roughly balanced data, and that precision-recall curves may compare models better when classes are imbalanced (source). A no-skill precision-recall baseline equals the share of positives (source): 100 of 10,100, or 0.0099, in the example.1 See the scikit-learn ROC curve guide for code.
Step 4: Regression metrics measure the size of the error
Regression metrics measure how far predictions fall from true values. Mean absolute error (MAE) is the average of the absolute gaps. Root mean squared error (RMSE) is the square root of the average squared gap. R² is one minus the model's squared error divided by the squared error of always predicting the mean. scikit-learn implements all three, plus median absolute error and mean absolute percentage error (scikit-learn 1.9.1).
Use six illustrative true values, 10, 12, 15, 18, 22, 25, and predictions that miss by one or two units. MAE is 1.1667, RMSE is 1.2247 and R² is 0.9464.1 Now let one prediction miss by 10: MAE rises to 2.6667 but RMSE jumps to 4.2426, because squaring weights big misses heavily.1 Pick RMSE when large misses are costly and MAE when they are noise. See linear regression assumptions for when residuals (the gaps between predictions and true values) behave.
Step-by-step description
- Four counts feed every classification metric.
- Precision and recall move with the cutoff.
- Area under the curve covers every cutoff.
- Regression uses absolute and squared error.
- Count true positives, false positives, false negatives and true negatives first.
- Precision and recall are computed at one threshold.
- Area under the curve ranks positives above negatives across all thresholds.
- Mean absolute error and root mean squared error measure the size of regression errors.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →The Staff-level answer
What metrics do you use to evaluate models?
I choose them from the task and the cost of each error, then measure on held-out data. For a classifier I start from the confusion matrix, not accuracy, because on imbalanced data a model that predicts the majority class scores high and finds nothing. I report precision and recall at the threshold we will ship, set by the cost of a false alarm against a miss, and use F1 only when both matter equally.
For ranking quality I use AUC when classes are balanced and the precision-recall curve when positives are rare. For regression I use absolute error when outliers are noise, squared error when big misses hurt, and R² for context. After launch I check the metric still tracks the product metric in a live A/B test, and watch inputs for drift.
What makes it Staff-level: each metric is tied to a cost, each is named with what it hides, the answer says where the number is measured, and it ends with a post-launch check against the product metric.
The rubric
| Dimension | Weak answer | Strong answer | Staff answer |
|---|---|---|---|
| Choice of metric | Lists accuracy, precision, recall | Matches the metric to the task | Ties each metric to a cost of error |
| Class imbalance | Trusts accuracy | Uses precision, recall, precision-recall curve | Shows the do-nothing baseline first |
| Threshold | Uses 0.5 by default | Knows precision and recall move with it | Sets it from error costs, then validates |
| Regression | Says "error" | Compares absolute and squared error | Explains outlier weighting and checks residuals |
| After launch | Stops at the test score | Mentions held-out data | Monitors for drift and re-checks the metric |
Three variants of this question
What metrics would you use for an imbalanced dataset?
Start with the do-nothing baseline, then report precision, recall and the precision-recall curve instead of accuracy. Google suggests that precision-recall curves may compare models better than ROC curves when classes are imbalanced (source).
How do you choose a metric for a regression model?
Ask how costly a large miss is. Use MAE when every unit of error costs the same, RMSE when large misses cost far more, and R² to see how much better you do than predicting the average.
What metric would you use for a model that outputs probabilities?
Use a metric that scores the probability itself. Log loss is the cross-entropy between predicted probabilities and outcomes, covered in cross-entropy loss, and the Brier score is the mean squared gap between probability and outcome. scikit-learn exposes both as neg_log_loss and neg_brier_score, and both need predict_proba (docs).
Common mistakes
- "The model has high accuracy, so it works." (wrong) Accuracy hides a missed rare class; the all-negative model above reaches 0.9901 (Google).1 (right) "Accuracy is high, but precision and recall on the positive class are the numbers I trust."
- "I use 0.5 as the threshold." (wrong) Google says 0.5 is a poor choice when classes are imbalanced or error costs differ (source). (right) "I set the threshold from the cost of a false alarm against a miss."
- "An AUC of 0.5 means 50% accuracy." (wrong) It means a random positive outranks a random negative half the time, which is chance (source). (right) "AUC 0.5 is no better than a coin flip at ranking."
- "F1 and AUC measure the same thing." (wrong) F1 score uses precision and recall at one threshold; AUC summarizes ranking across thresholds. (right) "F1 is a point on the sweep; AUC is the whole sweep."
- "Lower is better for every scorer." (wrong) scikit-learn lists error metrics under negated names such as
neg_mean_squared_error, so a bigger score is better (docs). (right) "I flip the sign back before I report the error."
Follow-up questions
How do you choose the classification threshold?
Compare the cost of a false alarm with the cost of a miss, sweep the threshold on validation data, and pick the point that minimizes total cost or meets a recall target. Then confirm it on a separate test set.
When do you use precision-recall instead of ROC-AUC?
Use the precision-recall curve when positives are rare, because its no-skill baseline is the positive share (0.0099 in the example) and it reacts to false alarms.1 ROC-AUC suits roughly balanced classes.
When is RMSE better than MAE?
RMSE is better when a large miss costs far more than several small ones, because squaring weights it heavily. In the toy data one miss of 10 raised RMSE from 1.2247 to 4.2426 but MAE only to 2.6667.1
What happens to precision if positives become ten times rarer?
Precision falls even if the model is unchanged. With the same hit and false-alarm rates, the example's 90 true positives become 9 against the same 30 false alarms, so precision drops from 0.75 to 0.2308 while recall stays 0.9.1
How do you know the metric still holds after launch?
Keep measuring it on fresh labeled data and watch the inputs for distribution shift. Before launch, a gap between training and held-out scores points to the trade-off in bias vs variance.
What metrics evaluate text generation or recommenders?
Text generation and recommenders use their own metrics, such as precision at k (the share of the top k recommended items that are relevant). See bilingual evaluation understudy (BLEU) against recall-oriented understudy for gisting evaluation (ROUGE) for text overlap, and collaborative filtering for how recommenders are built and judged.
- Choose metrics from the task, the cost of each error and the class balance.
- Count the confusion matrix first; accuracy can look excellent on a model that finds nothing.
- Precision, recall and F1 score are set at one threshold; AUC summarizes ranking across all of them.
- On rare positives, prefer the precision-recall curve to ROC-AUC.
- For regression, MAE treats all misses alike and RMSE punishes large ones.
Sources
- Classification: Accuracy, recall, precision, and related metrics (Machine Learning Crash Course) — Google for Developers, 26 Oct 2026
- Machine Learning Glossary: Metrics — Google for Developers, 26 Oct 2026
- Classification: ROC and AUC (Machine Learning Crash Course) — Google for Developers, 26 Oct 2026
- Thresholds and the confusion matrix (Machine Learning Crash Course) — Google for Developers, 26 Oct 2026
- Metrics and scoring: quantifying the quality of predictions (scikit-learn 1.9.1) — scikit-learn, 26 Oct 2026
- How to Calculate Precision, Recall, and F-Measure for Imbalanced Classification — MachineLearningMastery.com (secondary), 26 Oct 2026
- ROC Curves and Precision-Recall Curves for Imbalanced Classification — MachineLearningMastery.com (secondary), 26 Oct 2026
