LLM Evaluation

What metrics would you track to evaluate the performance of your ML pipeline?

By Interview Clozo · Published 27 October 2026 · Last verified 26 October 2026 · 12 min read

The metrics you would track to evaluate the performance of your ML pipeline fall into four groups: data (schema, missing values, drift), model (precision, recall, calibration), system (latency, errors, cost) and business metrics. A machine learning (ML) pipeline is the chain of data, features, model and serving code that turns raw inputs into predictions. The costly error is quoting accuracy alone, which can look excellent while the model misses one real case in three.

In this article

Before you start
Why interviewers ask this
ML pipeline metrics from first principles
The Staff-level answer
The rubric
Three variants of this question
Common mistakes
Follow-up questions
Sources

Before you start

This question belongs to the large language model (LLM) evaluation learning path; the ideas below apply to classic models and to language-model systems alike. If you want the model underneath most language-model systems first, read how transformers work.

Why interviewers ask this

The question separates candidates who know one metric from candidates who run a system. A model that is correct in a notebook can still fail in production through bad inputs, slow responses or a result nobody values, so a good answer covers more than the model.

A weak answer names accuracy and stops. A strong answer groups metrics by layer, picks the model metric from the cost of each kind of error, and says how each group is watched after launch. A Staff-level answer adds the monitoring side: which differences between training and serving to check, how to alert on trends, and how a new model is rolled out and compared with the old one. The question is open-ended on purpose, so the interviewer is listening for structure.

ML pipeline metrics from first principles

ML pipeline: An ML pipeline is the chain of data collection, feature computation, model training and model serving that turns raw inputs into predictions, so a fault in any stage can lower the quality of the final result even when the model itself is sound.

In plain words

Think of a restaurant. The ingredients are the data. The chef is the model. The kitchen's speed and uptime are the system. The customers' happiness is the business result. A great chef cannot save rotten ingredients or a broken oven. You need a check on each.

The analogy breaks in one place. A chef sees the plate go out. A model often learns the true outcome weeks later, or never.

In one precise sentence: you judge an ML pipeline by model quality, input health, serving health and delivered value, and you track each as a time series, not a one-off number.

Each layer catches a failure the other three miss, so track all four.CC BY 4.0 — source: What metrics would you track to evaluate the performance of your ML pipeline?

Step-by-step description

Precision and recall judge the predictions.
Missing values, drift and skew judge the inputs.
Latency, errors and cost judge the serving path.
Outcome metrics judge the value delivered.

Model metrics such as precision and recall judge the quality of the predictions.
Data metrics such as missing values, drift and skew judge the inputs.
System metrics such as latency, errors and cost judge the serving path.
Business metrics judge whether the pipeline helps users and the company.

Step 1: Model metrics and why accuracy misleads on imbalanced data

Model metrics measure how right the predictions are, and the right metric depends on which mistake costs more. Precision is the share of flagged cases that were truly positive. Recall is the share of truly positive cases the model found. The F1 score is the harmonic mean of precision and recall, so it is high only when both are. Calibration is how well predicted probabilities match real frequencies: of cases scored at 0.8 (illustrative), about 80% should be positive.

Take an illustrative batch of 1,000 cases with a 12% positive rate: 80 true positives, 20 false positives, 40 false negatives and 860 true negatives.¹ Accuracy is 0.94, but precision is 0.80, recall is 0.667 and F1 is 0.727.¹ A model that always answers negative scores 0.88 accuracy with zero recall.¹ Accuracy rewards the majority class; recall shows the 40 positives missed.

On imbalanced data accuracy (0.94) looks strongest while recall (0.667) shows that one positive case in three is missed.
Metric	Value in the illustrative batch
Accuracy	0.94
Precision	0.80
Recall	0.667
F1 score	0.727
Balanced accuracy (mean of recall and specificity, the share of true negatives correctly cleared: 860 of 880 = 0.977)¹	0.822
Accuracy of "always negative"	0.88

On imbalanced data accuracy is the highest bar and recall the lowest, so quote both.CC BY 4.0 — source: What metrics would you track to evaluate the performance of your ML pipeline?

Text description of the diagram

Accuracy is 0.94, but precision is 0.80, recall is 0.667 and F1 is 0.727.
A model that always answers negative scores 0.88 accuracy with zero recall.

Choose recall when a miss is costly (fraud, disease) and precision when a false alarm is costly (blocking a good customer). When scores drive decisions, add calibration.

Beyond one threshold: ROC-AUC, PR-AUC and regression errors

Precision, recall and F1 all depend on a decision threshold, and two measures avoid that choice. The area under the receiver operating characteristic curve (receiver operating characteristic (ROC)-area under the curve (AUC)) measures ranking quality across all thresholds, as the scikit-learn model-evaluation documentation defines it. The area under the precision-recall (PR) curve, called PR-AUC, is more informative than ROC-AUC when positives are rare (Saito and Rehmsmeier, 2015).

For a regression model, use the mean absolute error (MAE) or the root mean squared error (RMSE); scikit-learn describes both in the unit of the target. Report these beside the thresholded numbers, not instead of them, because a deployed system still has to choose one operating point from the cost of a missed positive against the cost of a false alarm.

Step 2: Data metrics catch problems before the model does

Data metrics measure whether the inputs look like what the model was built for, and they often warn earlier than model metrics because labels arrive late. Track the schema (types and columns), the missing-value rate per feature, value ranges, and each feature's distribution against a reference such as the training set. Google's Rules of Machine Learning say the best solution to skew is to monitor it explicitly so system and data changes do not slip in unnoticed. The ML Test Score paper (Breck et al., 2017) lists tests that data invariants hold and that training and serving are not skewed.

Measuring drift with the population stability index

The population stability index (PSI) is one common drift number: split a feature into bins that each hold an equal expected share (quantile bins), then sum (actual share minus expected share) times the natural log of their ratio.

PSI = \sum_{i} (a_{i} - e_{i}) \ln \frac{a_{i}}{e_{i}}

Here $a_{i}$ is the actual share and $e_{i}$ the expected share of bin $i$ . An empty bin needs a small floor value, because the logarithm of zero is undefined. With four bins that held a quarter of the data each in training and now hold 15%, 20%, 30% and 35% (illustrative), the PSI is about 0.105.¹ No universal alert level exists; set the threshold from your own history.

Step 3: System metrics keep the pipeline usable

System metrics measure whether the pipeline answers quickly, reliably and affordably, because a correct answer that arrives late is still a failure. Track latency as percentiles, since the average hides the slowest requests that users notice. Add throughput (requests handled per second), error and timeout rate, and cost per request. The training and data side has its own health numbers: job success and failure rate, data freshness (how old the newest training or feature data is) and training time.

The ML Test Score paper adds checks specific to machine learning: that models are not too stale, that they are numerically stable, that computing performance has not regressed, and that dependency changes trigger a notification (Breck et al., 2017). Staleness matters because a model trained on old data drifts away from current behaviour.

Per-stage numbers help you locate faults: log latency and error rate for feature lookup, model call and post-processing separately, so a slow response points to a stage. You now have the three technical layers; the last one decides whether they matter.

Step 4: Business and online metrics prove the pipeline is worth running

Business metrics judge whether the pipeline helps users and the company. They measure whether the pipeline delivers the outcome it was built for, such as conversions, fraud losses avoided, time saved or user retention. A model metric is a proxy for that outcome, and the proxy can drift away from it, so the final check is a live one.

The usual method is a controlled online experiment: send a small share of traffic to the new model while the old one serves the rest, and compare outcome metrics. The ML Test Score paper describes a mitigation in which new models see only a small fraction of traffic, gradually increased as the new model is observed to behave sanely (Breck et al., 2017). When true labels arrive late, track proxy signals such as user corrections or complaints in the meantime. The trade between the two settings is covered in pre-deployment versus live evaluation.

Knowing it is half the job. Saying it is the other half.

Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.