What's the trade-off in offline metrics?

In this article

1. Before you start
2. Why interviewers ask this
3. Pre-deployment metrics from first principles
4. The Staff-level answer
5. The rubric
6. Three variants of this question
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step of the large language model (LLM) evaluation learning path.

Why interviewers ask this

A candidate who answers with a list of metric names has not understood the question, because the cost of choosing the wrong metric is shipping the wrong model. Interviewers ask about the trade-off to see whether you know what a pre-deployment score can and cannot tell you.

A weak answer says "offline metrics are fast but online metrics are real" and stops. A strong answer says which errors the metric hides, shows that the threshold moves one metric against another, and says why a live experiment is still needed. A Staff-level answer ties the metric to the cost of each kind of error and says how the labeled set is kept honest. The same question appears in design rounds for search, ranking and LLM products, in different words each time.

Pre-deployment metrics from first principles

Offline metric

An offline metric (also called a pre-deployment or batch metric on this page) is a score computed on a fixed labeled dataset before a system serves real users, such as precision, recall or ranking quality, which lets teams compare model versions quickly, cheaply and repeatably without exposing anyone to an untested change.

Decision threshold

A decision threshold is the cut-off on a model's score above which the system labels an item positive, and moving it changes which items are flagged, so it trades one kind of error for another.

In plain words

Think of a driving theory test. It is quick, everyone sits the same paper, and scores are easy to compare. But passing it does not prove you drive well in traffic. If the exam questions were written years ago, they may no longer match what drivers face today. The analogy breaks in one place: a theory test has one fixed pass mark, while a model's pre-deployment score also depends on a threshold you choose.

In one precise sentence, an offline metric measures how well a model agrees with labels collected in the past, and the trade-off is speed and repeatability against distance from real user behavior.

Step 1: A fixed labeled set turns quality into a number

A fixed labeled set turns quality into a number before release. A team collects examples, each with a label saying what the right answer is. It runs a model over every example and counts the outcomes in a confusion matrix: true positives, false positives, false negatives and true negatives.

Input: a model and the labeled set. Output: one or more scores. Because the set never changes, two runs are comparable, and a change can be tested in minutes without any user seeing it. Ranking tasks use metrics such as normalized discounted cumulative gain (nDCG) and mean average precision (MAP), which also need labels saying how relevant each result is (Manning, Raghavan and Schütze, ch. 8). You now know what the cheap side of the trade-off is: a number that anyone can recompute.

Step 2: Accuracy hides what precision and recall show

Accuracy can look high while recall is zero. Take 1,000 examples of which 50 are truly positive (illustrative). A model that answers "negative" every time is right on 950 of them, an accuracy of 0.95, yet it finds none of the 50 positives, a recall of 0.¹

Precision is the share of flagged items that were truly positive, and recall is the share of truly positive items the model found (Google Machine Learning Crash Course). The always-negative model flags nothing, so its precision is undefined.¹ Accuracy rewards the majority class, so on imbalanced data it says little. This is the payoff of the lead: the metric you report decides which failure you can see. You now know why one number is never enough, and the next step shows what happens when the model's scores meet a threshold.

Step 3: The threshold trades precision against recall

Moving the decision threshold trades precision for recall. A lower threshold flags more items, so it finds more true positives but also more false alarms. A higher threshold flags fewer items, so it is cleaner but misses more.

In the same illustrative set of 1,000 examples, a threshold of 0.3 flags 237 items and gives a precision of 0.1983 and a recall of 0.94.¹ A threshold of 0.5 flags 100 items and gives 0.4 and 0.8.¹ A threshold of 0.7 flags 35 items and gives 0.7143 and 0.5.¹ F-measure is a single number that is high only when both precision and recall are; its balanced form, F1, is the harmonic mean of the two.¹

The lowest cost is 160 at a threshold of 0.5, while accuracy peaks at 0.965 and F-measure at 0.5882 at a threshold of 0.7.¹ So no metric is "right" until you say what each error costs.

Text description of the diagram

• Precision rises from 0.1983 to 0.7143 as the threshold rises from 0.3 to 0.7.
• Recall falls from 0.94 to 0.5 over the same range.

Precision rises from 0.1983 to 0.7143 as the threshold rises from 0.3 to 0.7, and recall falls from 0.94 to 0.5 over the same range.¹

Step 4: A pre-deployment score is a proxy for live behavior

A proxy gap separates live behavior from the pre-deployment score, because the labeled set is a snapshot that can differ from live traffic in three ways. First, selection bias: labels and interactions were collected under the current model, so the set favors candidates that resemble it; counterfactual evaluation such as inverse propensity weighting is the standard partial fix. Second, prevalence shift: precision depends on the share of positives. At the 0.5 threshold, with recall and false-alarm rate held fixed, precision is 0.2452 at 25 live positives per 1,000, 0.4 at 50 and 0.5846 at 100, while recall stays 0.8.¹ Third, a fixed set cannot show feedback loops, novelty effects or delayed outcomes.

So check over past launches whether score gains predicted A/B results, and treat a higher score as unproven until a live test supports it. A common practice is a curated set of regression cases; see golden sets and regression testing for LLM apps, and offline versus online evaluation for why both are needed.

1 · Labeled set2 · Accuracy trap3 · Threshold4 · Proxy gapReplay

Step-by-step description

1. A fixed set turns quality into a number.
2. Accuracy can hide missed positives.
3. The threshold trades precision for recall.
4. Live behavior can disagree with the score.

• A fixed labeled set turns quality into a number before release.
• Accuracy can look high while recall is zero.
• Moving the decision threshold trades precision for recall.
• A proxy gap separates live behavior from the pre-deployment score.

The Staff-level answer

What makes it Staff-level: it ties the metric to the cost of each error and names the proxy gap and the live test, instead of listing metric names.

The rubric

Three variants of this question

Why can a higher pre-deployment score still lose an A/B test?

The score measures agreement with past labels, not the live outcome. Traffic may have shifted, the labels may be stale, or the metric may reward something users do not value, so only the live test settles it.

How do you choose the decision threshold?

Put a cost on each kind of error and choose the threshold with the lowest total cost, then confirm it live. In the worked example the lowest cost is 160 at a threshold of 0.5, even though accuracy peaks at 0.7.¹

Which metric do you use when positives are rare?

Use precision and recall, or a combined score such as F-measure, rather than accuracy. In the example an always-negative model reaches an accuracy of 0.95 with a recall of 0.¹

Common mistakes

• "The model has ninety-five percent accuracy, so it works." (wrong) On imbalanced data a model that finds nothing can score that. (right) "I report precision and recall too, because accuracy hides missed positives."
• "I pick the threshold that maximizes F-measure." (wrong) That assumes both errors cost the same. (right) "I choose the threshold from the cost of each error."
• "The new model scores higher, so I ship it." (wrong) The score is a proxy for live behavior. (right) "I confirm the gain with a live A/B test where traffic allows, or a staged rollout with monitoring."
• "Offline metrics are useless because online tests exist." (wrong) Live tests are slow and expose users. (right) "I use batch metrics to filter candidates and the live test to decide."

Follow-up questions

When do you need an online experiment instead of more batch metrics?

Use one when the decision depends on user behavior, such as clicks, retention or task completion, or before a launch where a wrong choice is costly. Batch metrics narrow the candidates; the experiment picks among the survivors. See offline versus online evaluation.

Why do offline and online metrics disagree?

Labels come from past traffic under the old model, so selection bias favors similar candidates, and a different live positive rate moves precision. Feedback loops, novelty effects and delayed outcomes appear only live. Checking whether past score gains predicted A/B results shows how far to trust the proxy.

How do ranking metrics such as nDCG fit in?

Ranking metrics score the order of results rather than a single label, so they need graded relevance judgments (Manning, Raghavan and Schütze). They carry the same trade-off: cheap and repeatable, but only as good as the judgments behind them.

How do you keep a labeled set from going stale?

Refresh it with recent real cases, add every confirmed failure as a regression case, and compare its mix with live traffic. A larger platform can run these checks on every change; see designing an internal eval platform.

How does this apply to LLM outputs?

Open-ended outputs have no single right label, so teams use reference-based scores, rubric checks or a model acting as a grader. Each is a proxy with its own bias; read LLM-as-a-judge and the Ragas framework for two common approaches.

Sources

1. Classification: Accuracy, precision, recall, and related metrics (Machine Learning Crash Course) — Google for Developers, 26 Oct 2026
2. Evaluation in information retrieval (Introduction to Information Retrieval, ch. 8) — Manning, Raghavan and Schütze, Stanford NLP, 1 Jan 2008