Offline vs online evaluation: why do you need both?

In this article

1. Before you start
2. Why interviewers ask this
3. What each one is
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

Why interviewers ask this

The pair is a favourite because it separates people who have only trained models from people who have shipped them. A weak answer says "evaluate on a test set, then A/B test" and stops. A strong answer explains why the held-out score can mislead, how many cases or users each method needs, what an LLM judge can and cannot replace, and how you decide what to run online.

The question is reported by candidates in machine learning (ML) and AI engineering rounds, often as "your new model wins on the test set but loses in the A/B test, why"; it sits beside the other LLM evaluation interview questions.

What each one is

Offline evaluation

Offline evaluation, also called pre-deployment or batch evaluation, is the scoring of a model on a fixed, labelled or rubric-graded set of cases before any user sees it, so that different model versions can be compared on identical inputs.

Online evaluation

Online evaluation is the measurement of a model's effect on real users in production, usually by randomly splitting traffic between versions and comparing a behaviour metric such as acceptance rate or retention.

Metric skew

Metric skew is a gap between how a model scores on a pre-deployment metric and how it performs on the online metric that the business cares about.

Text description of the diagram

• Pre-deployment evaluation takes minutes to hours.
• Online evaluation takes days to weeks.

Offline evaluation in plain words

Think of a flight simulator. The pilot flies the same scripted emergencies every time, so two pilots can be compared fairly, and nobody is at risk. A simulator session also costs little, so a pilot can fly a hundred of them in a week. The analogy breaks in one place: the simulator contains only the emergencies someone thought to script.

A test set is the same: repeatable, cheap, safe, and limited to the cases its author collected. Examples are accuracy on a labelled set, or a judge model's score on a fixed prompt list. The Holistic Evaluation of Language Models (HELM) framework is a large instance, measuring 7 metrics across 16 core scenarios for 30 language models (Liang et al., 2022).

Online evaluation in plain words

Online evaluation is the real flight with passengers. Nothing about it is scripted, and the result is the one that counts. The analogy breaks in one place: a real flight is rarely repeated under the same conditions, while an online test must hold everything else equal to be read. Users bring requests the test set never contained, and what they do is the ground truth: they accept, edit, retry or leave.

The standard design is a randomised controlled experiment, an A/B test, where traffic is split between two versions. It is slower and riskier than a batch test. Chatbot Arena is a public example of live human-preference evaluation: users compare two anonymous model answers, and as of March 2024 the platform had amassed over 240,000 votes (Chiang et al., 2024).

Worked example

The cost of each method shows up in sample sizes. A test set of N cases measuring an accuracy of 82% has a 95% confidence interval. A confidence interval is a range computed from a sample; a 95% interval comes from a procedure whose ranges, recomputed over many samples, contain the true value about 95 times in 100. With 100 cases the interval runs from 0.7333 to 0.883, about 15 points wide; with 500 cases from 0.7839 to 0.8512, 6.7 points; with 2,000 cases from 0.8026 to 0.8362, 3.4 points (illustrative accuracy).¹ A second model scoring 84% on the same 500 cases has an interval of 0.8053 to 0.8695.

The intervals overlap, but overlap is the wrong test here: both models answered the same cases. McNemar's test is a paired test that compares two models on the same cases by counting only the cases where one model is right and the other wrong. If the new model fixes 15 cases and breaks 5, the exact p-value is 0.041, so the gap is significant at the 5% level; if it fixes 30 and breaks 20, p is 0.20, so these 500 cases cannot tell the gap apart from noise. Treating the two scores as independent samples gives p = 0.40, which is the right calculation only for different cases.¹

Text description of the diagram

• The interval is about 15 points wide at 100 cases.
• The interval is 6.7 points wide at 500 cases and 3.4 at 2,000.

An online test pays for its realism in traffic. Suppose 10% of users accept a suggestion and you want to detect a rise to 11%, at a 5% significance level and 80% power, using the standard two-proportion formula. You need 14,751 people per arm, 29,502 in total, which is 15 days at an illustrative 2,000 eligible users a day. A target lift of 2 points needs only 3,841 per arm, and a lift of half a point needs 57,763.¹ The effect size sets the bill.

How often does an idea a team believes in move the online metric? Kohavi and colleagues at Bing reported that, when ideas are tested in controlled experiments, "less than a third move the metrics they were designed to improve", and that the share is lower in a heavily tuned product (Bing blog, 2013). In 2013, over 250 experiments ran on a typical day there.

Trade-offs and edge cases

Use pre-deployment evaluation when you are iterating many versions, when a wrong release is costly, or when the check is cheap, for example regression tests on a golden set. Use online evaluation when the metric depends on user behaviour that a test set cannot contain: whether a suggestion is accepted, whether a conversation continues, whether retention moves. In practice the two form a funnel.

The batch test is a gate that removes bad candidates in minutes to hours, and the online test confirms the survivors. The online result decides the release. Google's guidance on live launches says when the call is easy: "The only easy launch decisions are when all metrics get better (or at least do not get worse)" (Google, 2025). When metrics trade off, the team has to weigh them.

1 · Change2 · Batch test3 · Gate4 · Online test5 · DecisionReplay

Step-by-step description

1. A new prompt or model.
2. Score a fixed set in minutes to hours.
3. Drop regressions.
4. Split real traffic.
5. Ship, stop or weigh the trade-off.

• The batch test is a gate that removes bad candidates in minutes to hours.
• The online test confirms the survivors.

Metric skew has several causes. The test set may not match live traffic, which is distribution shift. The test metric may not be the business metric: a higher answer-accuracy score does not ensure higher retention. Google's guidance expects some drop from held-out data to data from the next day, and says a large drop may mean some features are time-sensitive.

A gap between next-day data and live serving is different: it "probably indicates an engineering error" (Google, 2025). A third cause is training-serving skew: a difference between a model's performance in training and in serving, caused for example by a feature computed one way in the training pipeline and another way at serving time, or by a feedback loop between the model and the data it later learns from (Google, 2025).

LLM judges extend the batch side, but they do not replace online evaluation. Zheng et al. report that a strong judge model reached over 80% agreement with human preferences, similar to the agreement between humans, and also document position, verbosity and self-enhancement biases (Zheng et al., 2023). A judge agreeing with people on chat quality says nothing about whether your users keep using the product.

Online tests have their own limits: interference between users, novelty effects, and low traffic. Interleaving, which shows results from two rankers inside one list, is a common sensitivity trick in search and recommender systems. For retrieval pipelines the batch metrics are in retrieval-augmented generation (RAG) evaluation.

How to answer it in an interview

The spoken structure is: define each in one clause, give the weakness of each, then describe the funnel.

Common mistakes

1. Treating the test set as ground truth. (wrong) "It scores 84%, so it will win." (right) "It scores 84% against 82% on the same set; a paired test on the cases where one model is right and the other wrong says whether the gap is larger than noise, and online behaviour decides the release."
2. Skipping the sample-size check. (wrong) "Run the A/B test for a week." (right) "Compute users per arm from the baseline rate and the lift you care about."
3. Replacing online tests with a judge model. (wrong) "The judge prefers B, so ship B." (right) "A judge approximates human preference; it does not measure retention."
4. Peeking at the A/B test daily and stopping early. (wrong) "It was significant on day three, stop." (right) "Fix the duration or use a sequential method, or false positives rise."
5. Using one metric. (wrong) "Accuracy went up, done." (right) "Check guardrail metrics such as latency, cost and complaints as well."

Follow-up questions

Why do offline metrics and online metrics disagree?

Offline and online metrics disagree because the test set holds only the cases its authors thought of, while live traffic holds cases they did not. The metric can also differ from the goal, and the gap between them is metric skew. A third cause is training-serving skew, where a feature is computed one way in training and another way at serving, or a feedback loop distorts the data. See LLM observability for the logging that finds the missing cases, and why offline metrics run higher than online for the most common direction of the gap.

Offline up, online flat: what do you check first?

First check the offline gain itself: did it pass a paired test? A gain at p = 0.20, like the 30-fixed, 20-broken case in the worked example, is not a gain. Then check the experiment: the planned sample size was reached, users were randomly assigned, and the metric is the one fixed in advance. Next compare the test set with live traffic, then look for training-serving skew, and last ask whether the offline metric measures what the online metric rewards.

How do you A/B test prompts and models?

Randomly assign users, or sessions, to a variant, keep everything else equal, pick one primary metric and guardrails in advance, compute the sample size, and run for the planned duration. The same structure works for a prompt change as for a model change.

Can an LLM judge replace online evaluation?

No. A judge scales the batch side and agreed with human preferences more than 80% of the time in one study, but it measures a proxy (Zheng et al., 2023).

How many users does an A/B test need?

It depends on the baseline rate and the smallest lift worth detecting. With a baseline of 10% and a target of 11%, the formula gives 14,751 per arm at 5% significance and 80% power, and halving the lift roughly quadruples the sample.

What goes in a good regression set?

Real failures from production, edge cases the team has debated, and a stable core that never changes so scores stay comparable. Measures such as the confusion matrix help break the score down by class.

Sources

1. Large Scale Experimentation at Bing — Microsoft Bing blog (summarising Kohavi et al., KDD 2013), 8 Aug 2013
2. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — arXiv (Zheng et al.), NeurIPS 2023, v4 revised 2023-12-24, 9 Jun 2023
3. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference — arXiv (Chiang et al.), 7 Mar 2024
4. Holistic Evaluation of Language Models — arXiv (Liang et al.), v2 revised 2023-10-01, 16 Nov 2022
5. Rules of Machine Learning: Best Practices for ML Engineering — Google for Developers (page last updated 2025-08-25), 25 Aug 2025