In this article
Before you start
This page belongs to the LLM evaluation learning path; the neighbouring guide to designing an eval for a single feature covers what one test set should contain.
Why interviewers ask this
Candidates report this question, in several wordings, in system design rounds for roles at AI labs and data-labelling companies. It tests whether you treat evaluation as engineering infrastructure rather than a notebook someone runs before a launch.
A weak answer lists tools: "we run a benchmark and look at the score." A strong answer names the components (dataset, harness, graders, results store, gate), says where each runs, and explains what happens when two runs disagree by a small amount. The consequence of skipping that last part is real: a gate that blocks on noise stops teams shipping, and a gate that passes on noise lets regressions through. A Staff-level answer adds cost control, grader trust, and a loop that turns production failures into new test cases.
Eval platform from first principles
- Eval gate
- An eval gate is an automated check in the change pipeline that runs a test set against a candidate prompt, model or code change, compares the result with the current production version, and returns pass, hold or fail before the change can merge or deploy.
- Paired difference
- A paired difference is the per-case difference between a candidate's score and the baseline's score on the same test case, averaged over all cases; pairing removes the case-to-case difficulty that adds noise to a comparison of two separate averages.
In plain words
Think of a restaurant that changes a recipe. Before the new recipe goes on the menu, a taster cooks both versions. The dishes are the same fixed set. The taster compares the two versions dish by dish. The analogy breaks in one place. A language model gives a different answer on each run. So the tasting must be repeated. The difference is judged against the natural wobble, not by one bite.
In one precise sentence, the platform turns "did this change make the application worse?" into a repeatable experiment. Its decision rule is fixed in advance.
Step 1: Freeze a test set, run the application on it and grade every case
The platform starts with a versioned dataset: each case holds an input and either a reference answer or a check, and every change to the set gets a new version number. A score is therefore always tied to the set that produced it. In short, a versioned test set is run and graded on every candidate. A harness is the program that runs the application, with the candidate prompt, model or code, on every case and records each output. A grader is whatever turns an output into a score.
Use graders in this order: code checks first (format, exact fields, forbidden strings), a model judge for open-ended quality, humans for calibration and disputes. Judges carry known biases: Zheng et al. (2023) examine position, verbosity and self-enhancement bias. They also report that a strong judge reaches over 80% agreement with human preferences, the level at which humans agree with each other. Input: a candidate. Output: one score per case.
Step 2: Compare each change with the baseline using an error bar
The gate asks whether the difference between the candidate and the baseline, on the same cases, is larger than noise, measured with an error bar. Evaluations are experiments, and Miller (2024) shows how to analyse them: with formulas for measuring differences between two models and for planning an evaluation experiment, and with recommendations for reporting results that minimise statistical noise.
The usual common error is to print two pass rates, 0.90 and 0.88 (the illustrative figures of the worked example below), and call the lower one worse. A confidence interval is a range, computed from the data, that would contain the true value in most repeats of the same experiment; here it surrounds the paired difference. If the interval includes zero, the data cannot separate "worse" from "the same". Pairing matters because the same hard cases hurt both versions, so the interval around the paired difference is narrower than one built from two separate averages. The worked example below puts numbers on this. You now know what the gate computes; the next step is where it runs.
Step 3: Turn the comparison into a pass, hold or fail rule in CI
The gate runs as a required check in continuous integration, so every pull request that touches a prompt, a model identifier, a retrieval setting or tool code triggers it. Write the rule before you see results. A workable rule has three outcomes: fail when the interval lies wholly below zero, pass when the lower bound clears an agreed tolerance, hold in between and ask for a larger run or a human decision.
Every run writes to a results store: the dataset version, the candidate, the baseline, per-case outputs and scores, grader versions and the decision. That store is what makes a failure debuggable, because an engineer can open the cases that flipped from pass to fail. Store the per-case flips, not only the summary.
Report the paired difference per tagged slice, such as a language, a customer tier or a task type, because an average can hold while one segment collapses. Gate on a named list of critical slices, and treat a slice with few cases as hold, since its error bar is wide. You now have a gate that blocks a bad change; keeping the graders honest is the next step.
Step 4: Keep the graders honest
A gate is only as trustworthy as its graders. Calibrate every judge against a small human-labelled sample and track agreement over time; the process is described in LLM-as-a-judge. In the illustration used here, a judge that agrees with human labels on 86 of 100 cases has an agreement of 0.86.1 Raw agreement can overstate judge quality when one label dominates, so also report a chance-corrected measure such as Cohen's kappa. When the judge model, its prompt or its rubric changes, treat that as a change to the platform and re-run the calibration before it can gate anything.
Keep code checks for what code can decide, such as valid structure or a required citation, because they are free, repeatable and have no judge bias. For a retrieval-augmented generation (RAG) application, add component metrics, which Es et al. (2023) define; the Ragas page covers them. Safety checks that must never regress, such as those from LLM guardrails, belong in a separate hard-fail suite whose threshold is zero regressions, not a statistical tolerance.
Step 5: Feed production failures back into the test set
A fixed set ages: users find inputs the authors never imagined, so production failures feed the next test set. Close the loop by sampling logged production conversations, especially those a user flagged or a guardrail caught, having a person label them and adding them to the next dataset version. The score on a new version is not comparable with the old one, so the platform re-runs the baseline on every new version.
Two rules keep this honest. Cases added from a regression are tagged, so you can see whether a fix generalises. A held-out slice that engineers never look at guards against tuning prompts to the visible cases, which is the same failure that makes public benchmarks less informative over time; see benchmarks and perplexity.
Step-by-step description
- Run every case and score it.
- Paired difference with error bar.
- Pass, hold or fail.
- Calibrate judges on human labels.
- Failures join the next set.
- A versioned test set is run and graded.
- The candidate is compared with the baseline using an error bar.
- Production failures feed the next test set.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
Take a suite of 500 cases (illustrative). The baseline passes 450 and the candidate passes 440, a pass rate of 0.90 against 0.88.1 Pairing the cases shows where the 10 cases went: 425 pass in both versions, 25 pass only in the baseline, 15 pass only in the candidate and 35 fail in both.1
The paired difference is minus 2.00 points, with a standard error of 1.26 points, so the 95% confidence interval runs from minus 4.48 to plus 0.48 points.1 A comparison that ignores pairing gives a standard error of 1.98 points and an interval from minus 5.88 to plus 1.88 points, wider by a large margin.1
| Quantity | Value (computed, illustrative inputs) |
|---|---|
| Paired difference | -2.00 points |
| Standard error, paired | 1.26 points |
| 95% interval, paired | -4.48 to +0.48 points |
| 95% interval, unpaired | -5.88 to +1.88 points |
| Rule "block if proven worse" | pass |
| Rule "pass only if within 3 points" | hold |
| Cases for a 1-point half-width | 3,065 |
The rule matters as much as the data. Blocking only when the whole interval is below zero passes this change, though a drop of 4.48 points is still plausible. Requiring the lower bound to stay above minus 3 points holds it. Six times the cases cut the noise by about 2.5 times: the half-width falls from 2.48 points at 500 cases to 1.01 points at 3,000.1 Solving the half-width formula for one point gives the 3,065 cases in the table.1 For paired yes-or-no outcomes like these, McNemar's test, which looks only at the cases where the two versions disagree (25 and 15 here, illustrative), is the standard exact alternative to the normal interval.
Text description of the diagram
- The half-width is 2.48 points at 500 cases.
- The half-width is 1.01 points at 3,000 cases.
Cost is the other number to size. At 3 trials per case, one application call and one judge call per case-trial, a 500-case run makes 3,000 calls and, at 3,100 tokens per case-trial (illustrative), about 4,650,000 tokens.1 With 40 runs a day that is 186,000,000 tokens, or 46,500,000 if caching means only a quarter of the cases re-run.1 Multiply by your provider's current price to get money.
Trade-offs and edge cases
The gate trades speed for confidence. A small suite on every commit is cheap but noisy; a large suite is sensitive but slow. A common design runs a small smoke suite on every push and the full suite before merge or on a schedule, with the full suite required for model upgrades. Accept that the platform will sometimes say hold, and decide who resolves it.
Model version upgrades are the hard case, because the application code did not change but the behaviour did. Run the full suite with the new model as the candidate and the pinned old model as the baseline, read the per-case flips, then release gradually while comparing production metrics. Pre-deployment tests and production monitoring answer different questions, and you need both.
Other edge cases: sampling randomness means one run can differ from the next, so use repeated trials or fixed seeds where the provider allows; a judge that favours longer answers can reward a verbose prompt change, so add length-controlled checks; many tests per change raise the chance of a false alarm, so name the primary metric in advance; and data in the test set may include personal information, so apply the same access rules as production data.
Text description of the diagram
- Blocking only when the whole interval is below zero passes this change.
- Requiring the lower bound to stay above minus 3 points holds it.
How the platform scales across teams
An internal eval platform serves many teams by sharing infrastructure and leaving judgement to each team. Runs enter a job queue and parallel workers pick them up, with a rate limiter per model provider so that one team's large run cannot starve another's. An output cache is keyed on a hash of the prompt, the model identifier, the sampling parameters and the input, so an unchanged case is never paid for twice, and a change to any of those four fields produces a new key.
The results store holds one row per case, trial and run, with the dataset version, candidate, baseline, scores and grader versions, and a dashboard compares runs side by side and lists the flipped cases. Each team owns its dataset, rubric, slice tags and tolerance as versioned configuration. The platform owns the harness, the store, the gate logic and the judge infrastructure, and applies access rules so that one team's data stays visible only to that team.
How to answer it in an interview
Design an internal eval platform that gates every change.
I would build a shared service with four parts: a versioned test set per product, a harness that runs the changed prompt or model on every case, graders that are code checks first and a calibrated model judge second, and a results store. In continuous integration the gate compares the candidate with the production baseline on the same cases and decides on the paired difference with its error bar, not on raw scores. Small suites give wide bars, so I allow a hold outcome. Production failures are labelled and added back to the set.
The spoken structure: name the components, say what the gate compares, say why the error bar matters, then close with the feedback loop.
Common mistakes
- "The candidate scored 0.88 and the baseline 0.90, so block it." (wrong) The gap may be noise. (right) "I compare the paired difference with its confidence interval: fail if the whole interval is below zero, pass if the lower bound clears the agreed tolerance, otherwise hold."
- "The judge model is the ground truth." (wrong) Judges have position, verbosity and self-enhancement biases. (right) "I calibrate the judge against human labels and re-check when it changes."
- "We run the evals before a launch." (wrong) Launch-time checks miss regressions from each small change. (right) "The gate runs in continuous integration on every prompt, model or tool change."
- "We add new cases whenever we like." (wrong) Scores from different dataset versions are not comparable. (right) "Every dataset version gets a baseline re-run before it gates anything."
Follow-up questions
How do you build an evaluation harness that runs in CI to gate every model change?
Make the harness one command that takes a candidate and a dataset version, runs every case with a timeout and retries, writes per-case results to the store and exits with a status the CI system reads as pass, hold or fail. Cache outputs for unchanged cases to control cost.
How do you handle model version upgrades without breaking production?
Pin the model version, run the full suite with the new version as candidate, review the cases that flipped, and release gradually while production metrics are compared with the old version. Keep a rollback path.
How would you detect and handle LLM output regressions?
Detect them with the gate before release and with production sampling after it. Handle them by reading the flipped cases, finding the shared cause, fixing it, and adding the cases to the test set so the regression cannot return unnoticed.
A prompt tweak fixed one case and broke ten others. How do you regression-test an LLM app in CI?
Keep a golden set of past failures and typical traffic, run it on every prompt change, and look at per-case flips as well as the average. A change that fixes one case and breaks ten shows up as ten red flips.
How would you design an internal evaluation platform that lets teams measure and compare LLM features reliably?
Separate what is shared from what is per-team. Share the harness, the store, the gate logic and the judge infrastructure; let each team own its dataset, rubric and tolerance. Compare candidate and baseline per slice, and give teams a dashboard that shows runs side by side with the flipped cases. The eval design guide covers the per-team part.
- An eval gate runs a versioned test set on every change and compares the candidate with the baseline.
- Decide on the paired difference and its confidence interval, not on two raw scores.
- Write the pass, hold and fail rule before seeing results, and let small suites say hold.
- Calibrate judges against human labels and re-check whenever the judge changes.
- Feed labelled production failures back into the next dataset version and re-run the baseline.
Sources
- Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations (arXiv 2411.00640) — arXiv (Miller), 1 Nov 2024
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (arXiv 2306.05685) — arXiv (Zheng et al.), 9 Jun 2023
- Ragas: Automated Evaluation of Retrieval Augmented Generation (arXiv 2309.15217) — arXiv (Es et al.), 26 Sep 2023
