In this article
Before you start
This page is one part of the LLM evaluation learning path.
Why interviewers ask this
A prompt change that fixes one case and silently breaks ten others reaches users unless something catches it. Candidates report this scenario in interviews at AI labs and large tech companies, usually as "how do you regression-test an LLM app in continuous integration (CI)?" The question separates people who change prompts by feel from people who measure.
A weak answer says "I try a few prompts and see." OpenAI's docs name that habit directly, listing vibe-based evals as an anti-pattern. A strong answer names a set of real cases, a grader for each, a saved baseline and a gate that blocks the merge. A Staff-level answer adds what makes the gate trustworthy: a comparison that survives noise, a held-out slice, a judge checked against humans, and a set that grows when production finds a new failure.
Golden sets and regression testing from first principles
- Golden set
- A golden set is a fixed, versioned collection of test inputs, each paired with a trusted reference answer or a written pass rule, that an LLM app is run against to measure quality. It is built from your own traffic and edge cases, so a score on it means something about your product, unlike a public benchmark.
- Regression test
- A regression test is a rerun of a fixed set of cases after a change that checks whether anything that used to pass now fails. For an LLM app the change can be a prompt edit, a model upgrade, a retrieval tweak or a new tool, and the rerun happens automatically before the change ships.
- Frozen baseline
- A frozen baseline is the saved per-case results of the currently shipped version on the golden set, stored with the version of the set, that every candidate change is compared against. Freezing it makes the comparison reproducible and cheap, because the baseline is stored and does not have to be rerun for every candidate.
In plain words
Think of a teacher with an answer key and a fixed exam. After changing the lesson plan, the teacher reruns the same exam. Any question the class used to get right and now gets wrong is a regression. The analogy breaks in two places. A language model can answer the same question differently on each run. And a good golden set keeps growing as real failures appear, while a school exam stays fixed.
In one precise sentence, regression testing for an LLM app compares per-case results of a candidate against a frozen baseline on a golden set, and blocks the change when the comparison shows a real drop.
Step 1: Collect cases that mirror real traffic
To build a golden set from scratch, start with inputs. OpenAI's guidance is to mix production data, hard-coded correct answers written by domain experts, and historical data from logs. Anthropic's docs say to design evals that mirror the real task distribution and to include edge cases, such as irrelevant or missing input, overly long input and ambiguous cases where even humans would disagree.
Input: logs, expert answers and known failures. Output: a list of cases, each with an input, a reference answer or a pass rule, and a tag such as "long input" or "refusal". With no labels yet, write pass rules ("must cite the refund policy", "must not reveal the system prompt") instead of full answers. You now have the test material; the next step decides how each output is judged.
Step 2: Grade each output with the cheapest reliable grader
A grader turns one output into pass or fail. Anthropic lists the ladder: code-based grading (exact match or checking that a key phrase appears) is the fastest and most reliable but lacks nuance; LLM-based grading is fast and flexible, and should be tested for reliability before you scale it; human grading is the most flexible and highest quality but slow and expensive.
Use code graders wherever the answer can be checked by rule, and an LLM judge for open-ended text once you have compared it with human labels. OpenAI warns against ignoring human feedback: calibrate automated metrics against human evals. Make criteria measurable: Anthropic's example is fewer than 0.1% of 10,000 outputs flagged for toxicity, which means fewer than 10 flagged outputs.1
Step 3: Compare case by case, not total by total
A candidate change is judged against the frozen baseline one case at a time. Each case ends in one of four states: passes in both versions, fails in both, fixed (failed before, passes now) or broken (passed before, fails now). The totals can hide a swap. In the worked example below, a candidate fixes 6 cases and breaks 10, so the pass rate falls from 90% to 88%.1
Is that a real regression? Only the 16 changed cases carry information. If the change did nothing and results flipped by chance, a 10-to-6 split or a more lopsided one happens about 45% of the time (exact two-sided p-value 0.4545).1 This test on the changed cases is the exact McNemar test, which is a sign test on the discordant cases. So the two-point drop is not distinguishable from noise, yet the 10 broken cases are still worth reading, because they may share one cause. A 14-to-2 split would give a p-value of 0.0042, below a 0.05 threshold (illustrative), and block the change.1
Step 4: Gate every change and grow the set
The gate runs the comparison on every change. OpenAI describes continuous evaluation: run evals on every change, watch for new nondeterminism, and grow the eval set over time. Tools exist to run prompt evals inside continuous integration, such as the promptfoo GitHub Action.
A useful gate has two parts: a hard rule (zero failures on a small must-pass subset, for example 20 cases, illustrative) and a statistical rule (block when the paired comparison shows a real drop). Anthropic's success criteria are defined on a held-out test set, so tune prompts on a development slice and keep a held-out slice untouched. When production shows a new failure, add it as a case so it can never return unseen.
Step-by-step description
- Real traffic and edge cases become golden cases.
- Code, judge or human grades each output.
- Fixed and broken cases are counted.
- A real drop blocks the change; new failures return as cases.
- Each change is graded case by case against the frozen baseline.
- The merge gate blocks a change when the paired comparison shows a real drop.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
Take a golden set of 200 cases (illustrative). The shipped version passes 180 and the candidate passes 176. Comparing the same cases, 6 were fixed and 10 broken, 170 pass in both and 14 fail in both.1 The standard error (SE) of the baseline pass rate is the square root of 0.9 × 0.1 / 200, which is 2.12 percentage points, so a 2-point gap is about one standard error wide.1
| Quantity | Value (illustrative inputs, computed results) |
|---|---|
| Baseline pass rate | 180 of 200 (90%) |
| Candidate pass rate | 176 of 200 (88%) |
| Cases fixed / broken | 6 / 10 |
| Pass in both / fail in both | 170 / 14 |
| SE of the baseline pass rate | 2.12 points |
| 95% Wilson interval, baseline | 85.1% to 93.4% |
| 95% Wilson interval, candidate | 82.8% to 91.8% |
| Exact paired p-value, 10 broken vs 6 fixed | 0.4545 |
| Exact paired p-value, 14 broken vs 2 fixed | 0.0042 |
The Wilson interval is a confidence interval for a proportion that behaves well for pass rates near 100%.1 The paired p-value is the exact two-sided sign test on the changed cases only.1
How many cases do you need? Anthropic advises volume over quality and gives no number. The arithmetic gives one: the 95% margin of error on a 90% pass rate is 8.32 points at 50 cases, 4.16 at 200, 2.08 at 800 and 1.04 at 3,200.1 This margin on a single pass rate is a rough guide: whether a drop is detectable depends on how many cases change between versions, the discordant cases of the paired comparison, and the normal-approximation margin is crude at small sizes such as 50. Pick the size by the smallest drop you must catch.
Text description of the diagram
- The 95% margin of error on a 90% pass rate is 8.32 points at 50 cases.
- The margin of error is 4.16 points at 200 cases and 1.04 points at 3,200 cases.
Nondeterminism adds noise per case. If a case passes 80% of single runs, a majority vote of three runs passes it 89.6% of the time, and a case at 50% stays at 50%.1 Repeated runs separate steady cases from flaky ones, so quarantine the flaky ones instead of letting them trip the gate.
To make the harness as deterministic as possible, pin the model snapshot or version so the baseline and the candidate are comparable, and set temperature 0 and a fixed seed where the provider supports them. These settings reduce variation, but identical outputs are still not certain. Cache the outputs of unchanged prompts so reruns are cheap, and treat a model-version change as a change that must pass the gate like any other.
Trade-offs and edge cases
The main trade-off is cost against confidence: more cases, more repeats and an LLM judge all raise confidence and cost. A common pattern is a small fast subset on every change and the full set before release. Choose graders by the ladder.
Text description of the diagram
- Code-based grading is the fastest and most reliable but lacks nuance.
- Human grading is the most flexible but slow and expensive.
- LLM-based grading is fast and flexible but must be tested for reliability first.
Edge cases to name in an interview:
- Overfitting to the set. Tuning a prompt until the golden set passes teaches the prompt the set. Keep a held-out slice and never paste golden cases into few-shot examples or fine-tuning data. That also answers the leakage question.
- Versioning. Store the set in version control with a version number, add cases through reviewed changes, and re-run the baseline whenever the set changes, since scores are comparable only on the same version.
- Retrieval apps. For a retrieval-augmented generation (RAG) app, OpenAI's example thresholds for question answering over documents are context recall of at least 0.85, context precision over 0.7 and 70% or more positively rated answers; treat them as an illustration, not a standard. The Ragas page covers these metrics.
- Adversarial cases. Add known prompt-injection inputs so a prompt change cannot reopen a hole; see AI agent security.
- Noisy references. Open-ended tasks have no single right answer, so prefer pass rules over exact references.
- Assertion-style and scoring-style tests. An assertion-style test gives a hard pass or fail per case, such as "must cite the source", and suits a merge gate. A scoring-style test gives a graded score, such as a judge's rating from 1 to 5, and suits trend tracking and threshold gates on the average.
- Shadow eval. A shadow eval runs a candidate on copies of live traffic without showing its answers to users, then grades them, which tests the candidate on real inputs before release.
How to answer it in an interview
A prompt tweak fixed one case and silently broke ten others. How do you regression-test an LLM app in CI?
I keep a golden set: cases drawn from real traffic and logs, plus edge cases, each with a reference answer or a pass rule. Every prompt or model change runs against it, and I compare case by case with a frozen baseline, not just the totals. The gate blocks if a must-pass subset breaks or if the paired comparison shows a real drop, because single runs are noisy. When production shows a new failure, I add it as a case. I keep a held-out slice so I do not overfit the prompt.
The spoken structure: name the set, name the baseline comparison, name the gate, then say how noise and overfitting are handled.
Common mistakes
- "I tested it on a few prompts and it looked fine." (wrong) That is a vibe-based eval. (right) "I run a fixed golden set on every change and compare per case."
- "We use a public benchmark as our golden set." (wrong) It does not match your traffic. (right) "The set comes from our own logs and expert answers, with edge cases."
- "The pass rate dropped two points, so block it." (wrong) On a small set that can be noise. (right) "I check the fixed and broken cases with a paired test and read the broken ones."
- "The LLM judge scores everything, so I need no humans." (wrong) An untested judge can be unreliable. (right) "I validate the judge against human labels before I scale it."
Follow-up questions
What is a golden dataset and why is it foundational?
A golden dataset is the fixed set of real cases with trusted answers or pass rules that every change is measured on. Without it, quality is an opinion, and no comparison between versions is repeatable.
How many examples do you actually need in a golden set?
A golden set needs enough cases that the margin of error is smaller than the drop you care about. At a 90% pass rate, 200 cases give about a 4.16-point margin and 800 give about 2.08.1 Anthropic favors more automatically graded cases over fewer hand-graded ones.
How do you build an eval set when you have no labels yet?
An eval set with no labels starts from real or realistic inputs, with pass rules written for behavior that matters, such as "must cite the source" or "must refuse". Review a sample by hand, then add labeled cases as users give feedback.
How do you handle non-determinism in regression tests?
Nondeterminism in regression tests is handled by running each case several times, grading by majority or pass fraction, and gating on the paired comparison rather than one run. Pinning the model version, setting temperature 0 and a seed where supported, and caching unchanged outputs cut the variation, and flaky cases are quarantined and investigated.
How do you prevent your eval set from leaking into training or prompt optimization?
Eval-set leakage is prevented by keeping a held-out slice that is never used for tuning, keep golden cases out of few-shot examples and fine-tuning data, and refresh the held-out slice with new production cases.
When and how should you version golden sets?
A golden set is versioned whenever cases change, stored in version control, and recompute the frozen baseline on each new version. Compare scores only within one version.
- A golden set is a curated, versioned set of real cases with trusted answers or pass rules.
- Regression testing reruns it on every change and compares per case with a frozen baseline.
- A two-point drop on 200 cases can be noise; the paired comparison on changed cases tells.
- Grade with code where you can, an LLM judge once validated, and humans for calibration.
- Hold out a slice, version the set, and add every new production failure as a case.
Sources
- Define success criteria and build evaluations — Anthropic (Claude Platform Docs), accessed 2026-10-26, 26 Oct 2026
- Evaluation best practices — OpenAI API docs, accessed 2026-10-26, 26 Oct 2026
- promptfoo-action: the GitHub Action for Promptfoo — promptfoo (GitHub), accessed 2026-10-26, 26 Oct 2026
