Building an eval set for LLM applications: criteria, held-out cases and graders

In this article

1. Before you start
2. Why interviewers ask this
3. Building an eval set from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

It builds on how transformers work and assumes only the ideas below.

Why interviewers ask this

Without an eval set, a team cannot tell a real regression from noise. Anthropic's engineers (read October 2026) describe teams without evals as unable to tell real regressions from noise, facing weeks of testing whenever a new model ships. Interviewers for applied LLM roles use the topic to see whether a candidate treats evaluation as engineering rather than a feeling.

A weak answer says "I check a few examples by eye." A strong answer names the success criteria, where the cases come from, how each case is graded and how far a score can be trusted. A Staff-level answer adds the held-out split, the error bar and a plan to grow the set from production failures. Questions on this theme are collected on the LLM interview questions and natural language processing (NLP) interview questions pages.

Building an eval set from first principles

Eval set

An eval set is a fixed collection of test cases for one LLM application, where each case holds an input and the success criteria used to grade the output, so that any prompt or model change can be scored against the same list.

Held-out set

A held-out set is a slice of the eval set that you never look at while changing prompts and use only for the final check, so its score is not inflated by tuning against it.

In plain words

Think of an exam. The teacher writes the questions and an answer key before the course starts, and every change to the lessons is judged by the exam score, not by a feeling. The analogy breaks in two places. A model gives a different answer on a different run, and the grader is itself imperfect.

Anthropic's engineers define an eval as giving an AI an input and then applying grading logic to its output to measure success. In one precise sentence, an eval set turns the question "does the new prompt work?" into a score on fixed cases, with a stated uncertainty.

Step 1: Write success criteria before any test case

Success criteria are measurable statements of what a good application means, and they come first, before any test case is written: Anthropic's documentation says to define them and then design evaluations against them.

Its example criterion is an F1 score (one number from 0 to 1 that combines how many flagged items were right and how many true ones were found) of at least 0.85 on a held-out test set of 10,000 diverse posts, a 5% improvement over the current baseline, meaning the score of the system you have today (same page). Most applications need several criteria together, such as accuracy, a non-toxic rate, error severity and latency.

Writing them also removes ambiguity. Two engineers reading the same spec can interpret edge cases differently, and an eval suite resolves that. Input: the task description. Output: a short list of measurable targets. You now know what every case will be graded against.

Step 2: Collect cases that mirror the real task, including edge cases

The cases in an eval set should sample the real task, not invented examples. Anthropic's guide says to mirror your real-world task distribution and to include edge cases: irrelevant or nonexistent input, overly long input, and ambiguous cases where even humans would struggle to agree.

OpenAI's guide suggests mixing production data, correct answers written by domain experts and historical logs. Overly long input ties to the context window, and unusual characters change how text is split into tokens, which the byte pair encoding page explains.

Where production data is thin, a model can generate extra candidate cases to widen coverage, but a person should review them before they enter the set. Where a case is ambiguous, have two graders label it independently and agree the expected answer, or drop the case.

For size, Anthropic's own example criterion uses a held-out set of 10,000 posts, but the right size depends on the task and on the error bar you need (Step 5). Input: real traffic, logs and expert answers. Output: a list of cases with inputs and expected behaviour.

Step 3: Split the cases so the final test stays held out

Splitting the eval set into a development part and a held-out part keeps the final score honest. Each time you read a failure and edit the prompt, the prompt adapts to those cases, so their score rises for reasons that need not carry over to new inputs.

Anthropic's workflow ends with a test against held-out evals before shipping, separate from the cases used to refine the prompt, so the held-out set stays unseen until the final test. Use the development cases to iterate, and use the held-out cases once, to decide whether to ship.

If you look at the held-out score and keep tuning, it is no longer held out and you need fresh cases. The size of the split is a judgement: the held-out part must be large enough for the error bar in Step 5 to be useful. Input: all cases. Output: two lists with no overlap.

Step 4: Choose a grader for each criterion

Each criterion needs a grader, and the three kinds trade speed against nuance. Anthropic's documentation calls code-based grading the fastest and most reliable but lacking nuance, and human grading flexible but slow and expensive. Code-based graders are also brittle to valid variations that do not match the expected pattern.

An LLM judge is a model that reads an output and scores it against a rubric. Zheng et al. (2023) found strong judges agree with humans over 80% of the time, about as often as humans agree with each other, but they show position, verbosity and self-enhancement biases. Calibrate the judge on human labels, and judge with a different model from the one that wrote the output.

Anthropic also prefers volume: more automatically graded cases beat fewer hand-graded ones. OpenAI's interface asks for the same two pieces: a schema for the data (data_source_config) and the graders (testing_criteria), per its evals guide.

Step 5: Run the set and report a score with an error bar

An eval score is a sample estimate, so it needs an error bar. Miller (2024) treats evaluations as experiments whose questions are drawn from a larger unseen population, and says in its introduction that error bars remain noticeably absent from traditional question-and-answer evals.

A confidence interval is a range around a measured score that is expected to contain the true score in most repeats of the experiment. Because outputs vary between runs, run each case more than once. pass@k measures the likelihood of at least one correct solution in k attempts, which suits a product where the user can retry or pick the best result. For a product where every response must be right, use pass^k, the probability that all k attempts succeed, defined in the same post. Repeated runs of one case are not independent cases, so the interval should treat the runs of a case as a cluster (Miller, 2024).

Then keep the loop going: OpenAI recommends running evals on every change, watching production for new non-deterministic behaviour and growing the set over time. Scores are reported with an error bar.

1 · Criteria2 · Cases3 · Split4 · Graders5 · ScoreReplay

Step-by-step description

1. Write measurable success criteria first.
2. Collect real cases and edge cases.
3. Keep a held-out slice unseen.
4. Pick a grader for each criterion.
5. Report the score with an error bar.

• Success criteria come first, before any test case is written.
• The held-out set stays unseen until the final test.
• Scores are reported with an error bar.

Worked example

Take an illustrative application whose true pass rate is 0.85, and ask how wide the 95% interval is for different set sizes. The standard error (SE) is the typical size of the sampling noise in a measured rate, and the interval is 1.96 standard errors either side:¹

Here p is the pass rate and n is the number of cases.

The 95% interval at a pass rate of 0.85 is about 9.9 points either side at 50 cases and 2.2 points at 1,000 cases, so a score from 50 cases cannot separate prompts that differ by a few points.¹ Holding the interval to 5 points needs about 196 cases at this pass rate.¹ The normal approximation is rough at 50 cases with a pass rate of 0.85; a Wilson interval is safer there, as Miller (2024) discusses.

Text description of the diagram

• The 95% interval at a pass rate of 0.85 is about 9.9 points either side at 50 cases and 2.2 points at 1,000 cases.
• A score from 50 cases cannot separate prompts that differ by a few points.

Now compare two prompts on the same 100 cases (illustrative). Prompt A passes 80 and prompt B passes 82. B passes 6 cases that A fails, and A passes 4 cases that B fails, so the gain is 2 points. With b and a the discordant counts (cases only one prompt passes), the gain is $d=(b-a)/n$ and its standard error is ${\mathrm{SE}}_d=\sqrt{((a+b)/n-d^2)/n}$.

Here (6+4)/100 = 0.10, d = 0.02, the standard error is about 0.0316 and the half-width is about 6.2 points.¹ The paired 95% interval is about 6.2 points either side, from -4.2 to +8.2, and it includes zero, so the data cannot call B better.¹ With only 10 discordant pairs, McNemar's exact test is safer than the normal approximation.

For repeated attempts, take an illustrative per-attempt success rate of 0.60 and assume independent attempts. Then pass@k is one minus 0.40 to the power k: 0.60 for k of 1, 0.936 for k of 3 and 0.9898 for k of 5. The all-attempts metric pass^k is 0.60 to the power k: 0.216 for k of 3 and 0.0778 for k of 5, so the same 0.60 per-attempt rate looks reliable under one metric and weak under the other.¹

Trade-offs and edge cases

No grader wins every row, so mix them. Code graders are cheap but reject valid answers that differ in form, model graders handle nuance but vary between runs, and human graders are the reference for calibrating the others, according to Anthropic's engineering post.

Text description of the diagram

• Code-based graders are fast and reliable but brittle to valid variations.
• Model-based graders are flexible but non-deterministic and need calibration with human graders.
• Human graders are slow and expensive, and they calibrate the model-based ones.

Code-based graders are fast and reliable but brittle to valid variations. Model-based graders are flexible but non-deterministic and need calibration with human graders. Human graders are slow and expensive, and they calibrate the model-based ones.

Two kinds of eval need different targets. Capability evals should start at a low pass rate, and regression evals should sit near 100%; a capability task that the application now passes can graduate into the regression suite.

Public benchmarks are no substitute. They measure someone else's task distribution, and Anthropic advises mirroring your own. Finally, a set built once goes stale: add each new production failure to it, and keep any case you used while tuning out of the held-out part.

How to answer it in an interview

The spoken structure: criteria, cases, held-out split, graders, then the error bar and the growth loop.

Common mistakes

• "I tuned the prompt until the eval score hit the target." (wrong) That score is inflated by tuning on those cases. (right) "I iterate on development cases and report the held-out score once."
• "Accuracy rose two points, so the new prompt wins." (wrong) On a small set that is within the error bar. (right) "I report a paired interval, and only call a winner when it excludes zero."
• "The judge model said the answers are good." (wrong) Judges have position, verbosity and self-enhancement bias. (right) "I calibrate the judge on human labels and use a different model from the generator."
• "A public benchmark covers us." (wrong) It does not sample your inputs. (right) "I build cases from our own traffic and edge cases."
• "We built the set once." (wrong) Production keeps producing new failures. (right) "Every new failure becomes a case, and passing capability cases graduate to regression."

Follow-up questions

How many test cases do you need in an eval set?

Enough for the error bar to be smaller than the differences you care about. At an illustrative pass rate of 0.85, about 196 cases give a 5-point interval either side.¹

How do you know if an LLM judge can be trusted?

Compare its scores with human labels on a sample of cases, and test for order and length bias. Zheng et al. (2023) report agreement with humans above 80% for strong judges, but your rubric and domain may differ.

What is the difference between a capability eval and a regression eval?

A capability eval asks what the application can do now and starts at a low pass rate. A regression eval checks that nothing broke and should sit near 100%. Passing capability tasks can move into the regression suite.

How do you evaluate an application whose outputs change between runs?

Run each case several times and report the pass rate with an interval. A single run mixes real change with luck. Use pass@k when the user can retry or pick the best result, and pass^k when every response must be right; at an illustrative 0.60 per attempt and k of 3, they give 0.936 and 0.216.¹

How do you keep an eval set from going stale?

Run it on every change and add new failures from production. OpenAI's guidance is to grow the set over time. For more practice questions on this theme, see the LLM interview questions.

Sources

1. Define success criteria and build evaluations — Anthropic, Claude Platform Docs (accessed 2026-10-26), 26 Oct 2026
2. Demystifying evals for AI agents — Anthropic Engineering (published 2026-01-09, modified 2026-10-18), 18 Oct 2026
3. Evaluation best practices — OpenAI API docs (accessed 2026-10-26), 26 Oct 2026
4. Working with evals — OpenAI API docs (accessed 2026-10-26), 26 Oct 2026
5. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (arXiv 2306.05685) — arXiv (Zheng et al., NeurIPS 2023), 9 Jun 2023
6. Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations (arXiv 2411.00640) — arXiv (Miller), 1 Nov 2024