How would you design an eval for X?

In this article

1. Before you start
2. Why interviewers ask this
3. Designing an eval from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

After this page you can turn the open prompt "design an eval for X" into a structured answer: criteria, dataset, graders and a rerun loop. For the wider topic, see the LLM evaluation guide.

Why interviewers ask this

"Design an eval for X" is an open prompt where X is a feature, such as a summarizer or a support bot. It is reported by candidates as a common prompt in AI and machine learning interviews. Interviewers use it to see whether you treat quality as something you can measure, or as a feeling.

The consequence of getting it wrong is practical. Anthropic's engineering team describes teams without evals as stuck in reactive loops: they fix one failure, create another, and cannot tell real regressions from noise. A regression is a change that makes a case that used to pass fail.

A weak answer names a benchmark or says "I would test it manually." A strong answer starts from the task, states thresholds, and explains how the test set is built and graded. A Staff-level answer adds the statistics (how much noise the score carries) and the loop that keeps the eval current as production teaches you new failures.

Designing an eval from first principles

Eval

An eval is a repeatable test that runs an AI system on fixed inputs and scores each output against written rules, so two versions can be compared on the same evidence. It differs from a one-off manual check because the inputs and the grading rules are written down and can be rerun on every version.

Grader

A grader is the logic that turns one system output into a score or a pass or fail verdict. It can be a string comparison, a piece of code, a model, or a person.

In plain words

Think of a driving test. The examiner does not ask "do you seem like a good driver?" They fix a route, a checklist and a pass mark, and every candidate faces the same conditions. An eval does the same for a model: a fixed set of situations, a checklist, a pass mark. The analogy breaks in one place: a driving test is graded by one trained examiner, while model outputs vary from run to run and some graders are themselves models that can be wrong.

In one precise sentence, designing an eval means choosing criteria, cases and graders so that a change in the score reflects a real change in quality.

The workflow follows the one in OpenAI's evaluation guide: define the objective, collect a dataset, define metrics, then evaluate continuously. The four steps below add the sizing and grader choices an interviewer probes.

Step 1: Define success as measurable criteria

Success criteria are the written conditions that say when X is good enough. They must be specific and cover more than one dimension. Anthropic's documentation gives a sentiment-classification example: on a held-out set of 10,000 posts, an F1 score (the harmonic mean of precision and recall) of at least 0.85, 99.5% non-toxic outputs, and 95% of responses under 200 milliseconds.

Input: the feature and its users. Output: a short list of thresholds across quality, safety and latency. If you cannot say which number would make you ship or hold back, the criterion is not finished. "Held-out" means the cases were not used to tune the prompt, so they test generalization. In an interview, ask what a bad output costs the user before choosing thresholds: that decides how strict the safety line is.

Step 2: Build the dataset and size it for noise

The dataset is a golden set drawn from your real task; the golden sets and regression testing page goes deeper on keeping one. OpenAI names biased design as a common mistake: eval data that does not reproduce production traffic. Anthropic adds that you should include edge cases: irrelevant or missing input, overly long input, and ambiguous cases where even humans would struggle to agree.

Size matters because a pass rate is an estimate. Its margin of error is about 1.96 times the square root of p(1−p)/n, where p is the true pass rate and n the number of cases. At a pass rate near 80%, 30 cases give a margin of about 14.3 points and 1,000 cases about 2.5 points; the numbers are computed in the worked example below.¹ Input: real traffic samples and failure reports. Output: a labeled set big enough that its margin is smaller than the change you care about. In short, the dataset mirrors real traffic and includes edge cases.

Step 3: Choose a grader for each criterion

A grader should fit the criterion, not the other way round: each criterion gets a grader that fits it. Exact match checks whether the output equals a predefined answer, typically after normalizing whitespace and case, so it suits categorical outputs such as a label. A code check suits format rules, such as valid structure or a length limit. Subjective qualities such as tone or coherence need a model grader, and Anthropic's example scores tone on a Likert scale (a rating scale, such as one to five) with a model.

OpenAI says models are better at discriminating between options, so favor pairwise comparison or scoring against specific criteria over grading open-ended text. It also lists not calibrating automated metrics against human evaluation as a mistake. Calibrating means grading a sample by hand and checking that the model grader agrees. The LLM-as-a-judge page covers the failure modes.

Step 4: Run it on every change and grow it

Run the eval whenever the prompt, model or retrieval changes. OpenAI describes this as continuous evaluation: run evals on every change, watch for new nondeterminism, and grow the eval set over time. Compare versions on the same cases so a difference is not an accident of different inputs. This is also how you A/B test prompts or models before release: run both on the same golden-set cases, compare per-criterion scores, and check the gap against the margin; on live traffic, split requests randomly between the versions instead. The internal eval platform page shows how teams gate every change on that comparison.

Input: a candidate version and the golden set. Output: per-criterion scores for each version, plus a list of cases that flipped from pass to fail. Every failure found in production or manual testing becomes a new case. A harness runs the cases; the lm-eval paper describes it as orchestration that does not dictate which benchmarks matter. For retrieval systems, the Ragas page shows metrics that fit that part of the design.

1 · Criteria2 · Dataset3 · Grader4 · Run and growReplay

Step-by-step description

1. Write measurable success criteria.
2. Build cases from real traffic.
3. Pick a grader for each criterion.
4. Run on every change; add failures.

• Success criteria are measurable and cover quality, safety and latency.
• The dataset mirrors real traffic and includes edge cases.
• Each criterion gets a grader that fits it.
• The eval runs on every change, and production failures become new cases.

Worked example

Take X to be a feature that summarizes customer support tickets. Everything here is illustrative except the two sourced examples. For comparison, OpenAI's own summarization example asks for a Recall-Oriented Understudy for Gisting Evaluation (ROUGE) score called ROUGE-L of at least 0.40 (the longest-subsequence version) and a coherence score of at least 80% on a held-out set of 1,000 reference transcripts.

First, how much noise a pass rate carries. Assume a true pass rate of 80% (illustrative). The margin of error is 14.3 points at 30 cases, 7.8 at 100, 4.5 at 300 and 2.5 at 1,000.¹ To pin the pass rate within 5 points you need about 246 cases; within 3 points, about 683.¹ The normal approximation is rough at 30 cases, where n times p(1−p) is only 4.8,¹ so a Wilson interval is the better choice there.¹

Now compare two versions that score 80% and 84% on 100 cases each. Treating the two runs as independent samples, the margin of that 4-point gap is about 10.6 points, so the gap cannot be told apart from noise at 95% confidence.¹ When both versions run on the same cases, count the cases that flipped each way and test them with a paired method such as McNemar's test.

Text description of the diagram

• At a pass rate near 80% the margin is 14.3 points at 30 cases.
• The margin is about 2.5 points at 1,000 cases.

Second, the same ideas drive prioritization. Suppose 200 quality signals fall into 8 clusters, and each cluster has a severity from 1 to 5 and an effort from 1 to 4 (all illustrative). Score each cluster as frequency times severity divided by effort. With an effort budget of 10, picking clusters in score order selects six: wrong output format (score 104), hallucinated citations (63.3), slow responses (40), policy refusals ignored (35), off tone (30) and wrong language (18).¹ Together they cover 180 of the 200 signals, 90%, and the long-context and tool-call clusters wait.¹

Trade-offs and edge cases

Every grader type trades cost, accuracy and coverage. Exact match is cheap and unambiguous, but it marks a valid rewording as wrong. A code check enforces format and length rules, but it cannot judge quality. A model grader scales to tone and coherence, but it needs calibration against human labels before you trust it. Human review handles ambiguous cases best, but it is slow and costly, so it is kept for calibration samples and hard cases. The table below summarizes which grader suits which criterion.

Text description of the diagram

• Exact match suits categorical answers.
• A model grader suits subjective qualities such as tone and needs calibration against human labels.
• Calibrating means grading a sample by hand and checking that the model grader agrees.

Edge cases to name in an answer:

• Noise from the system itself. Model outputs vary between runs, so a single run of a case is a noisy measurement. Rerun borderline cases or report the margin, not one number.
• A set that stops resembling traffic. Production changes, so a golden set built last quarter drifts. Add new failures and retire cases that no longer match real use.
• Overfitting to the set. If you tune a prompt against the same cases every day, the score rises faster than real quality. Keep a held-out slice that nobody tunes against.
• Single-number summaries. One average hides a criterion that regressed while another improved; Anthropic's example uses several thresholds for this reason.
• Agent systems. For agents that call tools, the eval must also check actions and side effects; see AI agent security for the safety side.
• Judge biases. Zheng et al. (2023) describe model-judge biases toward the answer shown first (position), toward longer answers (verbosity) and toward the judge's own outputs (self-preference). Counter them by running pairwise grading in both orders and counting only consistent verdicts, and by checking the judge against human labels.
• Single run versus repeated runs. An agent can succeed on one attempt and fail on the next. Two common ways to summarize repeated attempts are pass@k, the chance that at least one of k attempts succeeds, and pass^k, the chance that all k succeed. Pass@k suits tasks where one success is enough; pass^k suits tasks that must work every time.

How to answer it in an interview

To structure your answer to "design an eval for X", say it in this order: criteria, dataset and its size, grader per criterion, then the loop that keeps it current. This is the site's own framework, built on the workflow in the OpenAI guide; it is not any company's official rubric. Say the structure first, then go deeper wherever the interviewer pushes.

Common mistakes

• "I would check a few outputs by hand and see if it looks right." (wrong) That is what OpenAI calls vibe-based evaluation. (right) "I would write thresholds first and measure them on a fixed set."
• "I would report one benchmark score." (wrong) A generic benchmark does not mirror your task. (right) "I would build cases from our own traffic, then add public benchmarks only as a sanity check."
• "The model grader said version B is better, so ship it." (wrong) The grader may be uncalibrated, and the gap may be noise. (right) "I checked the grader against human labels and the gap exceeds its margin."
• "I will build evals after launch." (wrong) Without a baseline from before release, you cannot tell a regression from noise. (right) "I define the eval before the first release and grow it afterwards."

Follow-up questions

How would you build an LLM eval framework from scratch?

Build the smallest loop first: a file of cases, a function that runs the system, graders that score each output, and a report that compares versions. Then add the pieces the four steps call for: versioned datasets, calibration against human labels, and a run on every change. The pieces are described in the LLM evaluation guide.

How many examples do you actually need in a golden set?

Enough that the margin of error is smaller than the effect you want to detect. At a pass rate near 80% (illustrative), about 246 cases give a 5-point margin and about 683 give 3 points.¹ Start smaller to learn what fails, then grow it from production failures.

You have 200 signals of quality problems and time for 10 fixes. How do you prioritize?

Group the signals into clusters, then score each cluster by how often it occurs, how bad it is and how much effort the fix needs. In the illustrative example above, the 10 fixes are modeled as an effort budget of 10 units, and taking clusters in score order while they fit the budget covers 90% of the signals.¹ Show the scoring, then say you would revisit the weights with the product owner.

How do you handle cost and latency of LLM judging at scale?

A strong answer sizes the workload and then cuts it. In the illustrative workload of 300 cases, 2 criteria and 5 changes a day, that is 3,000 judge calls a day; sampling 20% of cases on routine runs cuts it to 600.¹ Use a cheaper judge for routine runs, the full set before a release, and cache grades for unchanged outputs. Check any cheaper judge against human labels first.

What is a shadow eval, and when do you use one?

A shadow eval runs a candidate system on copies of live requests without showing its output to users, then grades the results. You use it when your golden set may not capture current traffic and a wrong answer in front of real users would be costly. It measures realistic inputs without exposing users to the candidate.

How do you source real production traffic for evals without breaking privacy?

The privacy in machine learning (ML) systems page covers the wider practice. A strong answer removes or masks personal data before cases are stored, keeps only what the eval needs, limits who can read it and for how long, and follows the product's consent and retention rules. Ask the privacy or legal owner which fields are allowed. Where traffic cannot be used, write realistic synthetic cases and label them as such.

Sources

1. Evaluation best practices (OpenAI API documentation; undated, accessed 2026-10-26) — OpenAI, 26 Oct 2026
2. Define success criteria and build evaluations (Anthropic documentation; undated, accessed 2026-10-26) — Anthropic, 26 Oct 2026
3. Demystifying evals for AI agents (Anthropic Engineering; undated, accessed 2026-10-26) — Anthropic, 26 Oct 2026
4. Lessons from the Trenches on Reproducible Evaluation of Language Models (arXiv 2405.14782) — arXiv (Biderman et al.), 23 May 2024
5. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (arXiv 2306.05685) — arXiv (Zheng et al.), 9 Jun 2023