How do you design CI-friendly eval suites?

In this article

1. Before you start
2. Why interviewers ask this
3. CI-friendly eval suite from first principles
4. The Staff-level answer
5. The rubric
6. Three variants of this question
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

If evals are new to you, start with the large language model (LLM) evaluation guide; a golden set is the usual core of the regression tier described below.

Why interviewers ask this

A flaky or slow eval gate teaches a team to ignore it, and then the checks meant to catch a broken prompt or model change protect nothing. The question separates candidates who treat model evals like ordinary unit tests from those who know that outputs vary between runs; Anthropic (2026) says agent behavior varies between runs, which makes results harder to interpret than they first appear.

A weak answer says "run the evals in CI." A strong answer separates fast regression checks from slow evals, picks graders by cost and trust, and sets a threshold from a baseline. A Staff-level answer also does the noise arithmetic and says what the build does when a check fails. Candidates report this question in design rounds for products backed by language models.

CI-friendly eval suite from first principles

Continuous integration (CI) gate

A continuous integration (CI) gate is a step in an automated build, the pipeline that runs on every code or prompt change, that runs a check and fails the build when the result falls below a set threshold. A bad change then cannot merge without someone noticing and deciding what to do.

Flaky gate

A flaky gate is a CI gate that sometimes fails a good change or passes a bad one because the measured pass rate moves by chance between runs, not because the system changed. Output variation, a suite that is too small and a threshold set too tightly are the usual causes.

Tiered eval suite

A tiered eval suite is an eval suite split into groups by cost and speed, such as a small group on every commit and a large group on a schedule, so that each trigger runs only the checks that fit its time and cost budget.

In plain words

Think of a smoke alarm and a yearly building inspection. The alarm is cheap and always on, so it catches fires early. The inspection is slow and deep, so it happens on a schedule. A CI-friendly suite works the same way. The analogy breaks in one place: a smoke alarm gives the same answer for the same room, but a language model check can pass on one run and fail on the next.

Four words are enough to read the rest. A task is one test with an input and a success rule. A trial is one attempt at a task. A grader is the code, model or person that scores the output of a trial. A regression suite is a set of tasks the system already passes, run to catch breakage.

In one precise sentence, a CI-friendly eval suite is a tiered set of automated tasks, with graders and thresholds, whose run time, cost and noise are small enough that its pass or fail result can be trusted on every change.

1 · Tier by trigger2 · Size for noise3 · Repeat trials4 · Pick graders5 · Wire into CI6 · Feed failures backReplay

Step-by-step description

1. Cheap checks per change, slow evals on a schedule.
2. Set the gate from a baseline and the standard error.
3. Pick pass@k or pass^k to match user expectations.
4. Code graders first; validate any model grader.
5. Filters, cache, artifacts and a failing exit code.
6. Every real failure becomes a new test case.

• Cheap checks run on every change and slow evals run on a schedule.
• The exit code of the gate decides whether the build passes.

Step 1: Split the suite into tiers by trigger

Cheap checks run on every change and slow evals run on a schedule. A tiered eval suite fits each trigger to its time budget: a smoke tier of a few dozen code-graded cases on every commit, a regression tier on changes that touch prompts, models or configuration, and larger or noisier evals nightly or weekly. Promptfoo's documented example runs on pull requests only when files such as prompts/** change, and it runs its red-team scan on a daily schedule (Promptfoo docs).

Regression evals should pass nearly 100%, while capability evals start at a low pass rate and give the team a hill to climb (Anthropic, 2026). So the per-change gate holds regression tasks, and capability evals run on the schedule. Input: a change. Output: the tier that runs. A golden set is the usual core of the regression tier.

Step 2: Size the suite and the gate for noise

A pass-rate gate on a small suite fails healthy builds because the measured rate moves by chance between runs. One run of N cases is a sample, and its spread is the standard error, $\sqrt{\frac{p(1-p)}{N}}$, where p is the true pass rate. At a true pass rate of 0.9, a 50-case suite has a standard error of 4.24 points and measures the rate within plus or minus 8.32 points (a 95 percent interval, using the normal approximation, a standard bell-curve shortcut); a 1,000-case suite measures it within plus or minus 1.86 points.¹

Text description of the diagram

• At a true pass rate of 0.9, a 50-case suite measures it within plus or minus 8.32 points.
• A 1,000-case suite measures it within plus or minus 1.86 points.

A gate at 95 percent on 50 cases needs 48 passes, so a system whose true rate is 95 percent fails it about 46 percent of the time. Set the gate from a measured baseline minus a margin of about two standard errors, which is 45 of 50 passes, and healthy builds fail only about 4 percent of the time.¹

The price is missed regressions: the same gate still passes a system that has dropped to 85 percent about 22 percent of the time,¹ so if a 10-point drop must be caught, grow the suite or the trials rather than tightening the threshold. The baseline-minus-two-standard-errors rule is this page's own arithmetic. Anthropic's success-criteria guidance asks for specific, measurable targets on a held-out test set, such as an F1 score of at least 0.85 on 10,000 posts, and the 95 in Promptfoo's sample threshold script is an example value. Input: baseline runs. Output: a gate wider than the noise.

Step 3: Repeat trials, and choose pass@k or pass^k to match what users expect

The measures pass@k and pass^k answer different questions about repeated trials. Running several trials per task turns one noisy result into a rate (Anthropic, 2026). pass@k is the chance that at least one of k attempts succeeds, and pass^k is the chance that all k succeed. For independent trials with per-trial pass rate p, $\text{pass@k}=1-(1-p{)}^k$ and ${\text{pass}}^k=p^k$.

At a per-trial pass rate of 0.75, three trials all pass with probability 0.4219, while at least one passes with probability 0.9844.¹ At a per-trial pass rate of 0.9, ten trials all pass with probability 0.3487, and a hundred trials all pass with probability about 0.003 percent.¹ Gate on pass^k where users expect consistent behavior, and on pass@k where a retry is cheap. Start each trial from a clean state so failures stay independent.

Text description of the diagram

• At a per-trial pass rate of 0.75, three trials all pass with probability 0.4219.
• At a per-trial pass rate of 0.9, ten trials all pass with probability 0.3487.

Step 4: Pick graders by cost and trust

A CI suite uses code-based graders wherever it can, and a model-based grader only after it has been checked against human labels. Anthropic (2026) lists code-based graders as fast, cheap, objective and reproducible, but brittle to valid variations; model-based graders as non-deterministic, more expensive and in need of calibration against human graders; and human grading as the slow, costly reference.

The Claude Platform docs advise testing a model grader's reliability first and then scaling it, and grading with a different model from the one that wrote the output. Anthropic (2026) says a task's score can be weighted, binary or a hybrid. Code-based checks are brittle to valid variations, so grade the final outcome rather than the exact path taken. See LLM-as-a-judge for calibration.

Step 5: Wire the suite into CI with filters, a cache, artifacts and an exit code

A CI job runs the suite on a trigger, saves its results as artifacts (files the build keeps for later) and returns a nonzero exit code when the gate fails. The exit code of the gate decides whether the build passes. Promptfoo's documented example uses --fail-on-error or a threshold script that exits with code 1 below its pass rate, paths: filters, a response cache and a final if: always() step that uploads the results (Promptfoo docs).

It writes machine-readable results as JSON (JavaScript Object Notation) and as Extensible Markup Language (XML) in the JUnit test-report format (a common XML layout that CI tools read to display test results), and tags each run with the commit hash so a result traces to a change. It stores application programming interface (API) keys as secure environment variables. Keep keys away from code of unknown origin; the AI agent security guide covers why.

Step 6: Feed failures back as new test cases

A CI eval suite stays useful when every real failure becomes a new test case. Read the failing transcripts, decide whether the model or the test was wrong, and add the case. Remove or rewrite ambiguous cases where even humans would struggle to agree on an assessment (Claude Platform docs).

Capability evals with high pass rates can graduate into the regression suite that runs continuously to catch drift (Anthropic, 2026). The Claude Platform docs say to prioritize volume over quality: more automatically graded cases of slightly lower signal beat fewer hand-graded ones. Designing an eval for a new task starts this loop.

The Staff-level answer

What makes it Staff-level: it names the noise in the gate, ties the threshold to a baseline, and says which evals stay out of the per-change path.

The rubric

This rubric is the page's advice, not a company's scoring sheet.

Three variants of this question

How would you stop an LLM eval gate from being flaky?

Measure the noise first, then widen the gate to the baseline minus a margin, add cases or trials until the interval is narrower than the drop you must catch, and keep nondeterministic graders out of the blocking tier.

How do you keep eval cost under control in CI?

Tier the suite, trigger it only on files that affect it, cache model responses, and spend trials only where stability matters. The sums are calls per run, which is cases times trials.

What do you do when the judge model changes or drifts?

Pin the judge version, rerun the human-labeled calibration set against it, and compare the baseline before and after. Treat a changed judge as a new baseline, never as a regression.

Common mistakes

• "Run the full suite on every commit." (wrong) It makes the gate slow and costly, so people bypass it. (right) "Tier by trigger: a small gate per change, larger evals on a schedule."
• "Fail the build below 95 percent." (wrong) A round number ignores noise and the baseline. (right) "I set the gate from a baseline run minus a margin from the standard error."
• "One passing run proves it works." (wrong) Outputs vary between runs. (right) "I run several trials and pick pass@k or pass^k to match user expectations."
• "An LLM judge replaces the assertion." (wrong) It is non-deterministic and needs calibration. (right) "Code graders first; the judge is validated against human labels."
• "Check the exact path the agent took." (wrong) Rigid checks reject valid solutions. (right) "Grade the final outcome."

Follow-up questions

How many test cases do you need for a CI eval gate?

Enough that the interval is narrower than the smallest drop you must catch. At a true pass rate of 0.9 the interval is about plus or minus 8.32 points at 50 cases and 1.86 at 1,000.¹ If the cases cannot grow, add trials; widening the margin cuts false failures but lets more real drops through.

Should a failing eval always block the merge?

A regression eval should, because it should pass nearly every time and a drop signals breakage. A capability eval starts at a low pass rate by design, so it reports a trend on the schedule instead of blocking (Anthropic, 2026).

How do you run evals in GitHub Actions?

Trigger the workflow with on: pull_request and paths: filters, run the eval with a failing exit code, cache responses, and upload the results with if: always() (Promptfoo docs).

How do you handle API keys and fork pull requests in eval CI?

Store keys as secure environment variables and never give them to code from an untrusted fork, a copy of the repository made by someone outside the team whose pull request can carry unreviewed code. Run fork changes without secrets, or after a maintainer reviews them; the AI agent security guide explains the risk.

How does this scale to a platform that gates every change?

A shared service runs the tiers, stores baselines and results, and routes failures to owners. See designing an internal eval platform that gates every change.

Sources

1. Demystifying evals for AI agents — Anthropic Engineering, 9 Jan 2026
2. Define success criteria and build evaluations — Claude Platform Docs (Anthropic), 26 Oct 2026
3. CI/CD Integration for LLM Evaluation and Security — Promptfoo documentation, 26 Oct 2026