In this article
Before you start
This page is one step of the LLM evaluation learning path.
Why interviewers ask this
The prompt appears in candidate reports from interviews at AI labs and platform companies: build evaluation infrastructure for many benchmarks and many models, and say how you handle test-set contamination and reproducibility. Treat those reports as reports, not as a public statement of what any company asks.
The question separates people who can call a model from people who can trust a number. A weak answer lists benchmark names and a script that loops over them. A strong answer pins every setting that changes a score, checks whether the test set leaked into training, and says how much of a score gap is noise. The authors of the lm-eval framework wrote that reliable evaluation of language models remains an open challenge, with sensitivity to evaluation setup and weak reproducibility among the causes (paper dated 2024, last revised May 2026). A Staff-level answer adds cost control and untrusted code.
Benchmark harness from first principles
- Benchmark harness
- A benchmark harness is a program that runs one or more language models over a fixed set of benchmark tasks under pinned settings and records every prompt, output and score, so that any result can be reproduced, audited and compared with another run.
- Test-set contamination
- Test-set contamination is the presence of benchmark questions, answers or close copies of them in a model's training data, so that a high score may reflect recall of the test rather than skill. It covers verbatim copies and lightly rewritten ones.
- Paired comparison
- A paired comparison is a way to compare two models by scoring both on the same questions and analysing the per-question differences, which removes the question-difficulty noise the two models share and gives a smaller error than comparing two separate averages.
In plain words
Think of an exam board. Every candidate gets the same paper, the same time limit and the same marking scheme, and the paper is kept sealed until the day. A harness plays the exam board for models. The analogy breaks in one place: a human has not usually read the exam paper in advance, but a model trained on web-scale text may have, and the board cannot simply ask it.
In one precise sentence, a harness is a function from (task version, model version, run settings) to a logged set of per-question results, plus a statistic with an error bar. Everything below makes that function repeatable.
Step 1: Freeze each task and its run settings
A task definition is a versioned record holding the dataset revision, the prompt template, any worked examples shown to the model, the sampling settings and the scoring rule. A task registry stores these records, and a run manifest stores which versions one run used, with a hash of the whole configuration.
Input: a benchmark. Output: an immutable task version. This matters because small setup changes move scores: the lm-eval developers document sensitivity to evaluation setup and difficulty of comparing methods as core problems. "Same benchmark name" therefore does not mean "comparable score". You now know what a score is attached to: a task version and a manifest, never a bare name.
Step 2: Screen the test set for contamination
Screening asks whether each test item, or a close copy, appears in the training data; the generative pre-trained transformer (generative pre-trained transformer (GPT)-3) paper (Brown et al., 2020), which measured 13-gram overlap between each benchmark and its training data, is an early example of a lab reporting this check. Where you can read the training corpus, build an index of n-grams and flag test items that share one. An n-gram is a run of n consecutive tokens, the small pieces a model reads. Where you cannot read it, as with a model from another lab, use indirect signals: compare accuracy on the original items with accuracy on freshly written rewrites, and keep a private held-out slice that was never published.
The choice of n is a trade-off. In a toy test with 200 items, where each "lightly edited" copy had one token in every six replaced, an 8-gram screen flagged all 30 verbatim copies but none of the 20 edited ones, because no 8 consecutive tokens survived the edit, while a 4-gram screen flagged all 50.1 A shorter window catches rewrites but, on real text with common phrases, also raises false alarms, so review flagged items rather than deleting them blindly. Published screens use long windows, such as the 13-grams of the GPT-3 paper, precisely to keep false alarms low, so they pair the screen with rewrites and held-out data. Output: a contamination flag per item.
Text description of the diagram
- In the toy test an 8-gram screen flags 30 verbatim copies and none of the 20 edited items.
- A 4-gram screen flags all 30 verbatim copies and all 20 edited items.
Step 3: Run the model through a pinned, cached, sandboxed adapter
A model adapter is one interface that hides how each model is served: text in, text and optional token probabilities out. Pin the model version and the sampling settings, and write each response to a cache keyed by a hash of the model, prompt and settings. A rerun then returns identical outputs and costs nothing, and a crashed run resumes where it stopped.
Schedule by cost: group requests by model so each model is loaded once, cap concurrency per model, batch small models together on one device, and shard large models across several devices, since model sizes vary widely. Closed application programming interfaces (APIs) often expose no token probabilities, so tasks scored by the log-likelihood of each multiple-choice option cannot run through the adapter and need a generation-based variant. For tasks that execute model-written or community-submitted code, run it in a sandbox. A sandbox is an isolated, disposable environment with no network, hard limits on time and memory, and no access to the host. The AI agent security page covers why untrusted input needs that boundary. Output: raw responses, logged.
Step 4: Score, then report an error bar
The scorer turns each response into a mark: exact match for short answers, unit tests for code, or a large language model (LLM) acting as the marker, called LLM-as-a-judge, which must itself be checked against human marks. Store the per-question marks, not only the average, because the statistics need them.
A score without an error bar is a number without a claim. The standard error of an accuracy score is the typical amount it would move if you drew a fresh set of questions from the same source, and a 95 percent confidence interval is a range built so that, over repeated question sets, 95 percent of such ranges contain the true accuracy. Miller (2024) treats evaluations as experiments and gives formulas for analysing results, comparing two models and planning how many questions to run. Report the interval, and compare models with a paired comparison. When questions come in groups, such as several questions on one passage, they are not independent, and Miller recommends clustered standard errors.
Step-by-step description
- Each task is a pinned, versioned record.
- Check items for contamination first.
- A cached adapter runs it; code goes in a sandbox.
- Report each score with a standard error.
- A task registry pins each task version and its run settings.
- A contamination screen checks the test set before the run.
- A pinned, cached model adapter runs the model, and untrusted code runs in a sandbox.
- The scorer reports each score with a standard error.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
Take a model with 80 percent accuracy on 1,000 questions (illustrative). Its standard error is 1.26 points, so the 95 percent interval is about 2.48 points either side.1 A second model scores 81 percent on the same 1,000 questions (illustrative): both are right on 740, only the second on 70, only the first on 60, and both wrong on 130.1
Is the one-point gap real? Comparing two separate averages gives a standard error of 1.77 points. Pairing the questions gives 1.14 points, and the paired interval for the gap runs from −1.23 to 3.23 points, which includes zero.1 The gap is not distinguishable from noise. To detect a gap of this size with 80 percent power, the same discordance pattern needs about 10,185 questions, assuming a two-sided test at the 5 percent level.1
| Quantity | Value (computed from illustrative inputs) |
|---|---|
| Standard error of 80% accuracy on 1,000 questions | 1.26 points |
| 95% interval half-width | 2.48 points |
| Standard error of the gap, unpaired | 1.77 points |
| Standard error of the gap, paired | 1.14 points |
| Paired 95% interval for the gap | −1.23 to 3.23 points |
| Questions needed for 80% power at this gap | 10,185 |
| Requests for 100 benchmarks × 20 models × 2,000 items | 4,000,000 |
| Input tokens at 800 per request | 3.2 billion |
| Output tokens at 400 per request | 1.6 billion |
The request budget shows why the cache and the scheduler from Step 3 are not optional extras.1
Text description of the diagram
- The standard error of the gap is 1.77 points unpaired.
- The standard error of the gap is 1.14 points paired.
Trade-offs and edge cases
Contamination screening needs the training data, which only the model's owner has. A lab evaluating its own model can index the corpus; an outside harness must rely on rewrites, private held-out slices and freshly written questions, and say so in the report.
Pinning has a cost. Fixing a prompt format makes runs comparable but may under-sell a model that prefers another format, so record the format and, where it matters, report more than one. Greedy decoding is the most repeatable setting, but outputs can still differ across hardware, batch sizes or provider updates, which is why the harness caches and logs every response rather than trusting a rerun to match. Sampled outputs need several runs and an interval.
Marker models add their own bias and drift when the marker is updated, so pin the marker version too. Public test sets leak over time, so mix public benchmarks with private ones. As of October 2026 the lm-eval developers' paper, revised in May 2026, still frames reliable evaluation as an open problem, so state what your harness does not control. Finally, a harness measures a model on a fixed set; how it behaves with real users is a separate question, taken up on the LLM evaluation path. To gate every change on such a harness, see how to design an internal eval platform.
How to answer it in an interview
Design an evaluation harness for a frontier model across many benchmarks. How do you handle contamination and reproducibility?
I would build one pipeline with four stages. Each task is a versioned definition, so prompt, examples, sampling settings and scorer are pinned and hashed. Before running, I screen each test set for contamination, using n-gram overlap where I can see the training data and rewritten or private held-out variants where I cannot. Model calls go through one cached adapter, with untrusted code in a sandbox. I log every output and report each score with a standard error, comparing models with a paired test, so a one-point gap counts only if the interval excludes zero.
The spoken structure: pin the task, screen the test set, run through a cached adapter, then report with an error bar.
Common mistakes
- "The benchmark name is enough to compare two scores." (wrong) Setup changes move scores. (right) "I compare scores only when task version, prompt format and scoring match, and I record them."
- "Model A scored higher, so A is better." (wrong) The gap may be inside the noise. (right) "I report an interval and use a paired comparison on the same questions."
- "I check contamination with exact string matching." (wrong) Lightly edited copies slip through a long window. (right) "I use shorter windows, rewrites and a private held-out slice, and review the flags."
- "I rerun the whole suite whenever something changes." (wrong) That wastes budget. (right) "Cached responses keyed by model, prompt and settings make reruns cheap."
Follow-up questions
How do labs evaluate frontier models?
Reported practice differs. One documented example is the GPT-3 paper, which measured 13-gram overlap with its training data and reported results on cleaned subsets. The reliable core is the one on this page: pinned tasks, contamination checks, logged outputs and statistics, with private test sets alongside public ones.
How do you handle test set contamination?
Screen items against the training data with n-gram overlap when you can read it, and use rewritten variants and a private held-out slice when you cannot. Report flagged items separately instead of silently dropping them.
How do you ensure reproducibility?
Version the task, pin the model and sampling settings, cache responses by a hash of the inputs and store a manifest with every run. Anyone holding the manifest can rerun it.
How would you run many benchmarks across many models without the bill exploding?
Cache responses, group requests by model, cap concurrency and run cheap checks first. Estimate tokens before launching: the illustrative suite above reaches 4,000,000 requests.1
What changes when the code comes from untrusted community submissions?
Run every submission in a disposable sandbox with no network and hard time and memory limits, and never give it your credentials. Treat its output as untrusted data.
How is this different from evaluating a retrieval-augmented system?
A retrieval-augmented generation (RAG) system adds a retrieval step whose quality must be scored too; the Ragas page covers those metrics. The harness skeleton stays the same.
- A harness maps a pinned task, model and settings to logged per-question results.
- Screen test sets for contamination, and use short windows or rewrites to catch edited copies.
- Cache responses by input hash for cheap, identical reruns, and sandbox untrusted code.
- Report every score with a standard error and compare models with a paired comparison.
- A one-point gap on 1,000 questions can be noise.
Sources
- Lessons from the Trenches on Reproducible Evaluation of Language Models (arXiv 2405.14782) — arXiv (lm-eval developers), 23 May 2024
- Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations (arXiv 2411.00640) — arXiv (Miller), 1 Nov 2024
- Language Models are Few-Shot Learners (arXiv 2005.14165) — arXiv (Brown et al.), 28 May 2020
