In this article
Before you start
The interview process and loops page covers the other rounds. This page assumes no statistics beyond the three ideas below.
Why interviewers ask this
The eval take-home is reported by candidates as a notebook plus a link to an academic article that describes the task; one candidate report says the notebook asked for work on HellaSwag, a sentence-completion benchmark. The questions after the code are then open: what do you think of the scores, what do they say about gender bias and fairness, and would you generate more data? Other reported variants ask you to build a small evaluation harness, to prune a benchmark to a small sample, to prove a flaky prompt pipeline improved, or to design the evaluation for a broken retrieval system. We found no public description of how companies grade this round, so read the rest as advice.
A weak answer repeats the headline number and says it looks good. A strong answer says how sure the number is, what it hides and what it cannot show. The same habit helps in a project deep dive and in the applied AI engineer rounds.
The eval take-home from first principles
- Slice
- A slice is a subset of the test set that shares one property, such as a user group, a topic or a difficulty level, so that you can compute the same metric for that subset and compare it with the rest.
In plain words
Think of a restaurant inspector who tastes five dishes and reports "the kitchen is excellent." Five dishes could be a lucky sample, and the inspector never tasted the dessert menu. The analogy breaks in two places. Dishes differ in kind, while test items share one format. The untasted dessert menu stands for the slice of cases the test set never covers.
In one precise sentence, an evaluation answer states a score, the uncertainty around it, the subsets where it differs and the data needed to settle what is still open.
Step 1: Put an interval on every score
A score is a sample estimate, so report it with a confidence interval. For accuracy, the standard error, the typical size of the chance wobble in a score, is , where p is the accuracy and n the number of test items. The 95% interval is the score plus or minus 1.96 standard errors, because 1.96 is the 97.5th percentile of the normal distribution. Miller (2024) describes standard errors for eval scores on this central-limit-theorem basis.
Input: a score and an item count. Output: a score with a range. Take an illustrative model at 80% on 500 items: the standard error is 1.79 points, so the interval runs from 76.5% to 83.5%.1 A bootstrap (Efron, 1979) is a method that re-draws 500 items with replacement from the test set many times and reads the middle 95% of the resulting scores; it gives 76.4% to 83.4% over 2,000 resamples, so the two methods agree.1
Comparing two models is a different question. If the two scores came from separate test sets (unpaired), the standard error of the gap combines both: for 80% and 82% on 500 items each, the gap is 2.0 points with an interval of −2.9 to +6.9.1 When both models are scored on the same items, compute the per-item difference, or use McNemar's test on the items where the models disagree. Shared easy and hard items cancel, so this interval is usually narrower and a 2-point gap can become significant; Miller (2024) covers paired differences.
You now know what one number is worth. The next step asks what it hides.
Step 2: Slice the same data and compare groups
Overall accuracy is an average, and an average can hide a gap. Compute the same metric on each slice and compare. Take illustrative groups A and B with 250 items each: A scores 84% and B scores 76%, so the overall score is 80% and the gap is 8 points.1
Apply Step 1 to the gap as well. Its standard error is 3.6 points, so the 95% interval for the gap runs from 1.0 to 15.0 points.1 The gap is unlikely to be zero, but its size is poorly pinned down. Say exactly that, and say what the slice cannot show: an accuracy gap is a measured difference, not a cause. For small slices, the Wilson interval behaves better than the normal approximation used here. If the notebook has no group labels, it cannot support a group-gap claim, though counterfactual prompts (Step 3) can still probe bias.
Step 3: Name the data that would settle the question
After the intervals and slices, say what you would add. Three kinds of data help. A larger held-out set narrows the interval: for a half-width of one point at 80% accuracy you need about 6,147 items.1 Counterfactual prompts, pairs of items that differ only in one group term, test bias more directly than a mixed set. Adversarial or hard items, selected where current models fail, restore the headroom that an easy set has lost; HellaSwag's own construction is an example.
Zellers et al. (2019) built HellaSwag so that its questions are easy for people (above 95% accuracy) while the models they tested scored below 48%. Those are 2019 figures and say nothing about current models. They used Adversarial Filtering, in which a series of discriminators selects machine-generated wrong answers that fool models, so a benchmark's difficulty is a property of its era. Its text has been public since 2019 and may sit in current training data, which would inflate scores. Multiple-choice tasks like it are commonly scored by comparing the model's log-likelihood of each ending, often length-normalised, so check how the notebook scores it.
End with the decision: what you would conclude today, what you would not, and the cheapest next measurement. A raw score gets a confidence interval, then group slices come next, then the new data that would settle the open question.
Step-by-step description
- Put an interval on the raw score.
- Compare the same metric across groups.
- Name the data that settles the rest.
- A raw score gets a confidence interval.
- Group slices come next, then the new data that would settle the open question.
| Quantity | Value (computed from illustrative inputs) |
|---|---|
| Accuracy, model A, 500 items | 80% |
| 95% interval for model A | 76.5% to 83.5% |
| Bootstrap interval, 2,000 resamples | 76.4% to 83.4% |
| Gap, model B (82%) minus model A, separate test sets (unpaired) | 2.0 points, interval −2.9 to +6.9 |
| Items for a one-point half-width at 80% | 6,147 |
| Gap, group A minus group B | 8.0 points, interval 1.0 to 15.0 |
Text description of the diagram
- Illustrative overall accuracy is 80%.
- Group A scores 84% and group B scores 76%, an eight-point gap.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →The Staff-level answer
Based on the scores you have, what do you think of the model, and what would you do next?
I would not read the headline score alone. First I put an interval on it: at 80% accuracy on 500 items, in my illustrative case, the interval runs from about 76.5% to 83.5%, so a model at 82% on a separate test set is not clearly better; on the same items I would run a paired comparison. Then I slice the same test set by group and item type, because an overall number hides gaps. Finally I name the data I would add: a larger held-out set, counterfactual prompts that change one group term, and hard cases. I would also say what these scores cannot show.
What makes it Staff-level: it turns a number into a claim with a stated size, then states its own limits and the cheapest next test.
The rubric
| Dimension | Weak answer | Strong answer | Staff answer |
|---|---|---|---|
| Correctness | Repeats the headline score | Computes the metric and checks it | Checks how the metric was scored and on which items |
| Uncertainty | None | Gives an interval | Runs a paired comparison on the same items and sizes the test set |
| Fairness | "Looks fair overall" | Compares groups | States the metric, the interval and why a gap is not a cause |
| Trade-offs | None | Names a limit | Names what each extra dataset would and would not settle |
| Communication | Long code, no conclusion | Short summary first | Decision, evidence, next step in one paragraph |
Three variants of this question
Based on the available scores, what do you think of the model performance?
Say the score, its interval and the size of the test set, then compare against a baseline or the other model on the same items. Add what the benchmark measures and when it was built, because difficulty is tied to its era.
Based on the available scores, what do you think of the model's gender bias and fairness compliance?
Compute the same metric per group and report the gap with an interval, as in Step 2. State which fairness definition you use (equal accuracy, equal error rates and so on), and say that overall accuracy cannot show fairness. Without group labels you cannot compute group gaps, but counterfactual prompts that swap a group term can still probe bias. Compliance needs a stated threshold from the policy you are judged against.
If you had access to the model, would you generate any other data to expand the evaluation?
Yes: a larger held-out set to narrow the interval, counterfactual pairs that change one group term, and hard items from where the model fails. Say which question each set answers, so the extra work is justified.
Common mistakes
- "The model scored 82% and the other 80%, so it is better." (wrong) Two scores alone do not show a difference. (right) "On the same items I would run a paired comparison before ranking them."
- "Overall accuracy is high, so the model is fair." (wrong) Overall accuracy is an average over groups. (right) "I would compute accuracy per group and report the gap and its interval."
- "The paper says models score below 48%, so this is a hard benchmark." (wrong) That is a 2019 figure. (right) "In 2019 the authors reported below 48%; I would check how current models score before calling it hard."
- "I would collect more data." (wrong) It names no purpose. (right) "I would add counterfactual pairs to test gender bias directly."
Follow-up questions
How many test items do you need to trust a score?
It depends on the precision you want. For a one-point half-width at 80% accuracy you need about 6,147 items, and halving the width needs four times as many.1
Does a high HellaSwag score prove a model is good?
No. The score measures one kind of commonsense completion, and the benchmark was built in 2019 to be hard for the models of that time (Zellers et al., 2019). Treat it as one signal among several.
How would you design an evaluation for a retrieval-augmented system that is broken?
Retrieval-augmented generation (RAG) first retrieves passages and then writes an answer from them. Build a question set with known source passages, and measure retrieval and answer correctness separately, so the report says where the system breaks.
How would you build a small evaluation harness?
Use a fixed test file with one input and expected output per line, and pluggable scorers (exact, substring, fuzzy). Pin the temperature and seed, repeat runs, cache responses, save per-item results, and log latency so slow responses are flagged. To prove a flaky pipeline improved, run both versions on the same items and compare them as paired results. For more questions of this kind, see the AI engineer question list.
How would you shrink a benchmark without losing the signal?
Pick the smallest subset whose ranking of several models matches the full set's, then size it with the Step 3 formula so its interval is narrow enough for the decision you must make.
What should you hand in besides the notebook?
A short written summary first: the decision, the evidence with intervals, the limits and the next measurement. Reviewers can then see your judgement without running anything. The same summary-first habit helps in the agent take-home and the RAG take-home. If you are unsure what tools a round allows, read the AI-assisted coding interview page.
- The eval take-home asks you to judge scores, check fairness and propose more data.
- Report every score with an interval; a gap smaller than the interval is not a result.
- Slice by group, and state that an accuracy gap is not a cause.
- Name the extra data and the question each set answers.
- Lead the write-up with the decision and its limits.
Sources
- HellaSwag: Can a Machine Really Finish Your Sentence? (arXiv 1905.07830) — arXiv (Zellers, Holtzman, Bisk, Farhadi, Choi; ACL 2019), 19 May 2019
- Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations (arXiv 2411.00640) — arXiv (Miller), 1 Nov 2024
- Bootstrap Methods: Another Look at the Jackknife — Annals of Statistics (Efron), 1 Jan 1979
