In this article
Before you start
This page is one question inside the LLM evaluation guide, which covers the neighboring topics.
Why interviewers ask this
Anyone can run a demo; the question is whether you can say if the product works today and whether it got better after a change. Interviewers use this question, reported by candidates in AI-engineering loops, to separate people who quote a leaderboard from people who measure their own system.
A weak answer names a public benchmark and stops. A strong answer says what "good" means for the task, where the test prompts come from, how answers are graded, and how sure the number is. A Staff-level answer adds the follow-up: a gain only counts if it is larger than the noise, and the grader itself must be checked.
The question has a second reading: "where do LLM applications stand today?" The method is the same. Public scores describe other tasks, so the state of your product is whatever you measure on your own prompts, with an error bar.
Evaluating an LLM application from first principles
- LLM application
- An LLM application is a product that wraps a language model in prompts, retrieved documents, tools or code so that it does one job for users, which means its quality depends on the whole pipeline and not on the model alone.
- Pass rate
- The pass rate is the share of eval-set prompts whose answers meet the success rule, written as a percentage, and it is the simplest score an application team reports.
In plain words
Think of a restaurant inspector. She does not taste one famous dish. She visits, orders what customers really order, and scores each plate against a checklist. She also knows two visits can differ by luck. Scoring every plate against a checklist beats tasting one dish, because the checklist covers many cases and can be repeated by someone else. The analogy breaks in one place: the inspector's checklist and menu stay fixed, but an LLM application changes whenever you edit a prompt, swap a model or update the documents it reads.
Step-by-step description
- Write the success rule and collect real prompts.
- Grade with code, humans and a checked judge.
- State the pass rate with its error bar.
- Sample production and feed failures back.
- Evaluation starts by defining what good means and collecting real prompts.
- Answers are scored with graders checked against human labels.
- The pass rate is reported with an error bar.
- Production traces are monitored and failures feed back into the prompts.
In one precise sentence, evaluating an LLM application means measuring task success on realistic inputs, with a grader you trust and an honest statement of uncertainty.
Step 1: Define what good means and collect real prompts
Evaluation starts by defining what good means and collecting real prompts. Start with the task, not the model. Write the success rule in terms a reviewer can check: the answer cites a supplied document, the code compiles, the refund amount is right, the tone is polite. A vague goal such as "helpful" cannot be scored.
Then collect prompts that look like real use: past user requests, support tickets, or traces from a pilot. Add the hard cases on purpose, such as ambiguous questions, long inputs and requests the application should refuse. This is the seed of a golden set. Input: the product's purpose and real traffic. Output: an eval set with a rule for each prompt.
Step 2: Score every answer with a grader you have checked
Three graders exist. Code checks exact facts, such as valid format or a correct number. Humans judge nuance but cost time. A judge model scales to thousands of answers. Zheng et al. (2023) report that strong LLM judges reach over 80% agreement with human preferences, the same level as humans agree with each other, and they also document position, verbosity and self-enhancement biases and limited reasoning ability.
So a judge is useful but must be calibrated: have humans label a sample, measure how often the judge agrees, and fix the judge prompt when it does not. The LLM-as-a-judge page shows how. Input: answers plus rules. Output: one pass or fail per prompt.
Step 3: Report the score with its error bar
A pass rate from a finite eval set is an estimate. A confidence interval is a range around that estimate that would contain the true pass rate in 95% of repeats of the same experiment, so it shows how much luck is in the number.
Now the follow-up. Versions A and B run on the same 200 illustrative prompts. A passes 150 (75.0%) and B passes 156 (78.0%), a 3.0-point gain.1 Because both versions saw the same prompts, compare them pairwise: B alone passes 16 prompts, A alone passes 10. The paired 95% interval of the gain runs from -2.0 to +8.0 points, so zero is inside it and the gain is not established.1 Treating the runs as independent would widen it to -5.3 to +11.3.1
How many prompts does each question need?
The number of prompts needed depends on the question you ask of the score. At a 75% pass rate the half-width of the 95% interval (normal approximation) is 12.0 points at 50 prompts, 6.0 at 200, 3.0 at 800 and 1.5 at 3,200.1 Pinning one pass rate to within 3 points at 75% takes about 801 prompts.1 Detecting a 3-point gain between versions takes about 1,134 paired prompts.1 Miller (2024) gives formulas for comparing two models and for sizing an evaluation. A paired comparison needs fewer prompts than two independent runs would, because both versions answer the same prompts and only the prompts where they disagree carry information about the gain. The table's single-rate intervals use the Wilson method.
| Quantity | Value (computed from illustrative prompts) |
|---|---|
| Version A pass rate | 75.0% (interval 68.6 to 80.5) |
| Version B pass rate | 78.0% (interval 71.8 to 83.2) |
| Gain of B over A | 3.0 points |
| Paired interval of the gain | -2.0 to 8.0 points |
| Prompts to estimate one pass rate to within 3 points at 75% | 801 |
| Paired prompts to detect a 3-point gain (80% power, 13% disagreement) | 1,134 |
Text description of the diagram
- At a 75% pass rate the half-width of the 95% interval is 12.0 points at 50 prompts.
- It is 6.0 points at 200 prompts, 3.0 at 800 and 1.5 at 3,200.
Step 4: Monitor production and feed failures back
An eval set describes yesterday's traffic. After launch, sample live traces, grade them with the same graders, and watch pass rate, refusals, latency and cost over time. Add each new failure to the eval set so the next version is tested on it. Safety behavior needs its own cases, such as prompt injection attempts and guardrail checks.
Public benchmarks still help when you choose a starting model. The Holistic Evaluation of Language Models (HELM) project of Liang et al. (2022) measures 7 metrics (accuracy, calibration, robustness, fairness, bias, toxicity and efficiency), a reminder that accuracy is one score among several. Your own eval set decides whether the application is good.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →The Staff-level answer
How do you evaluate the current state of an LLM application?
I start by writing down what a good answer means for the task, then build an eval set from real traffic plus hard cases. I grade with code where I can, humans on a sample, and a judge model at scale, after checking the judge's agreement with the human labels. I report the pass rate with an interval and compare versions pairwise, because a three-point gain on two hundred prompts can be noise. After launch I sample live traces and add failures to the set.
What makes it Staff-level: it names the grader's own error and the uncertainty of the score, not only the metrics.
The rubric
| Dimension | Weak answer | Strong answer | Staff answer |
|---|---|---|---|
| Correctness | Quotes a public leaderboard score | Defines success for the task and scores it | Defines success per slice and tracks it over versions |
| Depth | Says "use an LLM to grade" | Uses code, humans and a judge model | Checks judge agreement with human labels and fixes bias |
| Trade-offs | Ignores cost and speed | Weighs quality against cost and latency | Sets thresholds for each and names which to give up |
| Communication | Lists tools | Explains steps in order | States the number, its error bar and the decision it supports |
| Follow-up readiness | Cannot say if a gain is real | Mentions a larger test set | Compares versions pairwise and sizes the set for the gain it must detect |
Three variants of this question
How do you evaluate a retrieval-augmented generation (RAG) application?
Score retrieval and generation separately: did the right documents come back, and is the answer supported by them. The Ragas page covers metrics such as faithfulness for that split.
How do you evaluate an LLM agent?
Judge the final outcome against the task rule, then inspect the steps: which tools it called, whether it recovered from errors and what it cost. A strong answer also tests that it stops safely when it cannot finish.
Does a high public benchmark score mean the model suits my application?
No. A benchmark measures its own tasks, and a model can score well on it and still fail your prompts. Use it to shortlist models, then decide with your eval set; see LLM benchmarks and perplexity.
Common mistakes
- "I checked ten examples and it looks good." (wrong) Ten hand-picked examples show nothing about the rest. (right) "I run a fixed eval set of realistic prompts and report the pass rate with its interval."
- "Version B scored higher, so ship it." (wrong) A small gap on a small set can be noise. (right) "I compare versions on the same prompts and ship only when the interval excludes zero."
- "The judge model is objective." (wrong) Judges have documented biases. (right) "I measure the judge's agreement with human labels first."
- "Accuracy is the metric." (wrong) One score hides cost, safety and robustness. (right) "I track several metrics and name the trade-offs."
- "We evaluated at launch." (wrong) Traffic and models change. (right) "I keep grading sampled production traces and add failures to the set."
Follow-up questions
Is a three-point gain on 200 prompts real?
Not on its own. In the example above the paired interval of the gain runs from -2.0 to +8.0 points, so zero is plausible. Add prompts or look for a bigger effect.
Can an LLM judge replace human graders?
Not entirely. Zheng et al. (2023) report over 80% agreement with human preferences for strong judges, but also known biases, so keep a human-labeled sample to calibrate and audit the judge.
How many test prompts do you need?
It depends on the question. To pin one pass rate to within 3 points at 75%, the normal approximation gives about 801 prompts.1 To detect a 3-point gain between two versions on the same prompts, with 80% power and the 13% disagreement of the worked example, about 1,134 paired prompts are needed.1 Hard slices need their own prompts, since a slice of 50 prompts has the widest bar, and why models hallucinate helps in picking failure cases.
What do you monitor after launch?
Sample live traces and grade them with the same graders, then watch pass rate, refusals, latency and cost by slice. Alert on a drop larger than the interval of the monitored sample, since its own size sets that interval.
How do you stop a prompt change from breaking old behavior?
Run the full eval set on every change and compare with the last accepted version, the practice behind golden sets and regression testing. Teams often automate this as a gate, as in an internal eval platform that gates every change.
- Define success for your task first, then collect real prompts plus hard cases.
- Grade with code, humans and a judge model, and check the judge against human labels.
- Report the pass rate with an interval; compare versions pairwise on the same prompts.
- Public benchmarks shortlist models; your own eval set decides.
- Keep evaluating after launch and feed failures back into the set.
Sources
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (arXiv 2306.05685) — arXiv (Zheng et al.), 9 Jun 2023
- Holistic Evaluation of Language Models (arXiv 2211.09110v2) — arXiv (Liang et al.), 1 Oct 2023
- Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations (arXiv 2411.00640) — arXiv (Miller), 1 Nov 2024
