In this article
Before you start
This page belongs to the data science case study interviews path. The data science interview questions page lists neighbouring prompts.
Why interviewers ask this
An answer that names only "accuracy" fails the first follow-up, because the interviewer then asks what happens to speed, safety and price. Candidates report prompts such as how to define key performance indicators (KPIs) for a generative AI feature, how to convince a non-technical stakeholder of the return, and what to do when a model is ten times more capable and ten times more costly.
These questions separate people who treat a model as a score from people who treat it as a product. A weak answer lists metrics with no thresholds. A strong answer picks a few criteria, says how each is tested, and names the trade-off between them. A Staff-level answer also says how the pre-launch numbers will be checked against real users. For how these questions appear outside large technology companies, see the domain case rounds outside big tech.
Measuring AI products from first principles
- Success criterion
- A success criterion is a target for one aspect of an AI product, written so that it is specific and measurable, such as a score threshold on a named test set. A product usually needs several, because quality, safety, speed and price can each fail while the others look fine.
- Cost per useful outcome
- Cost per useful outcome is the cost of one request divided by the share of requests that succeed. It prices what the user actually receives, so a cheap model that fails often and an expensive model that rarely fails can be compared on one scale.
In plain words
Think of a car dashboard. It shows speed, fuel and engine temperature, because no single dial tells you whether the car is fine. Measuring an AI product works the same way. The analogy breaks in one place. A dashboard has fixed dials, but for an AI product you must choose the dials, set the warning levels and build the test track yourself.
In one precise sentence, measuring an AI product means choosing several measurable criteria, testing them on an evaluation set that looks like real use, weighing them against cost, and confirming the result with live users. The Anthropic documentation on success criteria (accessed 2026-10-26) puts it this way: "Most use cases need multidimensional evaluation along several success criteria."
Step 1: Define success criteria
A success criterion turns "good performance" into something a test can fail. The Anthropic guidance (accessed 2026-10-26) asks for criteria that are specific, measurable, achievable and relevant. Its usable example is an F1 score of at least 0.85 on a held-out set. The F1 score is the harmonic mean of precision (the share of positive calls that were right) and recall (the share of real positives the model found).
Metrics fall into three kinds: task-specific, generic and operational, as the same page groups them. Task-specific metrics include the F1 score and perplexity, generic ones include accuracy, precision and recall, and operational metrics include response time and uptime.
Weight the criteria by the use case. The page notes that strong citation accuracy can matter for a medical app and matter less for a casual chatbot. Severity matters too: its sentiment example requires at least 90% of errors to be minor inconveniences, so at most 10% may be serious.
Text description of the diagram
- Metrics fall into three kinds: task-specific, generic and operational.
- Operational metrics include response time and uptime.
You now have a scorecard. The next step is the test that fills it in.
Step 2: Build the evaluation set
An evaluation set is only useful if it looks like the real work. The Anthropic guidance (accessed 2026-10-26) says to design evaluations that mirror your real-world task distribution, and to include edge cases. It names empty or irrelevant input, very long input, harmful input, and ambiguous cases where humans disagree.
Grading should be automatic wherever possible. The page lists multiple-choice, string match, code-graded and graded-by-a-large-language-model (LLM) questions, and it favours more automated questions over fewer hand-graded ones. An LLM grader is a second LLM that scores an answer against a written rubric. Check it against a sample of human-graded answers before you trust it, because a grader can be wrong in a consistent direction; Zheng et al. (2023) study such judge biases and how well model judges agree with people.
Input: real-looking requests with a grading rule. Output: one score per criterion, run on the same examples every time. That sameness lets you tell whether a new prompt or model helped.
Step 3: Put cost next to capability
Cost per useful outcome belongs beside capability because a model's price per request says nothing about how many requests actually succeed. The Anthropic list of common criteria (accessed 2026-10-26) includes latency and price, so they are criteria in their own right, not afterthoughts.
Take two illustrative models. Model A costs 1 cost unit per request and succeeds on 70% of requests, while Model B costs 10 units and succeeds on 85%.1 On quality alone Model B wins. Per useful outcome, Model A costs 1.43 cost units and Model B costs 11.76, which is 8.24 times more for 1.21 times the successes (illustrative).1 The 15 extra successes per 100 requests that Model B delivers cost 60 units each (illustrative).1
Counting success rates narrows the tenfold price gap to 8.24 times, but it does not remove it. The decision rests on the 60: with a value V per success, Model B is ahead when 0.85V − 10 is above 0.70V − 1, which is when V is above 60 cost units (illustrative).1 That is the return on investment (ROI) question, and it is why the cost step comes before any launch decision.
Success rates can also reverse the order. An illustrative cheap model at 1 unit per request with 5% success costs 20 units per useful outcome, more than Model B's 11.76.1
Text description of the diagram
- Model A costs 1.43 cost units per useful outcome and Model B costs 11.76.
- Model B costs 8.24 times more for 1.21 times the successes.
Step 4: Confirm with live users
A pre-launch score measures the test set, not your users. The Anthropic guidance (accessed 2026-10-26) pairs evaluations with two live methods: A/B testing against a baseline or earlier version, and implicit user signals such as task completion rates. An A/B test is an experiment that sends similar users to the old and new versions at random and compares an outcome metric between the groups; Kohavi, Tang and Xu (2020) is a widely used reference on running them; the reading an A/B test like a product owner page walks through interpreting one.
Pick the live outcome metric before the test starts, and keep the earlier criteria as limits. If task completion rises but the safety criterion or the cost per useful outcome breaks its threshold, the change does not ship as it is.
The four steps run in order: define criteria, build an evaluation set, set cost beside capability, then run a live A/B test. Pre-launch scores come before the live A/B test, and the live result decides.
Step-by-step description
- Write measurable success criteria.
- Test on inputs that mirror real use.
- Compare cost per useful outcome.
- Confirm with an A/B test.
- The four steps run in order: define criteria, build an evaluation set, set cost beside capability, then run a live A/B test.
- Pre-launch scores come before the live A/B test.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
The Anthropic worked example (accessed 2026-10-26) holds a sentiment-analysis model to an F1 score of at least 0.85, 99.5% non-toxic outputs, and 95% of responses under 200 milliseconds, measured on a held-out set of 10,000 posts. Each percentage becomes an auditable count. A 99.5% non-toxic target allows at most 50 toxic outputs in 10,000, and a 95% speed target allows at most 500 responses of 200 milliseconds or longer.1 The page's safety example, fewer than 0.1% of 10,000 outputs flagged, means fewer than 10 flagged outputs.1
Text description of the diagram
- At most 50 of 10,000 outputs may be toxic under a 99.5% non-toxic target.
- At most 500 responses may take 200 milliseconds or longer.
The source also sets an F1 target of a 5% improvement over the baseline without saying whether that is relative or absolute. State which one you mean when you copy the pattern.
Now the live step, with illustrative numbers. Two equal groups, each of illustrative size 5,000, see the old and the new version; task completion is 62.0% for the old and 64.5% for the new, a gain of 2.5 points or 4.03% relative.1 A two-proportion z statistic for that gap is 2.59, above 1.96, the usual cutoff for a two-sided 5% test, so the gap is unlikely to be chance.1 To translate it for a stakeholder, 2,000 assisted tasks a month at 5 minutes saved each is 166.67 hours (illustrative).1
| Quantity | Value |
|---|---|
| Held-out posts in the source's example | 10,000 |
| Toxic outputs allowed at 99.5% non-toxic | 50 |
| Responses allowed at or over 200 milliseconds | 500 |
| Flagged outputs allowed at under 0.1% | fewer than 10 |
| A/B completion, old against new (illustrative) | 62.0% against 64.5% |
| Relative lift (illustrative) | 4.03% |
| z statistic (illustrative) | 2.59 |
| Hours saved by 2,000 tasks at 5 minutes (illustrative) | 166.67 |
Trade-offs and edge cases
Criteria conflict, and the page's own advice is to weight them by use case. A stronger model may add latency, and a safer filter may lower task quality. Write the order of priority down before the tests run, so the decision is not made after seeing the scores. Cost per useful outcome also assumes a failure costs nothing beyond the request; retries or a human fallback add to the cost of each failure and raise the figure.
Three edge cases catch candidates. First, an LLM grader can carry its own bias, so sample its grades against people. Second, an evaluation set goes stale when real traffic changes, so refresh it from recent requests, edge cases included. Third, pre-launch and live results can disagree, because the test set is only a sample of use. The source pairs evaluations with A/B tests and implicit signals for this reason.
Current practice adds a few habits. Teams track guardrail metrics, limits that must not worsen while the lead metric improves, and they watch live traffic for drift, where inputs change and scores fall. Hallucination rate, the share of answers with unsupported claims, is a common quality criterion, though its definition differs between teams.
For recommendation products, such as the video-recommendation questions candidates report, the same pattern applies: pre-launch ranking checks, then a live comparison with limits on harm and cost. Practice differs between teams on which single outcome metric leads, so say which one you choose and why. The watch-time investigation walkthrough shows how a live metric is diagnosed when it moves.
How to answer it in an interview
How would you measure the success of an AI product?
I would not pick one number. I would write a few criteria, each with a threshold and a test set: task quality, a safety limit, latency, and cost per useful outcome. I would build an evaluation set that mirrors real traffic plus edge cases, grade it automatically, and check the grader against a human sample. Then I would run an A/B test against the current version and watch task completion. The trade-off: a model ten times costlier is worth it only if each extra success it delivers is worth more than it costs. In my numbers that is 60 cost units per extra success, so I would also route easy requests to the cheaper model.
The spoken structure: name the criteria, say how each is tested, bring in cost per useful outcome, then confirm live.
Common mistakes
- "Accuracy is our metric." (wrong) One number hides safety, speed and price. (right) "I would track several criteria, weighted by the use case."
- "Safety cannot be measured." (wrong) The source gives a measurable safety criterion. (right) "I would set a limit on flagged outputs over a fixed test set."
- "Hand-graded answers are always better." (wrong) A few hand-graded items cover little. (right) "I would automate grading for volume and spot-check it with people."
- "The best model on the benchmark wins." (wrong) It ignores price. (right) "I would compare cost per useful outcome and then the live result."
Follow-up questions
What KPIs would you define to measure generative AI solution success?
Pick one outcome metric, such as task completion, and add limits for quality, safety, latency and cost per useful outcome. Give each a threshold and a test set, as in the success-criteria guidance.
How would you convince a non-technical stakeholder about ROI of generative AI?
Translate the result into their units: hours saved or tasks completed, against the cost per useful outcome. Show the A/B lift first and the model scores second, and state which numbers are measured and which are assumed.
What are the north star success metrics for an AI agent for a healthcare financial platform?
Lead with one outcome metric, such as resolved requests (a north star metric is the one outcome metric the team optimises, with the others as limits), and weight the limits toward correctness and privacy, since the source notes that criteria should be weighted by use case. Add a safety threshold and a cost limit.
How would you evaluate YouTube's video recommendations?
Combine pre-launch ranking checks with a live A/B test, and add limits for harm and cost. The product analytics case path and the ads and marketplace domain questions show related search and ranking cases.
Ten times the model capability, ten times the cost: what do you do?
Compute cost per useful outcome and the cost of each extra success, then ask what a success is worth. Route easy requests to the cheaper model if the quality limits still hold.
- Measuring AI products takes several criteria, each with a threshold and a test set.
- Metrics are task-specific, generic or operational, and are weighted by use case.
- An evaluation set mirrors real traffic and includes edge cases; grade it automatically and check the grader.
- Compare models on cost per useful outcome, not on price or score alone.
- Confirm pre-launch results with an A/B test and implicit user signals.
Sources
- Define success criteria and build evaluations (page carries no publication date; accessed 2026-10-26) — Anthropic (Claude Platform Docs), 26 Oct 2026
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (arXiv 2306.05685) — arXiv (Zheng et al.), 9 Jun 2023
- Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing — Cambridge University Press (Kohavi, Tang, Xu), 2 Apr 2020
