What are LLM product success metrics?

In this article

1. Before you start
2. Why interviewers ask this
3. LLM product success metrics from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step of the generative AI system design interview guide learning path.

Why interviewers ask this

A design answer that ends with "and we will track accuracy" leaves the interviewer unable to tell whether the system works. Success metrics turn a design into something a team can run, tune and defend, so the question usually closes a generative AI design round or comes up when you propose any change.

A weak answer lists one number, such as thumbs-up rate or tokens per second, and stops. A strong answer separates what users do, what users get, how fast they get it and what it costs, then gives each a target. A Staff-level answer adds the limits: every metric is a proxy, held-out scores can fall once real users arrive, and an improvement in one layer must not hide a regression in another. This page builds that answer from the ground up, using the stage-by-stage retrieval-augmented generation (RAG) evaluation page as the companion for retrieval quality.

LLM product success metrics from first principles

Persistence rate

Persistence rate is the share of accepted model outputs that are still unchanged in the user's work after a set time, such as 30 or 600 seconds. It separates output that users took from output that users kept, so it is a stricter signal of lasting value than acceptance alone.

Time to first token (TTFT)

Time to first token is the time between submitting a prompt and receiving the first token of the model's response. It covers queueing and reading the whole prompt, so it grows with prompt length and load, and it sets how soon the product feels responsive.

Time per output token (TPOT)

Time per output token is the average time between tokens after the first one, found as end-to-end latency minus time to first token, divided by the output tokens minus one. It sets how smooth streaming feels once the answer has started.

In plain words

Think of a restaurant. Orders served tell you how many people use it. Plates returned untouched tell you whether the food was any good. The wait for the first dish and the gap between courses tell you how it feels to sit there. Cost per meal served tells you whether it can stay open. No single count covers all four, and a kitchen that serves tiny portions can push "orders served" up while the diners leave hungry. The analogy breaks in one place: diners can tell the waiter what was wrong, but most users of a model never say anything.

In one precise sentence, LLM product success metrics are indicators for usage, quality, speed and cost, each with a target, chosen so that improving one cannot silently damage another.

Step 1: A usage funnel separates shown, accepted and kept

Acceptance rate is the fraction of model outputs shown to a user that the user accepts. Ziegler et al. (2022) define it for code completion as accepted completions divided by shown completions, and the mean counts in their Appendix A give 503.94 accepted per 1,872.05 shown, an acceptance rate of 26.9%.¹ In that study acceptance rate correlated best of all usage measurements with perceived productivity, at a correlation of 0.24, which is weak.

The same paper (Ziegler et al., 2022) cautions that it cannot recommend acceptance rate as the singular criterion of quality, because splitting one suggestion into two would raise it without adding user benefit. Its funnel runs from opportunity to shown, accepted, and then unchanged after a set time, which gives the persistence rate.

The funnel carries over to any feature: exposure (the answer was shown), adoption (the user took it) and retained value (it was still in use later). For a support chatbot the stages become conversation started, answer shown, problem resolved without escalation, and no return with the same problem. You now have the usage layer; the next step covers the quality signals around it.

Step 2: Behaviour and sampled review show quality better than thumbs

Thumbs ratings measure quality poorly because few users give them. In one scientific research assistant, thumbs feedback was sparse, under 2% of reports and available only for self-selecting users, while link clicks occurred on at least 15% of reports and tracked repeat usage more closely.

Behaviour separated good output from poor output in that data: high-quality reports had a click-through rate (CTR) of 5.5% against 3.8% for low-quality ones, a relative lift of about 44.7%, and a return rate of 62.3% against 54.7%.¹ Arize likewise notes that explicit signals are often sparse and biased toward unusually good or bad experiences.

A binary rating also cannot say what went wrong. A study of practitioners reports that simple signals cannot reveal which failure occurred, and some teams add a dropdown of reasons for dissatisfaction. A strong answer therefore combines behaviour (clicks, returns, edits) with a regularly sampled set of conversations graded against a written rubric. Next comes the layer users feel first: speed.

Step 3: Speed is two numbers, reported as percentiles

LLM latency splits into the wait for the first token and the pace after it. End-to-end latency equals time to first token plus total generation time, as Anyscale documents, and longer prompts typically lengthen the first wait because the model must read the entire input first.

BentoML gives the per-token formula and a worked case: end-to-end latency of 2,200 milliseconds, first token at 80 milliseconds and 220 output tokens give a time per output token of 9.68 milliseconds. It also warns that a fast first token followed by slow generation still leads to a poor experience.¹

Latency is reported as percentiles, not averages. Google's Site Reliability Engineering book notes that an average hides a long tail, and its monitoring chapter gives a service with an average of 100 milliseconds at 1,000 requests per second where 1% of requests might easily take 5 seconds. Interactive products, such as a voice agent, care most about both latency numbers; batch pipelines care about throughput and cost.

Step 4: Targets and cost turn indicators into decisions

A metric becomes a decision rule only with a target. Google's book states that a natural structure for a service level objective (SLO) is the service level indicator (SLI), the measured number, at or below a target, and that a proxy indicator is acceptable when the ideal one is hard to obtain. Its workbook example measured 3,663,253 requests and proposed three objectives: 97% availability, 90% of requests under 450 milliseconds and 99% of requests under 900 milliseconds.

For an LLM product the same pattern gives targets such as a 95th percentile first-token time, a resolution rate and a spend ceiling. Cost per resolved task is total spend divided by tasks that actually succeeded; it is our own definition, not a vendor standard, and it exposes waste that cost per request hides.¹

Add a guardrail to every target: a metric that must not worsen while you improve another, for example a resolution rate floor while you cut cost.

1 · Usage2 · Quality3 · Speed4 · TargetsReplay

Step-by-step description

1. Shown, accepted and kept.
2. Behaviour beats thumbs.
3. Two latencies, as percentiles.
4. Targets, guardrails and cost.

• Persistence rate measures whether an accepted output is still unchanged later.
• Latency is reported as percentiles, not averages.
• A guardrail is a metric that must not worsen while you improve another.

Worked example

Two illustrative code-assistant products each show 10,000 suggestions. Product A uses ratios of the mean counts in Ziegler et al. (2022), Appendix A: 503.94 accepted per 1,872.05 shown, 328.96 unchanged after 30 seconds and 240.69 after 600 seconds. That gives an acceptance rate of 26.9% and persistence of 65.3% after 30 seconds and 47.8% after 600 seconds.¹ Product A gets 2,692 accepted, 1,757 unchanged after 30 seconds and 1,286 unchanged after 600 seconds.¹ Product B is illustrative: acceptance 0.35 but persistence 0.30 after 600 seconds, so Product B gets 3,500 accepted and only 1,050 unchanged after 600 seconds.¹ B wins on acceptance rate and loses on kept value.

Text description of the diagram

• Product A has 2,692 accepted and 1,286 unchanged after 600 seconds.
• Product B gets 3,500 accepted and only 1,050 unchanged after 600 seconds.

For speed, take 20 illustrative first-token times in milliseconds: 10 of 300, 5 of 350, 3 of 400, one of 2,500 and one of 4,000. The mean is 622.5, the median is 325 (the average of the 10th and 11th sorted values, 300 and 350) and the 95th percentile by nearest rank is 2,500, so the average hides a tail that one user in ten meets.¹

For the objectives of the Google workbook example, 3% of 3,663,253 requests gives 109,897 allowed failures, 10% gives 366,325 and 1% gives 36,632, each rounded down.¹

For a chatbot, take 10,000 illustrative conversations: 6,200 resolved, 2,300 escalated and 1,500 abandoned, with illustrative spend of 3,100 cost units. Spend per resolved task is 0.50 units against 0.31 per conversation, so cost per conversation understates the price of a success.¹

Trade-offs and edge cases

Reported practice differs, and no single accepted standard exists for product-level success metrics, so the four layers combine the funnel of Ziegler et al., the objective method of Google's book and the latency terms of two inference documents. State that boundary rather than present the set as settled.

Held-out scores can fall online. Ziegler et al. (2022) report, citing earlier work, that an IntelliCode metric, recall at five, fell from 90% in pre-deployment evaluation to 70% when used online, so a benchmark score is a starting point, not proof of product success.

Metrics carry confounds, and slicing exposes them. The same study (Ziegler et al., 2022, §5.2) reports an acceptance rate of 23.5% on weekends against 21.2% in weekday working hours. Those figures describe particular time slices, so they are not directly comparable with the 26.9% ratio of means over the whole study, and they show that a headline number moves with who is using the product and when. Slice by language, user group and time before reading a change as an improvement. Acceptance rate can be gamed by splitting one suggestion in two, and thumbs ratings are sparse and self-selected, so neither is read alone.

A correlation is not a cause. The paper itself says acceptance correlates with perceived productivity, and perceived productivity is not measured time saved. Treat each metric as a proxy and validate it against sampled review.

Graders, guardrails and A/B tests

Automated graders and online experiments cover what behaviour alone cannot. A rubric-based grader is often a second model that scores samples at scale, called LLM-as-a-judge, and its scores are trusted only after they are checked against a human-graded sample. Arize lists rubric and LLM-as-a-judge scoring among its evaluation methods and notes that a response can be fluent while still being wrong, unsupported or unsafe.

A safety or groundedness indicator, such as the share of answers not supported by their sources, therefore belongs beside the quality layer as a guardrail. A change is declared a win only through an online A/B test, which splits live traffic between the old and new versions and compares the funnel, the graded sample, the latency percentiles and the guardrails. State the guardrails before the test starts, so a win cannot be declared on one metric alone. Ship behind an A/B test with guardrails.

Text description of the diagram

• Acceptance rate can be gamed by splitting one suggestion in two.
• Thumbs ratings are sparse and self-selected.

The same discipline shapes cost work such as making RAG fast and cheap, and a shared gateway is where many of these numbers are collected, as the LLM gateway interview questions show.

How to answer it in an interview

The spoken structure: name the four layers, give one example number per layer, then state the guardrail and the proxy caveat. A system design answer for a chat product is the natural place to use it.

Common mistakes

• "A higher acceptance rate means a better product." (wrong) It is a proxy that can be gamed and varies by segment and time. (right) "I track acceptance next to persistence and slice it by segment."
• "Average latency is our latency target." (wrong) An average hides the long tail. (right) "I set the target on the 95th and 99th percentiles."
• "Tokens per second is the latency metric." (wrong) Time to first token and time per output token are separate, and a fast start with slow streaming still feels bad. (right) "I report both and the end-to-end total."
• "Thumbs-up rate measures satisfaction." (wrong) It is sparse and self-selected. (right) "I use behaviour and a graded sample, and add a reason dropdown for dislikes."
• "The benchmark score predicts success." (wrong) Online results can fall well below held-out ones. (right) "I confirm on live traffic before I call it a win."

Follow-up questions

What metrics would you use to measure an LLM product?

Use one indicator per layer: the shown, accepted and kept funnel for usage, return rate and graded samples for quality, first-token and per-token percentiles for speed, and spend per resolved task for cost. Give each a target and a guardrail.

How do you measure success when thumbs feedback is sparse?

Lean on behaviour that every user produces, such as clicks, edits and return visits, and on a recurring sample of conversations graded with a rubric. In the Asta data, clicks covered far more reports than thumbs.

What is the difference between time to first token and time per output token?

Time to first token is the wait before anything appears and grows with prompt length and load; time per output token is the pace of streaming afterwards. A product needs both to be good, because a fast start followed by slow generation still feels poor.

What if acceptance rate rises but persistence falls?

Treat it as a warning that users take output they later remove, for example from shorter or more eager suggestions. Check the kept count per shown output, slice by segment, and roll back if kept value fell.

How do you measure the success of a RAG system?

Measure the product layers on this page and add stage metrics for retrieval and generation, as in evaluating a RAG system stage by stage. Also ask first whether you need retrieval at all.

How do you know a prompt change improved the product?

Run it as an A/B test on live traffic against the current prompt, and compare the shown, accepted and kept funnel, a graded sample and the latency percentiles. Call it a win only if the guardrails, such as the unsupported-answer rate, did not worsen.

Sources

1. Productivity Assessment of Neural Code Completion (arXiv 2205.06537) — arXiv / ACM MAPS 2022 (Ziegler et al., GitHub), 13 May 2022
2. Service Level Objectives, Site Reliability Engineering book, ch. 4 — Google, 1 Apr 2016
3. Implementing SLOs, Site Reliability Engineering workbook — Google, 1 Jul 2018
4. Monitoring Distributed Systems, Site Reliability Engineering book, ch. 6 — Google, 1 Apr 2016
5. Understand LLM latency and throughput metrics — Anyscale documentation, 26 Oct 2026
6. Key metrics for LLM inference — BentoML documentation, 26 Oct 2026
7. LLM evaluation: methods, metrics, RAG and agent evals guide — Arize, 24 Oct 2026
8. Understanding Usage and Engagement in AI-Powered Scientific Research Tools: The Asta Interaction Dataset (arXiv 2602.23335) — arXiv, 1 Feb 2026
9. Results-Actionability Gap: How Practitioners Evaluate LLM Products in the Wild (arXiv 2604.16304) — arXiv, 1 Apr 2026