In this article
Foundations: what is different about AI products
What does an AI product manager do differently from a regular product manager?
The core job is the same as in any product management role: choose the problem, the user and the outcome. The difference is that the feature is probabilistic, so it is right some of the time. You therefore define acceptable error rates, build an evaluation set, decide what happens when the model is wrong, and plan monitoring after launch, because quality can change without any code change.
Why it is asked: it separates people who treat a model as a normal feature from people who plan for variable behaviour.
How do you decide whether a problem needs AI at all?
Start from the user outcome and a simple baseline. The first rule in Google's machine learning guide is "Don't be afraid to launch a product without machine learning" (Google, Rules of Machine Learning, updated August 2025). If rules or a lookup meet the target, ship them. Use a model when the rules cannot cover the variety of inputs or the baseline misses the target by a clear margin.
Why it is asked: reaching for AI by default is a common and costly habit.
Text description of the diagram
- Define the metric before building anything.
- A simple rule baseline shows what a model must beat.
- Add a model only if the baseline misses the target.
How do you define success for an AI feature?
Write the metric before building. The same guide's second rule is "First, design and implement metrics" (Google, August 2025). Pair a user outcome, such as tasks completed or time saved, with a quality measure of the model, such as the share of answers judged correct. State the target number, the date and who signs it off.
Why it is asked: a feature without a metric cannot be judged shipped or failed.
What makes a good success criterion?
A good success criterion is specific, measurable, achievable and relevant. The Claude documentation on success criteria lists exactly those four traits and says to base targets on benchmarks, prior experiments or expert knowledge (Claude documentation, as of October 2026). "Better answers" fails; "at least 90 of 100 test cases rated correct by two reviewers" passes.
Why it is asked: vague goals are how AI projects drift.
Metrics and evaluation
What is an evaluation set and how do you build one?
An evaluation set is a fixed collection of realistic inputs, each with a known good outcome or a scoring rule, used to compare versions of a model or prompt. Build it from real user requests, add the hard and rare cases, have two people label a sample, and keep it separate from anything used to tune the system. Read the full explanation: evaluating before and after launch.
Why it is asked: without it, every change is judged by anecdote.
How many test cases do you need?
The answer depends on how precisely you need the score: for an assumed pass rate of 0.80, a margin of 0.05 at 95% confidence needs 246 cases.1 Telling two versions apart by 5 points needs far more (about 900 per version on separate samples, at 80% power), because both scores carry that noise.1 Fewer cases leave wider doubt. Say that you would start with a smaller set to learn, then grow it where failures cluster.
Why it is asked: it tests whether you can reason about noise in a score.
Explain precision and recall for an AI feature.
Take a system that flags support tickets for a human to handle; it flags a ticket when its score passes a threshold, a cut-off you choose. Precision is the share of flagged tickets that truly need a human; recall is the share of all truly relevant items that the system finds. Read the full explanation: precision and recall. They pull against each other: in our illustrative ticket example, a low threshold gave precision 0.60 and recall 0.90, and a high one gave 0.86 and 0.60.1
Why it is asked: it shows whether you can choose the error that costs less.
Text description of the diagram
- At the low threshold precision is 0.60 and recall is 0.90.
- At the high threshold precision is 0.86 and recall is 0.60.
How do you choose between higher precision and higher recall?
Ask which mistake hurts the user more. If a wrong flag wastes a reviewer's minute, favour recall. If a wrong action is hard to undo, such as sending an email or blocking an account, favour precision. In the example, the low setting misses 20 of 200 tickets that need a human; the high setting misses 80.1
Why it is asked: it turns a metric into a product decision.
How do you measure the quality of a generative feature with no single right answer?
Score against a written rubric with a few dimensions, such as correctness, completeness and tone, using trained human reviewers on a sample and, where it has been checked against those humans, a model as a grader. Track agreement between reviewers. Read the full explanation: large language model (LLM) evaluation methods.
Why it is asked: many AI features produce free text.
How would you run an experiment on an AI feature?
Randomise users into control and treatment, pick one primary metric, and add guardrail metrics. A guardrail metric is a measure that must not get worse while the primary metric improves, such as complaint rate, latency or cost per request. Run long enough to cover weekly patterns, and read results together with sampled transcripts, because averages hide a few severe failures. Read the full explanation: A/B testing interview questions.
Why it is asked: AI features can raise engagement while lowering trust.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Build decisions, cost and speed
Should you use prompting, retrieval or fine-tuning?
Move up in cost only when the cheaper step fails on your evaluation set. Prompting changes instructions. Retrieval-augmented generation is a technique that fetches relevant documents and adds them to the prompt, and it helps when the model lacks facts. Fine-tuning retrains the model's weights, the numbers it learned during training, and helps with style or a narrow task. Read the full explanation: fine-tuning versus prompt engineering, and retrieval-augmented generation.
Why it is asked: it tests judgement about cost against gain.
Should we build our own model or buy access to one?
Compare on your evaluation set, not on reputation. Buying is faster and is the default for a first version; building makes sense when data is a moat, when privacy rules block sending data out, or when volume makes hosted pricing larger than the cost of running your own. Name the switching cost and keep your evaluation set portable.
Why it is asked: it separates strategy from enthusiasm.
How do you estimate and control the cost of an AI feature?
Cost per request is tokens in times the input price plus tokens out times the output price, where tokens are chunks of text, roughly three quarters of a word each. With illustrative values of 1,500 input and 400 output tokens, and prices of 3 and 15 units per million, one request costs 0.0105 units and 1,000 requests cost 10.5.1 Control it with shorter prompts, caching, smaller models for easy cases and limits per user.
Why it is asked: margin depends on it.
How do you trade quality against latency?
Set a latency budget from the user experience first, then choose the smallest model and shortest output that meet the quality target on your evaluation set. Stream long answers so the user sees progress, and use a fast model for routing or simple cases. Report quality and latency together, never one alone.
Why it is asked: users leave slow features, but wrong fast answers lose trust.
Risk and responsibility
How do you handle hallucinations?
Reduce them, detect them and limit their harm. Ground answers in retrieved sources, ask the model to cite, refuse when it lacks support, check outputs against rules or a second pass, and show users the source. Measure the hallucination rate on your evaluation set and set a launch threshold. For high-stakes cases, keep a human in the loop.
Why it is asked: it is the most visible failure of generative products.
How do you think about bias and fairness in an AI feature?
Check performance by user group, not only overall, because an average can hide a group the feature serves badly. Compare error rates across groups on your evaluation set, look at the training and retrieval data for gaps, and decide the acceptable gap before launch. Document the findings and the remaining limits.
Why it is asked: unequal quality is a product, legal and trust risk.
How would you apply a risk framework such as NIST's?
The AI Risk Management Framework 1.0 from the U.S. National Institute of Standards and Technology (NIST), released in January 2023, organises the work into four functions: govern, map, measure and manage (NIST AI 100-1, section 5, NIST, January 2023). Say you would assign owners, list the uses and affected people, measure risks with your evaluation set, and act on the results. NIST also publishes a voluntary Playbook.
Why it is asked: companies want a product manager who can structure risk work.
When do you keep a human in the loop?
When errors are costly, hard to reverse or legally sensitive, or when the model's confidence is low. Design the review step so it is quick: show the model's draft, its sources and a one-click correction. Measure how often reviewers edit or reject, since that rate is a live quality signal. Read the full explanation: human-in-the-loop agents.
Why it is asked: it tests design of the whole workflow, not only the model.
Launch, monitoring and communication
How do you launch an AI feature safely?
Roll out in stages, for example 1%, 5%, 25% and then everyone, with a pass condition at each stage on quality, guardrail metrics and support tickets, and a switch that turns the feature off quickly. Brief support teams, publish known limits to users, and review sampled outputs daily during the first weeks.
Why it is asked: AI failures surface in the long tail, after launch.
How do you monitor an AI feature after launch?
Track the evaluation score on fresh samples, user feedback, edit and reject rates, cost and latency. Watch for model drift, which is a drop in quality because inputs, users or the underlying model changed over time. Alert on thresholds and rerun the evaluation set on every model or prompt change. Read the full explanation: evaluating before and after launch and distribution shift.
Why it is asked: a launched model decays without anyone touching it.
How do you explain an AI feature's limits to executives or customers?
Use plain numbers and examples: "It is right about nine times in ten on our test set, and here is what it does on the tenth." Show one good and one bad output, state what the feature will never do, and say how you will improve it. Avoid words such as "intelligent" that promise more than you can measure.
Why it is asked: expectations set the first month of feedback.
How these interviews are run
AI product manager rounds often mix a product-sense case, a metrics case, a technical conversation about models, and behavioural questions. Prepare a story where a launch missed its target, and use the situation, task, action, result (STAR) method to tell it. Read the full explanation: product sense interviews. Practise sizing and estimates with market sizing, and prepare questions to ask the interviewer, such as how the team measures model quality. For the rest of the product management loop, see product management interviews.
- Define the metric and a rule baseline before choosing a model.
- Build an evaluation set from real cases, including the hard ones.
- Choose precision or recall by which mistake costs the user more.
- Estimate cost per request from tokens and prices, and set a latency budget.
- Plan guardrails, human review, a staged launch and monitoring after launch.
Sources
- Rules of Machine Learning: Best Practices for ML Engineering — Google for Developers (page updated August 2025), 25 Aug 2025
- AI Risk Management Framework (AI RMF 1.0) — U.S. National Institute of Standards and Technology (as of October 2026), 26 Jan 2023
- Define success criteria and build evaluations — Claude developer documentation (undated page, as of October 2026), 24 Oct 2026
- Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1 — U.S. National Institute of Standards and Technology (January 2023), 26 Jan 2023
- AI RMF Playbook — U.S. National Institute of Standards and Technology (as of October 2026), 24 Oct 2026
