In this article
- Start here: the learning path
- How does a data science case study round work?
- How do you choose the metrics for a case?
- How do you read an A/B test result?
- How do you investigate a metric drop?
- How do design and domain cases differ from analytics cases?
- How do you measure an AI product?
- Questions interviewers ask on data science cases
- All 9 guides in Product and analytics cases
- Follow-up questions
- Sources
- Data science case study
- A data science case study is an interview exercise in which you are given a vague business or product problem and are judged on how you clarify it, pick measurable goals, analyse evidence and recommend a decision, not on a single correct answer.
- North star metric
- A north star metric is the one number that best captures the value a product delivers to its users, chosen so that raising it should raise long-term success, and supported by a few input metrics that explain its movement.
- Minimum detectable effect
- A minimum detectable effect is the smallest change in a metric that an experiment is sized to detect reliably, set before the test starts and used to calculate how many users it needs.
- Peeking
- Peeking is the practice of checking an experiment's significance repeatedly while it runs and stopping when it first looks significant, which raises the rate of false positives above the planned level.
Start here: the learning path
Read the case method first, because every round follows it. Then take the topics in this order: metrics, then experiments, then investigating a metric drop, then domain and design cases, then measuring AI products. If you are new to experiments, the analytics hub explains how an A/B test is analysed, and product sense and guardrail metrics explains how to pick a user and a problem. Practice questions are collected in data science interview questions.
Text description of the diagram
- A case round follows one sequence: clarify, define success, hypothesise, analyse, recommend.
- Choose one primary metric, a few input metrics that explain it, and guardrail metrics that must not get worse.
Read these articles in this order:
- Meta product architecture interview: format and how to prepare
- Domain case rounds outside big tech: questions and answers
- Measuring AI products: which metrics to track and why
- Reading an A/B test like a product owner
- Responsible-AI scenario interview questions answered: fairness, privacy, misuse
- Ads and marketplace domain questions, explained from zero
- YouTube watch time dropped 5% last Tuesday: how do you investigate?
- How would you improve data quality on a product like Siri?
- How to answer AI-safety questions in engineering interviews
- How do you build a roadmap and prioritise by goal, capacity and impact?
- How do you use AI in your daily work? How to answer it
How does a data science case study round work?
A case round follows one sequence whatever the topic: clarify, define success, form hypotheses, analyse, recommend. First clarify the goal and the user by asking questions, because the prompt is deliberately vague. Then define success as a metric, with a baseline and a time window. Next list hypotheses ranked by how likely and how costly they are to check, say what data you would pull for each, and work through the likeliest. Finish with a recommendation, its risks and the next step you would take, in two or three sentences.
Interviewers score the sequence more than the final answer. A candidate who jumps to a model or a chart before asking what the product is for usually loses points early. The product sense interview article covers the first two steps in depth, and correlation versus causation covers the trap in the fourth.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →How do you choose the metrics for a case?
Choose one primary metric tied to the user's goal, a few input metrics that explain it, and guardrail metrics that must not get worse. Rodden, Hutchinson and Fu describe a Google framework for user-centered metrics with five categories, Happiness, Engagement, Adoption, Retention and Task success (HEART), and a process for mapping product goals to metrics (Rodden et al., 2010). Start from the goal, then ask what behaviour would show the goal is met, then pick the number that measures that behaviour.
State the weakness of your metric before the interviewer does. Engagement can rise because users are confused, revenue can rise because prices rose, and an average can hide a segment that got worse. Each metric you name needs a definition with a numerator, a denominator and a time window, and one sentence on how it could mislead.
How do you read an A/B test result?
Read it in three steps: check the experiment was set up fairly, check the difference is larger than noise, then check it matters. In the simplest form of a controlled experiment, users are randomly assigned to a control or a treatment, metrics are collected and statistical tests check whether the variants differ (Kohavi, Henne and Sommerfield, 2007). Randomisation is what lets a difference be read as an effect.
The common failure is peeking. Miller (2010) argues that stopping as soon as a result looks significant makes the reported significance levels meaningless. We simulated it: 20,000 A/A tests, where both groups have the same 10% conversion rate, so every significant result is a false positive.1
| Number of looks (equally spaced) | False positive rate in the simulation |
|---|---|
| 1 (planned) | 5.0% |
| 2 | 8.4% |
| 5 | 14.0% |
| 10 | 19.1% |
Text description of the diagram
- One planned look gives a 5.0% false positive rate.
- Ten looks give a 19.1% false positive rate.
The simplest remedy is to set the sample size before the test. With a 10% baseline conversion rate and a minimum detectable effect of one percentage point, a normal-approximation calculation at 5% significance and 80% power needs about 14,128 per group (variance taken at the 10% baseline for both groups), close to the 14,400 that Miller's rule of thumb of 16 times the variance over the squared effect gives; at two percentage points it is about 3,532 per group.1
Halving the detectable effect roughly quadruples the users needed. Fixing the sample size is not the only valid design: sequential methods such as the always-valid p-values of Johari et al. (2015) are built for continuous monitoring, so they allow valid early stopping, but you choose the method before the test starts rather than improvising it after a peek.
How do you investigate a metric drop?
Work from the cheapest explanation to the most expensive: data, then timing, then segments, then causes. First check the data: has logging changed, is a pipeline late, did a definition change? Then check timing: is the drop outside the usual weekly and seasonal swing, and when exactly did it start? Then split by segment, such as platform, country, app version and new versus returning users, to see where the drop is concentrated. Only then ask what changed: a release, an experiment, a campaign or an outside event.
State each hypothesis with the query that would confirm or reject it, and rule out the cheap ones first. A drop that appears in every segment at the same minute points to data or a global release; a drop in one version points to that release. Also check data quality at the start: duplicated events, missing logs, a changed definition or a late pipeline can each fake a drop. Finish by naming the one change or fix you would test first.
How do design and domain cases differ from analytics cases?
Design cases ask you to build a product; domain cases ask how a specific business works as data. The Meta product architecture interview article covers the design round in which you build a user-facing product end to end, from requirements to architecture. The ads and marketplace domain questions article covers how campaigns, auctions, frequency caps and seller ratings are modelled as data.
Both still use the case method: clarify, define success, then work through the design or the data model. Domain cases outside big tech, such as health, fintech or pricing, work the same way: learn how money or care flows through the business, then name the metric that tracks it. The difference is the output. An analytics case ends in a recommendation backed by numbers; a design case ends in a structure that could be built and measured.
How do you measure an AI product?
Measure an AI product on quality, cost and risk together, because a better answer that costs ten times more or fails in a harmful way is not a win. The National Institute of Standards and Technology (NIST) released its AI Risk Management Framework 1.0 on 26 January 2023 as a voluntary framework to improve the ability to incorporate trustworthiness into the design, development, use and evaluation of AI systems (NIST, read Oct 2026). Its page shows four elements: govern, map, measure and manage.
In a case round, name a quality metric on a held-out set, a cost per request, a latency target and a risk or fairness check for the segments most affected. Say how you would check each of them before launch and after it.
Questions interviewers ask on data science cases
Case rounds mix three question types. Open product questions ("how would you measure success?") are answered by naming one primary metric tied to the user's goal, with guardrails. Analysis questions ("a metric dropped, what do you do?") are answered by checking data, timing, segments and causes in that order. Statistics questions ("what is a p-value?") are answered by saying it is the probability of a result at least as extreme as the one observed if there were no real effect. The data science interview questions list collects short answers to the statistics, machine learning and Structured Query Language (SQL) questions that surround the case round, and the classical machine learning hub covers the modelling side.
Follow-up questions
How would you tell whether a new feature is improving engagement?
Define engagement as a metric first, run a randomised experiment where you can, and compare the treatment with the control on the metric and on guardrails. If you cannot randomise, say what your estimate assumes. See A/B test analysis in the analytics hub.
What could go wrong with that metric?
It can move for the wrong reason, hide a segment that got worse, or reward a behaviour you do not want. Name one such failure for your metric and the guardrail that would catch it, as in the product sense interview article.
How would you define a north star metric for a live-events feature?
Pick the number that shows users getting value, such as attendance of events people planned, not page views of the event list. Add input metrics for discovery and for follow-through and a guardrail for complaints. Defend the choice with the user's goal. The product sense interview article shows how to tie a metric to a guardrail.
How long should an A/B test run?
With a fixed-sample design, until the sample size is reached, set in advance from the baseline and the minimum detectable effect, and for whole weeks if behaviour varies by weekday. Stopping early when the result looks significant raises false positives, as the table above shows, unless you chose a sequential design made for early stopping. The A/B test analysis section of the analytics hub covers reading the result.
How do I prepare for a case round?
Practise the sequence aloud on three open prompts: clarify, define success, hypothesise, analyse, recommend. Time yourself, and learn to state a metric's weakness. The interview loops article describes the stages and rounds of a full interview loop, and the behavioral interview hub covers the conversational rounds.
- A case round is a sequence: clarify, define success, hypothesise, analyse, recommend.
- Choose one primary metric, a few input metrics and guardrails, and state how each could mislead.
- Fix the experiment's sample size in advance; peeking turns a planned 5% false positive rate into 14% at five looks.
- Investigate a metric drop from data to timing to segments to causes.
Sources
- Practical Guide to Controlled Experiments on the Web: Listen to Your Customers not to the HiPPO — KDD 2007 (Kohavi, Henne, Sommerfield), 12 Aug 2007
- Measuring the User Experience on a Large Scale: User-Centered Metrics for Web Applications — Google Research, CHI 2010 (Rodden, Hutchinson, Fu; year from the venue, day approximate), 10 Apr 2010
- Always Valid Inference: Continuous Monitoring of A/B Tests — arXiv 1512.04922 (Johari et al.; month from the arXiv id, day approximate), 15 Dec 2015
- How Not To Run an A/B Test — Evan Miller, 18 Apr 2010
- AI Risk Management Framework — NIST (AI RMF 1.0 released 26 January 2023), 26 Jan 2023
