How do you answer a metric dropped case interview?

In this article

1. Before you start
2. Why interviewers ask this
3. A metric dropped interview question: how to investigate it, from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

For the wider set of analyst prompts, see data analyst interview questions. You do not need advanced statistics, only comfort with rates and shares.

Why interviewers ask this

A metric drop is a common alarm for an analyst, and the way you respond shows how you think when the cause is unknown. Candidates report this prompt in analyst and product-analytics interviews, usually worded as "a key metric dropped ten percent, how do you investigate?"

A weak answer jumps to one cause, such as "the latest release broke something", and proposes a fix. A strong answer asks how the metric is defined, rules out reporting problems, narrows the drop with a decomposition and segments, and only then names causes it can test. A Staff-level answer also says what it would not conclude from the data it has, and what it would monitor afterwards. The order below is a practical structure, not a company's official rubric.

A metric dropped interview question: how to investigate it, from first principles

Mix shift

A mix shift is a change in the share of traffic, users or orders that each segment contributes to a blended metric, which moves the blended number even when no segment's own rate changes. Spotting it is the main reason to segment before you search for a cause.

Simpson's paradox

Simpson's paradox is a phenomenon in which, in the words of the Wikipedia entry, "a trend appears in several groups of data but disappears or reverses when the groups are combined". It is a close relative of a mix shift: the blended trend can differ from every group's trend.

Metric decomposition

A metric decomposition is a rewriting of one number as a product or sum of simpler parts, such as orders equal visits times conversion rate, so that you can see which part moved. It turns "why did it drop" into a narrower question about one factor.

In plain words

Think of a restaurant's average bill per table. If lunch guests replace dinner guests, the average falls. No menu price changed. The analogy breaks in one place: a real price cut also lowers the average, and the average alone cannot say which happened.

In one precise sentence, a blended metric is a weighted average of segment rates, so it moves when a rate moves or when the weights move.

A segment is a slice of users or events that share a trait, such as platform, country, signup month or acquisition channel.

The investigation runs in six steps, from confirming the drop to testing hypotheses. Each step narrows the cause before the next begins.

Step 1: Confirm the drop is real

Confirming the drop is real means checking that the number fell in the world and not only in the report. Start by asking how the metric is defined, over which period and compared with what. Then look for reporting causes: a tracking tag removed in a release, a changed event name, a broken data pipeline, a dashboard filter, or a redefined numerator or denominator. Many analytics tools finish processing recent data with a delay, so the latest days can look low; check the tool's own documentation for its freshness rules.

The practical test is a second source, checked for each affected segment. Compare the dashboard with a raw table, such as orders in the payments system against orders in the analytics events. If the two disagree, you have a data issue, not a business problem.

Input: the reported drop. Output: a confirmed drop, or a data fault to fix first. You now know which number to explain.

Step 2: Check timing and seasonality

Seasonality is a pattern in a metric that repeats on a calendar cycle, such as weekdays against weekends, month-end peaks or holidays. Checking it means comparing like with like: this Tuesday with recent Tuesdays, this week with the same week last year, and the drop with the usual week-to-week range.

Also read the shape of the drop. A sudden step on one date points to a discrete event, such as a release, an outage or a pricing change. A slow slide over weeks points to gradual causes, such as a changing user mix or rising competition. A dip that matches last year's calendar is likely normal variation.

Input: the daily or weekly series. Output: a start date for the drop, or the finding that it is within the normal range. That date guides Step 5.

Step 3: Decompose the metric into its components

Decomposing a metric means writing it as parts and finding which part moved. Orders equal visits times conversion rate; revenue equals orders times average order value; a signup funnel is a chain of step-to-step rates. Compute each part for the period before and after the drop.

Two outcomes are common. If visits fell and conversion held, the problem is upstream, in traffic. If conversion fell, find the funnel stage where it fell: product page to cart, cart to payment, payment to confirmation. Each answer removes most of the possible causes.

Input: the metric and its formula. Output: the one or two components that moved. In Structured Query Language (SQL) this is one query that computes each component per day. You now know where to look, but not yet for whom.

Step 4: Segment by platform, geography, cohort and channel

Segmenting a metric means computing its rate and its share of volume for each slice, before and after the drop. Useful cuts are platform (web, mobile app), geography, user cohort (new against returning, signup month) and acquisition channel. In SQL this is a group by query over both periods; in a spreadsheet it is a pivot table, a topic to revise with the Excel interview questions.

Read the result as one of three outcomes. If every segment's rate fell together, suspect a shared cause: a data fault, a release or an outage. If one segment's rate fell, suspect a local cause in that segment. If no segment's rate fell but shares moved, you have a mix shift, and the drop is a change in who arrived, not in how anyone behaved.

Input: the component that moved. Output: the segment, or the mix, that explains it. Choose cuts from what Step 3 showed.

Step 5: Look at internal and external changes

Looking for changes means lining up the start of the drop, in the affected segment, with everything that changed around that date. Internal changes include releases, experiment ramps, pricing or checkout edits, tracking changes and campaign budget moves. External changes include holidays, competitor launches, outages at a payment provider or a platform, and policy changes.

Use a change log, release notes and the marketing calendar. If an experiment is running, also check that its users were split between variants as designed, as covered in the A/B test analysis questions for analysts; a skewed split can make a result look like a drop. Timing is evidence, not proof: two changes can share a date.

Input: the segment and its start date. Output: a short list of candidate causes with dates. Rank them by how well the timing and the affected segment match.

Step 6: Form hypotheses and test them

A hypothesis is a specific cause that makes a prediction you can check. "The checkout release broke payment on one browser" predicts that completion fell for that browser from the release hour and not for others. Write the prediction before you look.

Test the likeliest and cheapest causes first. Look for the predicted pattern in the data, compare affected and unaffected users, and, where the change is reversible, roll it back or run a controlled experiment. Then close the loop: state your conclusion and your confidence, recommend an action, and say what you would monitor so the next drop is caught sooner.

Input: ranked causes. Output: a tested cause, a recommendation and an alert. The whole answer is now complete.

1 · Confirm2 · Timing3 · Decompose4 · Segment5 · Changes6 · TestReplay

Step-by-step description

1. Check the drop is real.
2. Rule out the calendar.
3. Find the part that moved.
4. Find the slice or the mix.
5. Line up releases and events.
6. Test a specific hypothesis.

• The investigation runs in six steps, from confirming the drop to testing hypotheses.
• Each step narrows the cause before the next begins.

Worked example

Take an online shop with two traffic channels. Every figure here is illustrative.¹ In week A, organic search sends 60,000 visits that convert at 4.0%, and paid social sends 40,000 visits that convert at 1.0%. That is 2,400 plus 400 orders from 100,000 visits, so blended conversion is 2.8%.¹

In week B a paid campaign grows. Organic search sends 45,000 visits and paid social sends 55,000, and both rates are unchanged. Blended conversion is 2.35%, a fall of 0.45 percentage points, or about 16% in relative terms, and no channel got worse.¹

Decomposition separates the two effects. The mix effect multiplies each channel's change in share of visits by its week A rate and adds them: minus 0.45 points. The rate effect multiplies each channel's week B share by its change in rate: zero.¹ Blended conversion is 2.8% in week A and 2.35% in week B when only the traffic mix changes.

Now suppose the organic search rate also slips to 3.6% in week B. Blended conversion falls to 2.17% when the organic search rate also drops, a fall of 0.63 points: 0.45 from the mix and 0.18 from the rate, when the mix is measured at week A rates and the rate at week B shares.¹ The rate effect, about 0.2 points, is the part to investigate inside a channel; the mix effect is a question of where traffic comes from.

Text description of the diagram

• Blended conversion is 2.8% in week A and 2.35% in week B when only the traffic mix changes.
• Blended conversion falls to 2.17% when the organic search rate also drops.

Trade-offs and edge cases

The cause usually leaves a recognisable pattern across segments. A pipeline or definition fault usually hits every segment at once; a tracking fault in one app, version or browser looks like a single-segment drop, which is why Step 1 checks the second source per segment. A mix shift shows flat segment rates while the shares move. A real change shows one segment's rate falling. Normal variation stays within the usual weekly range.

Text description of the diagram

• A pipeline or definition fault usually hits every segment at once.
• A mix shift shows flat segment rates while the shares move.

The blended trend can mislead even for experts. The 1973 University of California, Berkeley admissions data, originally analysed by Bickel et al. (1975), are available for the six largest departments as the R undefined dataset. Pooled over those six, 2,691 men applied and 44.5% were admitted, while 1,835 women applied and 30.4% were admitted.² In Department A, 62.1% of men and 82.4% of women were admitted.²

Women applied more to low-admit departments: 92.8% of women applied to Departments C to F, against 48.5% of men.² A confounding variable is a third factor, here the choice of department, that influences both the grouping and the outcome. The fix is to segment by it. Department by department, the Wikipedia entry reports that of 85 departments, 6 were significantly biased against men and 4 against women.

Several causes can act together. The worked example had a mix effect and a rate effect in one drop. Decomposing the change keeps you from crediting a whole fall to one story.

The split depends on the weights. With the opposite base week (mix at week B rates, rate at week A shares) the same 0.63-point fall splits 0.39 mix and 0.24 rate; the midpoint method gives 0.42 and 0.21.¹ State the convention you use; the conclusion that most of the fall is mix holds under all three.

Small segments are noisy. A segment with few users can swing widely by chance, and cutting the data many ways will produce some striking differences by luck. Choose cuts from the decomposition and check that a difference is larger than the segment's usual swings.

The order is flexible. If a release shipped yesterday and the step on that date is sharp, you can reach Step 5 early, but still run Step 1 first, because a release is also a common way to break tracking.

How to answer it in an interview

The spoken structure: verify the data, rule out the calendar, decompose, segment, match to changes, then test.

Common mistakes

• "The metric dropped, so something broke." (wrong) The cause may be a data fault, a mix shift or normal variation. (right) "First I check whether the drop is real and whether any segment's rate actually fell."
• "I would start by fixing the latest release." (wrong) That is a guess before any narrowing. (right) "I would line up the drop's start date with the change log after I know which segment moved."
• "I would split by every dimension and report the biggest gap." (wrong) Many cuts produce chance differences. (right) "I choose cuts from the component that moved and check the gap against normal swings."
• "I would look at the overall number." (wrong) The overall number can hide opposite moves in the segments. (right) "I compute each segment's rate and its share, before and after."
• Starting without questions. (wrong) You cannot explain a number you have not defined. (right) "How is the metric defined, over what period and against what baseline?"

Follow-up questions

How do you investigate a metric dropped interview question when you have no data in the room?

Say the order out loud and name what each step would show: reporting check, calendar comparison, decomposition, segments, change log, tests. In this format the structure and the reasoning about what each result would mean are what is tested.

What if the metric dropped on every platform together?

A simultaneous drop across all segments points to a shared cause: a tracking or pipeline fault, a release that touched everything, or an outage. Check the data first, then the change log around the start date.

How do you tell a real drop from normal variation?

Compare the new value with the same weekday and week in earlier periods and with the usual range of weekly swings. A value inside that range is normal variation; one clearly outside it needs a cause.

What if the drop is only in one country?

Segment that country further, by platform, channel and cohort, and look for local causes: a holiday, a payment method issue, a regional outage, a pricing or language change, or a local campaign ending.

How would you find the segment behind the drop with SQL?

Group by the segment for both periods, compute volume share and rate for each, join the two results and rank segments by the change in rate and the change in share. The SQL interview questions cover the joins and window functions this needs.

How do you make the next investigation faster?

Build a dashboard that shows the metric next to its components and key segments, with release markers on the time axis; for dashboard questions, see the Tableau developer interview questions. The data analyst question guide lists neighbouring case prompts.

Sources

1. Simpson's paradox — Wikipedia, 21 Sep 2025
2. UCBAdmissions: Student Admissions at UC Berkeley (R datasets documentation) — R Project, datasets package, 26 Oct 2026
3. Sex Bias in Graduate Admissions: Data from Berkeley (Bickel, Hammel, O'Connell) — Science (origin of the analysis), 7 Feb 1975
4. Confounding — Wikipedia, 1 Sep 2025