Reading an A/B test like a product owner

In this article

1. Before you start
2. Why interviewers ask this
3. Reading an A/B test from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step of the product and analytics case interview learning path.

Why interviewers ask this

Candidates report product-sense and analytics rounds that hand over test results and ask "should we launch?". The skill being tested is judgment under mixed evidence. A weak answer reads one green number and says ship. A strong answer asks whether the test was valid, states the size of the change with its uncertainty, looks for harm on protective metrics, and names what is being traded.

Questions of this kind reported by candidates include defining success for a launch, explaining why some metrics went red while others went green, and describing an A/B testing problem you hit yourself. The same habits help in the rounds on data science interview questions and in product rounds such as the Meta product architecture interview.

Reading an A/B test from first principles

A/B test

An A/B test is a controlled experiment that randomly assigns users to version A (control) or version B (the change) and compares a chosen metric between the two groups, so that a difference can be attributed to the change rather than to who happened to see it.

Overall evaluation criterion

An overall evaluation criterion is the single metric, or small weighted set of metrics, that a team agrees in advance defines success for an experiment, so a launch decision does not depend on whichever number happened to move.

In plain words

Picture two shop windows on the same street. You change one display and count the sales from each. If window B sells more, was it the display or luck? A test answers that by asking how often luck alone would produce a gap this big. The analogy breaks in one place. Shoppers choose a window, but in an A/B test users are assigned at random, which is what makes the comparison fair.

In one precise sentence: an A/B test estimates the effect of a change by comparing randomly assigned groups, and reading it well means judging validity, size, uncertainty, harm and trade-off together.

Step 1: Check the test was run correctly

Before you read any result, check that the test was valid. A valid test fixed its sample size in advance, ran to that size and split users as planned. Evan Miller (2010) explains that significance calculations assume the sample size was fixed in advance. If you instead run a test until it looks significant, "all the reported significance levels become meaningless". Stopping at the first good-looking moment is called peeking.

Also compare the planned split with the observed one. A sample ratio mismatch means some users were lost or mis-assigned unevenly, and no downstream number can be trusted until that is explained. For a book-length treatment, see Kohavi, Tang and Xu (2020). You now know whether the result is worth reading at all.

Step 2: Read the effect size and the interval, not just the p-value

A dashboard that shows a "95% chance of beating original" is answering a narrow question: assuming there is no real difference between A and B, how often would a difference at least this large appear just by chance? That probability is the p-value. A result is called significant when the p-value falls below a threshold chosen in advance, the significance level, commonly 5% or 1% (Miller, 2010). The p-value is not the probability that B is better. The American Statistical Association's 2016 statement makes the same point: a p-value does not measure the probability that a hypothesis is true.

Read two more things. The effect size is how big the change is, in the metric's own units. A confidence interval is a range of effect sizes that the data cannot rule out at the 95% level, so a wide interval means the test is imprecise. A tiny effect can be significant and still not matter to the business. You now know how big the change is and how sure the test is.

Step 3: Check the guardrails before you celebrate

Most products care about more than one number. An overall evaluation criterion (OEC) is the one metric, or small weighted set of metrics, that a team agrees in advance defines success for an experiment. Guardrail metrics, also called counter-metrics, sit beside it. A green main metric with a red guardrail needs a trade-off decision, not an automatic launch. Defining success means naming the primary metric, the guardrails that must hold and the smallest change worth shipping.

Scanning many metrics also creates false greens. At the 5% level, the chance of at least one false positive across k independent metrics is 1 minus 0.95 to the power k. With 5 independent metrics it is 0.2262, and with 20 independent metrics it is 0.6415.¹ This is why one metric is named as primary before the test starts, and why a lone green among many red or flat metrics deserves suspicion.

Text description of the diagram

• With 5 independent metrics at the 5% level the chance of at least one false positive is 0.2262.
• With 20 metrics it is 0.6415.

Step 4: Decide and name the trade

The decision is a sentence a colleague can disagree with: ship, hold or rerun, plus the trade being accepted. If the main metric is green and a guardrail is red, you weigh the size of each effect against what it costs the business. If the interval spans zero, the test could not tell, and a rerun with a larger sample is the honest answer.

Say the trade aloud: "this gains X but costs Y; I accept that because Z."

1 · Check the setup2 · Read the main metric3 · Check guardrails4 · DecideReplay

Step-by-step description

1. Fixed sample size, clean user split.
2. Effect size with an interval.
3. Did anything protective get worse?
4. Name the trade: ship, hold or rerun.

• A test is checked for a fixed sample size and a clean split before any result is read.
• The main metric is read as an effect size with an interval, not as a single p-value.
• Guardrail metrics are checked next, and a guardrail that moves the wrong way can outweigh a green main metric.
• The decision names the trade, then says ship, hold or rerun.

In the figure, a test is checked for a fixed sample size and a clean split before any result is read. The main metric is read as an effect size with an interval, not as a single p-value. Guardrail metrics are checked next, and a guardrail that moves the wrong way can outweigh a green main metric. The decision names the trade, then says ship, hold or rerun.

Text description of the diagram

• A green main metric with a red guardrail needs a trade-off decision.
• A flat main metric with an interval that spans zero means the test could not tell.

In the table, a green main metric with a red guardrail needs a trade-off decision. A flat main metric with an interval that spans zero means the test could not tell.

Worked example

All numbers here are illustrative: a signup page test where version B changes the call to action. Users are split evenly between arms. The test needed 57,761 per arm, computed below, and was rounded up to 60,000 per arm, fixed in advance.¹ Version A converts at 10.00% (6,000 conversions) and version B at 10.50% (6,300 conversions). That is a gain of 0.50 percentage points, or 5.0% relative.¹

A two-proportion z-test gives z of 2.86 and a p-value of about 0.004. The 95% interval for the gain runs from 0.16 to 0.84 percentage points, so the gain is real but its size is uncertain.¹ Now the guardrail, a page-complaint rate measured per visitor so that it does not depend on how many people convert: it is 2.00% for A (1,200 complaints) and 2.30% for B (1,380 complaints). That rise of 0.30 points has a p-value below 0.001 and an interval of 0.14 to 0.46 points.¹ Both the gain and the cost are real. For every thousand visitors, B adds about 5 conversions and about 3 complaints, so the call depends on what each is worth.

The small test shows why sample size is planned before launch: the same rates cannot be told apart from chance with 10,000 per arm, and the interval even includes a loss.¹ The required 57,761 per arm comes from the standard two-proportion sample-size formula at the 5% level with 80% power.¹ Do not stop the 10,000-per-arm test the day it turns green; per Step 1, that breaks the fixed-sample assumption.

The chart puts the three 95% intervals against zero. At 10,000 per arm the gain runs from -0.34 to 1.34 points and spans zero. At 60,000 per arm it runs from 0.16 to 0.84 points, above zero. The complaint rise runs from 0.14 to 0.46 points, also above zero, so the question is what each is worth.¹

Text description of the diagram

• At 10,000 per arm the gain runs from -0.34 to 1.34 points and spans zero.
• At 60,000 per arm the gain runs from 0.16 to 0.84 points, above zero.
• The complaint rise runs from 0.14 to 0.46 points, also above zero.

Trade-offs and edge cases

A test result is trustworthy only when the test was valid and its effect is judged against what matters, so these are the cases where the textbook reading goes wrong.

• Statistically significant is not the same as important. With enough users, a tiny effect becomes significant. Judge the interval against the smallest change worth shipping.
• Segments tell stories, and also lie. Splitting results by country or device multiplies comparisons, which raises the false-positive risk shown in Step 3. Treat a segment finding as a lead for the next test.
• Effects can fade or build. Users may react to novelty at first, or need time to adapt. A short test can mislead about the long run, so say how long the test ran.
• Users can affect each other. In a network or marketplace product, the control group may feel the change through the treated group, which blurs the comparison.
• The number may move for outside reasons. A sudden metric shift unrelated to the test needs investigation first, as in investigating a sudden watch-time drop.
• A split mismatch needs a test, not an eyeball. Compare the observed counts with the planned split using a chi-square goodness-of-fit test. With a planned 50/50 split and 50,000 against 49,000 observed assignments (illustrative), the statistic is 10.10 and the p-value is about 0.0015, so chance rarely produces that gap.¹ Set the mismatch alarm at a much stricter threshold than 5%, so that routine noise does not stop tests.
• Peeking has current fixes. Sequential designs, such as group-sequential boundaries or always-valid p-values, allow early looks at a planned cost in power; Miller's note points to sequential or Bayesian designs as the way past the problem.
• Pre-period data can sharpen a test. Deng et al. (2013) show that using each user's data from before the experiment reduces variance, so the same effect can be detected with fewer users.
• A test is not always possible. Where users cannot be randomised, compare against historical data with care, and state the added doubt.

How to answer it in an interview

The spoken structure: validity first, then effect size with interval, then guardrails, then a named trade and a decision.

Common mistakes

• "The dashboard says 95%, so B is 95% likely to be better." (wrong) The figure says how often chance alone would produce a gap this large, not how likely B is to be better. "If A and B were identical, a gap this large would be rare, so I treat it as unlikely to be chance and check its size." (right)
• "I stopped the test as soon as it went green." (wrong) That breaks the fixed-sample assumption behind the p-value. "I fixed the sample size in advance and read the result at the end." (right)
• "The main metric is green, so we ship." (wrong) Guardrails may have moved the wrong way. "I read the main metric and the guardrails together before deciding." (right)
• "It's significant, so it matters." (wrong) Significance says nothing about business size. "I compare the interval with the smallest change worth shipping." (right)

Follow-up questions

Metrics moved in different directions: how do you interpret the results and decide next steps?

Check validity first, then rank metrics by what was declared primary and what is a guardrail. Look at the size and interval of each move, and name the trade. If the picture is still unclear, rerun with a larger sample or a narrower change.

What A/B testing issues did you run into?

Good answers name a real one: peeking before the planned sample size, a user split that did not match the plan, too many metrics scanned, a test too small to detect the change, or a result that faded after launch. Say what you changed in your process afterwards.

Is feature X better than feature Y?

Start by defining "better": one primary metric, the guardrails that must hold, and the smallest change worth caring about. Then outline a randomised test with a fixed sample size and say how you would read it using the four steps above.

What if you cannot run an A/B test?

Say so plainly and lower your confidence. You can compare before and after, or use difference-in-differences, which compares the change in a group that got the feature with the change in a similar group that did not, so shared trends cancel. A holdout group, a small random share of users kept on the old version, is the cleaner option when you can plan it, because outside factors can still explain a before-and-after difference. Recommend a small, reversible rollout where you can.

How would you balance ads and follower posts and monitor effectiveness?

Treat it as a trade-off between ad revenue and user experience. Name an overall evaluation criterion, set guardrails such as time spent and return visits, and watch both sides over a long enough window. The ads and marketplace domain questions page covers the ads side from zero.

Sources

1. How Not To Run an A/B Test — Evan Miller, 18 Apr 2010
2. The ASA Statement on p-Values: Context, Process, and Purpose — The American Statistician (Wasserstein and Lazar), 7 Mar 2016
3. Trustworthy Online Controlled Experiments — Cambridge University Press (Kohavi, Tang, Xu), 2 Apr 2020
4. Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data — WSDM 2013 (Deng, Xu, Kohavi, Walker), 4 Feb 2013