In this article
The basics
What is an A/B test, and how does it differ from split testing?
An A/B test is a controlled experiment: users are randomly assigned to a control (A) or a variant (B), and a metric is compared between the groups. Split testing is the same method under another name; some people reserve it for tests of whole pages at different addresses. Say that random assignment is what lets you call the difference an effect of the change.
Why it is asked: it checks you can define the method before using it. Read the full explanation: statistics for machine learning and data roles.
Why randomise users instead of comparing before and after?
A before-and-after comparison mixes the change with everything else that moved that week: season, campaigns, outages. Random assignment spreads those factors evenly across both groups, so the only systematic difference left is the change. That is the link to the older point that correlation is not causation, which the correlation versus causation page covers.
Why it is asked: it tests whether you understand why the design supports a causal claim.
What is the null hypothesis, and what does a p-value tell you?
The null hypothesis says the variant has no effect. A p-value is the probability of seeing a difference at least as large as the observed one if the null were true. It is not the probability that the variant works. A small p-value means the data would be surprising under no effect; it says nothing about size.
Why it is asked: misreading p-values is the most common error. Read the full explanation: the t test.
How do you choose the metric and the guardrails?
Pick one primary metric tied to the goal, sometimes called an overall evaluation criterion, that the change should move, and a few guardrail metrics that must not get worse, such as latency or unsubscribes. Kohavi, Tang and Xu's book on trustworthy online experiments is a standard reference. Fix the list before the test starts.
Why it is asked: choosing metrics after seeing results invites false findings. Read the full explanation: product sense interviews.
Design and sample size
How do you calculate the sample size?
Fix a baseline rate, the smallest lift worth detecting, a significance level and a power, then solve for users per group. A rule of thumb is n = 16 times the variance divided by the squared difference (Miller, 2010, updated 2013). For a 10% baseline and a one-point lift that is about 14,400 per group; the full formula gives 14,752.1
Why it is asked: it separates people who can plan a test from people who read dashboards. Read the full explanation: statistics for machine learning and data roles.
What are minimum detectable effect and power?
The minimum detectable effect (MDE) is the smallest true difference the test is designed to detect. Power is the chance the test detects a real effect of that size, commonly set at 80%. Halving the MDE needs about four times the users, because the required sample grows with one over the effect squared. Choose the MDE from business value, not from what traffic allows.
Why it is asked: it checks you can trade sample size against sensitivity.
How long should the test run?
Until the planned sample size is reached, rounded up to whole weeks so each weekday appears equally, since behaviour differs between weekdays and weekends. Do not stop early because the result looks good. If traffic is low, accept a larger effect size or a longer run, instead of ending the test sooner.
Why it is asked: duration is where discipline shows. Read the full explanation: data analyst interview guide.
What is the unit of randomisation, and why does it matter?
It is the entity assigned to a variant, usually a user rather than a page view. If you randomise by session, one person can see both versions and contaminate the comparison. When users influence each other, as in a marketplace or social network, even user-level assignment leaks between groups, and you may need clusters or time-based switching.
Why it is asked: it reveals whether you think about independence. Read the full explanation: machine learning system design.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Reading the results
Should the test be one-sided or two-sided?
Use a two-sided test unless a change in only one direction would lead to a different decision and you decided that before the test. A two-sided test flags harm as well as benefit, which matters because a change meant to help can hurt. Switching to one-sided after seeing the data halves the p-value for free, and breaks the error rate you planned.
Why it is asked: it tests whether you protect against wishful analysis. Read the full explanation: the t test.
The variant shows an 8% relative lift with p = 0.064. Do you ship it?
Not on that evidence alone. With 1,000 of 10,000 control users converting against 1,080 of 10,000, the lift is 8% relative, the z statistic is 1.85 and the two-sided p-value is 0.064; the 95% interval for the difference runs from about minus 0.05 to plus 1.65 points.1 Zero is inside it. Say what you would do next: extend to the planned size or accept the risk.
Why it is asked: it tests judgment under a borderline result. Read the full explanation: the t test.
What is peeking, and why is it a problem?
Peeking is checking significance repeatedly and stopping when it first looks significant. Each check is another chance for noise to cross the threshold, so the real false-positive rate exceeds the nominal 5%. Miller shows the effect. In our simulation of 20,000 no-effect tests with ten looks, 19.42% declared a winner at some look, against 4.94% at the final look only.1
Why it is asked: it is the classic way tests go wrong. Read the full explanation: statistics for machine learning and data roles.
Text description of the diagram
- In our simulation 19.42% of no-effect tests declared a winner at some look.
- Only 4.94% did so at the final look alone.
What is the multiple comparisons problem?
If you test many metrics or segments, some will look significant by chance. With 20 independent metrics at the 5% level, the chance of at least one false positive is 64.15%.1 Control it by naming one primary metric in advance, or by tightening the threshold: dividing 0.05 by 20 gives 0.0025, the Bonferroni correction.
Why it is asked: it checks you do not fish for significance.
How do you read a confidence interval?
A 95% confidence interval is a range built so that, over many repeated experiments, about 95% of such ranges contain the true difference. It shows size and uncertainty together: an interval that spans zero is consistent with no effect, and a narrow interval far from zero is a clear result. Report the interval with the p-value.
Why it is asked: it separates statistical from practical significance. Read the full explanation: the Gaussian distribution.
Trustworthiness
What is sample ratio mismatch, and how do you check it?
A sample ratio mismatch (SRM) is the situation in which the observed split of users differs from the intended one (Fabijan et al., 2019). Check it with a goodness-of-fit test. For a planned 50/50 split, 50,500 against 49,500 gives a p-value of 0.00157, which is a flag; 502 against 498 gives 0.899, which is fine.1
Why it is asked: an SRM means the data cannot be trusted. Read the full explanation: data analyst interview guide.
What do you do when you find an SRM?
Do not read the result. The cause is usually in assignment, logging or filtering, and the fix depends on the root cause, which the SRM FAQ says may need significant debugging. It reports that about 6% of Microsoft experiments and about 10% of LinkedIn triggered experiments showed one.
Why it is asked: it tests whether you protect decisions from bad data.
What is the novelty effect, and how do you identify it?
The novelty effect is a temporary lift because a feature is new, so early results overstate the lasting effect. The opposite, a temporary dip while users adjust, is the primacy effect, also called change aversion (Kohavi, Tang and Xu, 2020). Identify either by plotting the effect by day since first exposure and comparing new with returning users. If it decays or grows, run longer or keep a long-term holdout.
Why it is asked: a short test can mislead about the long term. Read the full explanation: product sense interviews.
Variance reduction and limits
What is an A/A test, and why run one?
An A/A test splits users into two groups that both get the same experience. A healthy system shows no significant difference about 95% of the time at the 5% level, so a higher rate points to a faulty assignment, logging or analysis step. The SRM FAQ lists A/A experiments among its rules of thumb for diagnosing causes (Vermeer, read October 2026).
Why it is asked: it shows you validate the measuring instrument before trusting it.
What is CUPED variance reduction?
CUPED (controlled-experiment using pre-experiment data) adjusts each user's metric using their behaviour before the experiment, which removes variation that has nothing to do with the treatment. The Bing paper (2013) reports a variance reduction of about 50%. In general variance falls by the square of the correlation: a correlation of 0.7 cuts it by 49%.1
Why it is asked: it shows you know how to get power without more traffic.
When can you not run an A/B test?
When traffic is too low to reach the sample size, when users interact so that groups cannot be separated, when the change cannot be randomised, or when withholding the change is unethical. Then use alternatives such as a before-and-after study with a comparison group, and say what causal claim each can and cannot support.
Why it is asked: it tests whether you know the limits of the method. Read the full explanation: machine learning system design.
How do you prepare for an A/B testing interview?
Practise computing a sample size and a p-value by hand once, then explain peeking, SRM and multiple comparisons aloud. Rehearse the borderline-result question above. Link the answer to a decision: what you would ship and why. See also distribution shift for what changes after launch.
Why it is asked: it shows how you learn.
- Randomisation is what turns a difference into an effect; fix the metric, the sample size and the stopping rule first.
- A p-value is not the chance the variant works; report an interval and the size of the effect.
- Peeking and many metrics inflate false positives: 19.42% with ten looks in our simulation, 64.15% with 20 metrics.
- Check the split before the result: a sample ratio mismatch means do not read the numbers.
Sources
- How Not To Run An A/B Test — Evan Miller (published 2010-04-18, updated May 2013), 1 May 2013
- Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data — WSDM 2013 (Deng, Xu, Kohavi, Walker), 4 Feb 2013
- Diagnosing Sample Ratio Mismatch in Online Controlled Experiments: A Taxonomy and Rules of Thumb for Practitioners — KDD 2019 (Fabijan et al.), page by Lukas Vermeer, 25 Jul 2019
- Sample Ratio Mismatch: frequently asked questions — Lukas Vermeer (page shows no date; read 2026-10-24), 24 Oct 2026
- Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing — Cambridge University Press (Kohavi, Tang, Xu), 2 Apr 2020
