In this article
Experiment design and sample size
The first seven questions are the ones asked before any data exists. Analyst rounds sit inside the wider data analyst interview guide, and the query side of the same rounds is in the Structured Query Language (SQL) interview questions.
What is an A/B test, and what is the null hypothesis?
An A/B test is a controlled experiment that randomly assigns users to a control arm and a variant arm, then compares a metric between them. The null hypothesis is the claim that nothing differs; for means it says the two population means are equal, tested against the alternative that they are not. The National Institute of Standards and Technology (NIST) handbook, written jointly with the Semiconductor Manufacturing Technology (SEMATECH) consortium, notes a common use is testing whether a new treatment beats the current one.
Why it is asked: it checks that you can say what is being tested before touching data. Read the full explanation: NIST handbook, two-sample t-test.
How do you choose the primary metric and the guardrail metrics?
Pick one primary metric tied to the goal of the change, and fix it before launch. A guardrail metric is a critical metric designed to alert experimenters about violated assumptions. Kohavi, Tang and Xu (2020, ch. 21) name two types: organisational and trust-related. Page speed is an organisational example; the split check below is a trust-related one.
Why it is asked: it separates "what moved" from "what we promised to protect". Read the full explanation: Kohavi, Tang and Xu, ch. 21.
What is the randomisation unit, and why does it matter?
The randomisation unit is the thing assigned to an arm, usually the user rather than the page view. Choose the unit whose behaviour the metric measures and that cannot leak between arms; if one user can see both versions, the arms blur. Analysing page views when users were randomised understates the standard error, so the p-value comes out too optimistic.
Why it is asked: a wrong unit quietly breaks every later step.
How do you calculate the sample size for an A/B test?
Fix the significance level, the power and the effect you care about, then solve for users per arm. For two conversion rates, NIST Technical Note 2106 squares a sum of two z-values scaled by the spread and divides by the squared gap between the rates. A baseline of 10% rising to 11%, two-sided 0.05 and 80% power, needs about 14,750 per arm.1
Why it is asked: it shows you plan the test instead of stopping when the number looks good. Read the full explanation: NIST Technical Note 2106 and NIST/SEMATECH, sample sizes.
What is the minimum detectable effect, and why does it drive the cost?
The minimum detectable effect is the smallest true change a test is sized to detect. Required sample grows with the inverse square of it: halving the lift from one point to half a point raises the need from about 14,750 to roughly 58,000 per arm, close to four times.1 Set it from the smallest change worth shipping, not from the result you hope for.
Why it is asked: it tests whether you link business value to test length. Read the full explanation: NIST Technical Note 2106.
What is statistical power?
Statistical power is the probability that a test detects an effect of a given size when that effect is real. It rises with sample size. For a difference of half a standard deviation at the 0.05 level, Statology's R output gives power of about 18% at 10 per group, 48% at 30 and 70% at 50. Plan for a high target, such as 80%, as this page does.
Why it is asked: it separates "not significant" from "not enough data". Read the full explanation: Statology, minimum sample size for a t-test.
Does a t-test need at least thirty observations per group?
No. Statology states there is no minimum sample size required to perform a t-test. The real cost of a small sample is lower power, so a real effect is often missed. Thirty is a habit, not a rule. In A/B tests, sizing for small lifts usually demands far more users than thirty anyway.
Why it is asked: it catches a memorised rule applied without understanding. Read the full explanation: Statology, minimum sample size for a t-test.
Checking the experiment is valid
What is a sample ratio mismatch, and why check it?
A sample ratio mismatch is a gap between the observed split of users across arms and the designed split, for example a 50/50 test that does not come out near 50/50. Kohavi, Tang and Xu say the SRM check belongs in every experiment, because it protects internal validity and the trustworthiness of the results.
Why it is asked: it shows whether you validate the data before celebrating a lift. Read the full explanation: Kohavi, Tang and Xu, ch. 21.
How do you test for SRM, and what do you do if it fails?
The check compares the observed user counts per arm with the designed split, using a chi-square test with one degree of freedom. A published case had 579,286 visitors, so 289,643 expected per arm. With an illustrative split of 287,500 and 291,786, chi-square is about 31.7 although the shares look close (49.6% and 50.4%).1 A mismatch means the results are not trusted until the cause is found.
Text description of the diagram
- The check compares the observed user counts per arm with the designed split.
- A mismatch means the results are not trusted until the cause is found.
Why it is asked: it tests whether you know a small p-value cannot rescue broken assignment. Read the full explanation: Guess The Test, SRM explained.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Choosing a test and reading the result
Which statistical test do you use for a conversion rate and for revenue per user?
For a conversion rate, compare two proportions. For a mean such as revenue per user, use a two-sample t-test: the difference in sample means divided by the square root of the sum of each group's variance over its size (NIST). In an illustrative case of 2,000 observations per arm, means 10.20 and 10.50 and standard deviations 4.0 and 4.2, t is about 2.31 and the two-sided p-value about 0.02.1
Why it is asked: it checks you match the test to the metric type. Read the full explanation: NIST/SEMATECH, two-sample t-test.
Does the t-test require equal variances?
No. NIST says the variances of the two samples may be assumed equal or unequal, and the unpaired statistic in the previous answer keeps each group's variance separate instead of pooling them. When arms differ in size or spread, the unequal-variance form is the safer default. NIST Technical Note 2106 also has a section titled "Student's t-test: Unequal variance case".
Why it is asked: it probes assumptions behind a routine test. Read the full explanation: NIST/SEMATECH, two-sample t-test.
How do you explain a p-value?
A p-value is the probability of seeing a difference at least as extreme as the observed one, in either direction, if the null hypothesis were true. It is not the probability that the variant is better, and it is not the size of the effect. Fix the significance threshold before the test starts; a level of 0.05 is the one used in the sizing example above, computed in the same script.1
Why it is asked: misreading it is the most common error in analyst rounds. Read the full explanation: NIST/SEMATECH, two-sample t-test.
What is a confidence interval, and how do you read it in an A/B test?
A confidence interval is a range of values for the true effect, built so that 95% of intervals made this way would contain it at the 0.05 level. Read it against the smallest lift worth shipping. A false positive is a type I error, which the 0.05 level caps; a false negative is a type II error, whose rate is one minus the power, so 80% power means 20%. A two-sided test looks for a change in either direction.
Why it is asked: it checks you can read an estimate and its uncertainty, not only a verdict.
The result is statistically significant but small. Do you ship?
Significance says a difference is unlikely to be noise, not that it matters. Compare the estimated lift and its interval with the effect worth shipping and the cost of the change. Illustratively, take two arms of equal size, each with a hundred thousand visitors, converting at 10.0% and 10.3%: the lift is 0.3 points, a relative lift of 3%, and z (the gap measured in standard errors) is about 2.22, a p-value of about 0.03, so the result is significant. If the team needs at least +0.5 points to cover the build cost, the 95% interval of +0.04 to +0.56 points mostly sits below that bar, so do not ship on this result alone.1
Why it is asked: it separates statistical from practical significance.
What goes wrong when you test many metrics or segments?
Each extra metric or segment is another chance of a false positive. With twenty independent metrics each tested at 0.05, the chance of at least one false positive is about 64%.1 The Bonferroni correction divides the threshold by the number of tests, giving 0.0025 here. Better still, name one primary metric before launch and treat the rest as exploratory.
Why it is asked: it tests whether you resist digging for a winner.
Why is peeking at results and stopping early a problem?
Checking repeatedly and stopping at the first significant look inflates false positives. In our simulation of tests with no true difference and five equally spaced looks at the 0.05 level, about 14% ended significant at some look, not 5%.1 Fix the sample size and stopping rule in advance, or use a sequential method built for repeated looks.
Why it is asked: stopping early is the tempting mistake in real work.
Pitfalls and the ship decision
What are novelty and primacy effects?
A novelty effect is an early lift from users reacting to something new, which fades. A primacy effect is the reverse: existing users resist a change at first. Look at the lift by day or by new and returning users, and run the test through whole weekly cycles, so a short-lived swing is not read as the lasting effect.
Why it is asked: it checks you think about time, not only one pooled number. Read the full explanation: how to answer a metric dropped case.
The result is flat. What do you tell the team?
Report it honestly: no difference was detected at the sample you had. Then check the interval width against the effect worth shipping. If the interval rules out any worthwhile effect, stop and move on; if it is wide, the test lacked power and needs more data or a larger change. "No significant difference" is not "no difference".
Why it is asked: it tests calibrated communication when the answer is disappointing. Read the full explanation: Statology, minimum sample size for a t-test.
How do you decide whether to ship?
Work in order. First confirm validity: the split check passes and the other guardrails hold. Then read the primary metric with its interval against the pre-set threshold. Then weigh cost, risk and any segment that moved the wrong way. Ship when a trusted result clears the threshold, no guardrail is violated and the lift is worth the cost of running the change.
Why it is asked: it shows whether you can turn statistics into a decision.
Follow-up questions
What are common A/B testing questions in an analyst interview?
The nineteen above cover the recurring themes: hypotheses, sample size and power, the split check, the choice of test, p-value and interval reading, error types, multiple metrics, early stopping and the ship decision. For the whole round, start from the data analyst interview guide and practise the data-pulling side with SQL interview questions.
How do I answer an A/B test question out loud?
Use the same order as the experiment: the hypothesis and primary metric, the sample size, the validity checks, the test and the decision. Say the trade-off aloud, for instance why you fixed the stopping rule. For explaining a result to a non-technical listener, the behavioral interview guide covers the communication side.
Do analysts need Excel or Tableau for A/B test results?
The statistics do not depend on a tool, but results are often summarised in spreadsheets or dashboards. If the round tests those, see Excel interview questions and Tableau developer interview questions.
What is a good p-value for an A/B test?
There is no universal good value. Choose the significance threshold before the test, such as 0.05, and size the test for it. A p-value below the threshold is not proof, and the effect size still has to be worth shipping.
- State the null hypothesis, one primary metric and the minimum detectable effect before the test starts.
- Sample size grows with the inverse square of the effect: half the lift needs close to four times the users.
- Check the traffic split first; a sample ratio mismatch means the results are not trusted until the cause is found.
- Match the test to the metric, and read significance together with effect size and the interval.
- Many metrics and repeated looks inflate false positives, so fix the metric, sample size and stopping rule in advance.
Sources
- Sample Ratio Mismatch and Other Trust-Related Guardrail Metrics (ch. 21, Trustworthy Online Controlled Experiments) — Cambridge University Press (Kohavi, Tang, Xu), 13 Mar 2020
- 1.3.5.3 Two-Sample t-Test for Equal Means (NIST/SEMATECH e-Handbook of Statistical Methods) — NIST/SEMATECH, 26 Oct 2026
- 7.2.2.2 Sample sizes required (NIST/SEMATECH e-Handbook of Statistical Methods) — NIST/SEMATECH, 26 Oct 2026
- NIST Technical Note 2106: Comparing Instruments — NIST, 26 Oct 2026
- Sample Ratio Mismatch (SRM) Explained — Guess The Test, 26 Oct 2026
- The Minimum Sample Size for a t-test — Statology, 26 Oct 2026
