In this article
Core ideas
What are the null and alternative hypotheses?
The null hypothesis is the default claim of no effect or no difference, written H0, for example that a population mean equals a stated value. The alternative hypothesis (HA) is what you suspect instead. A test asks whether the data are too incompatible with H0 to keep it. The alternative can point one way, greater or less, or both ways, not equal (Penn State statistics review, accessed October 2026).
Why it is asked: it shows whether you can frame a question before touching data. Read the full explanation: statistics interview guide.
What is a p-value, in plain words?
A p-value is the probability, under a specified statistical model such as the null hypothesis, of seeing a summary of the data at least as extreme as the one observed (American Statistical Association (ASA) statement, 2016). A small value means the data fit the null model poorly, if the model's other assumptions hold (ASA, 2016). It measures incompatibility, not how likely a hypothesis is.
Why it is asked: it is the most misquoted idea in statistics. Read the full explanation: t test and its p-value.
Does a p-value of 0.03 mean there is a 3% chance the null hypothesis is true?
No. The ASA states that p-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone (ASA, 2016). The p-value assumes the null is true and asks how surprising the data are. The reverse question needs prior beliefs, which is Bayesian territory. The 0.03 here is illustrative.
Why it is asked: it is the standard misconception check. Read the full explanation: Bayes theorem interview questions.
When do you use a one-tailed versus a two-tailed test?
The direction of the alternative decides. A right-tailed test uses "greater than", a left-tailed test "less than", and a two-tailed test "not equal" (Penn State, accessed October 2026). Choose the tail before seeing data, and use one tail only when an effect in the other direction would be treated like no effect. In the z-test example below, the two-sided p-value is 0.0455 and the one-sided value is 0.0228.1
Why it is asked: it tests whether you pick the test before looking at the result. Read the full explanation: t test.
What does "fail to reject the null" mean, and why not "accept"?
It means the data gave too little evidence against the null, not that the null is true: a Type II error, missing a real effect, is always possible. Penn State also notes that rejecting the null does not prove the alternative true (Penn State, accessed October 2026). Say "no significant evidence of a difference" rather than "proved there is none".
Why it is asked: the wording shows whether you understand what a test can and cannot conclude. Read the full explanation: t test.
Errors, power and sample size
What are Type I and Type II errors?
A Type I error rejects a null hypothesis that is true. A Type II error fails to reject one that is false (Penn State, accessed October 2026). The probability of the first is α and the probability of the second is β. In the criminal-trial analogy (University of Texas, accessed October 2026), convicting an innocent person is Type I and freeing a guilty one is Type II.
Why it is asked: it is the vocabulary every other answer depends on. Read the full explanation: probability interview questions.
What is the significance level α, and who chooses it?
The significance level is the probability α of rejecting the null when it is true, and the analyst sets it before collecting data (University of Texas, accessed October 2026; National Institute of Standards and Technology (NIST) Special Publication (SP) 800-22, 2010). The usual 0.05 is a convention, and the ASA warns against deciding only by whether a p-value passes a threshold (ASA, 2016).1
Why it is asked: fixing α first stops you choosing a cutoff to suit the result. Read the full explanation: probability interview questions.
What is statistical power, and what affects it?
Power is the probability of correctly rejecting a false null hypothesis, and you want it high, close to 1 (Penn State Statistics (STAT) 800, accessed October 2026). It rises with a larger sample, a larger true effect, a larger α and less noise in the data. Unlike α, β is not a single fixed value (NIST, 2010); it depends on the true effect and on n, and with n fixed, lowering α raises β (Penn State power analysis, accessed October 2026). In the example below, power is 0.67 when the true mean is 106.1
Why it is asked: it separates people who only know α from people who can design a test. Read the full explanation: A/B testing interview questions.
Why does lowering α raise the chance of a Type II error?
With the sample size fixed, decreasing α increases β, and lowering both needs more data (Penn State power analysis, accessed October 2026). A stricter cutoff makes you reject less often, including when the null is false. For the same data in the worked example, power falls from 0.67 at α = 0.05 to 0.43 at α = 0.01.1
Why it is asked: it tests whether you see the trade-off, not two independent dials. Read the full explanation: A/B testing interview questions.
How do you work out the sample size you need?
Fix α, the power you want and the smallest effect worth detecting, then solve for n. For a one-sample, two-sided z-test with known σ, the formula is:
Here z for 1 − α/2 is the normal cutoff for the significance level (1.96 at α = 0.05), z for 1 − β is the cutoff for the power (0.8416 at a power target of 0.80, a common convention), σ is the standard deviation and δ is the smallest effect worth detecting. With σ = 15 and δ = 6, n = (2.8016 × 15 / 6)² = 49.06, so round up to 50.1 Two-sample tests use a different formula. Choose δ from business meaning, not from the data.
Why it is asked: experiments are designed before they are run. Read the full explanation: A/B testing interview questions.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Reading and running a test
How do you run a z-test step by step?
Take H0: μ = 100 with σ = 15 known, n = 36 and an observed sample mean of 105 (illustrative). The standard error is 15/√36 = 2.5, so z = (105 − 100)/2.5 = 2. The two-sided p-value is 0.0455, below 0.05, so you reject H0.1 State the conclusion in context and report the effect size, not only the verdict.
Why it is asked: it checks that you can do the arithmetic, not only recite definitions. Read the full explanation: Gaussian distribution.
When do you use a t-test instead of a z-test?
Use a t-test when the population standard deviation is unknown and estimated from the sample, which is the usual case; the z-test applies when it is known. The t distribution has heavier tails to account for that extra uncertainty and approaches the normal distribution as the sample grows. Both end by comparing a statistic with a critical value or converting it to a p-value.
Why it is asked: choosing the right test is the first practical decision. Read the full explanation: t test.
What assumptions does a t-test make, and what if they fail?
A t-test assumes independent observations and a sample mean that is roughly normal, which holds when the data are roughly normal or the sample is large enough for the central limit theorem to help. The classic two-sample version also assumes equal variances; Welch's t-test drops that. For small, heavily skewed samples, use a rank-based test such as Mann–Whitney, or a permutation or bootstrap test. None of these fixes non-independent data (NIST statistical methods handbook, accessed October 2026).
Why it is asked: it shows you know a test is only valid under conditions. Read the full explanation: t test.
When do you use a chi-square test?
A chi-square test compares observed counts in categories with the counts expected under the null hypothesis. Use the goodness-of-fit version for one categorical variable against a stated distribution, and the test of independence for two categorical variables in a table, such as variant against converted. It needs counts, not percentages, and expected counts that are not very small; for small tables use Fisher's exact test (NIST statistical methods handbook, accessed October 2026).
Why it is asked: much product data is categorical, so t-tests do not apply. Read the full explanation: A/B testing interview questions.
How is a confidence interval related to a hypothesis test?
For the same data and level, a two-sided test rejects a value exactly when the confidence interval excludes it. In the z-test example, the 95% interval for the mean runs from 100.1 to 109.9, which excludes 100 and matches the rejection.1 The interval also shows how large the effect might be, which a p-value cannot.
Why it is asked: it tests whether you can connect two views of the same evidence. Read the full explanation: t test.
Does a small p-value mean the effect is important?
No. Statistical significance asks whether an observed effect is due to chance; practical significance asks whether it is large enough to be useful in the real world (Penn State STAT 800, accessed October 2026). With σ = 15 and a very large sample of 10,000, a true difference of only 0.5 gives a p-value below 0.001.1 Report the effect size and an interval.
Why it is asked: it separates a result from a decision. Read the full explanation: A/B testing interview questions.
Does a significant result prove causation?
No. A test measures how compatible the data are with the null; whether a treatment caused a difference depends on study design, such as random assignment, not on the p-value. Even rejecting the null does not prove the alternative true (Penn State, accessed October 2026).
Why it is asked: it checks that you do not over-claim from data. Read the full explanation: correlation versus causation.
Multiple tests and method
What is the multiple comparisons problem?
The family-wise error rate (FWER) is the probability of at least one false positive among a family of tests (MedCalc manual, accessed October 2026). For m independent tests at α = 0.05 it is 1 minus 0.95 to the power m: 0.2262 for five tests and 0.6415 for twenty.1 MedCalc gives the same 0.64 for twenty. Running many tests and reporting any hit is therefore misleading.
Text description of the diagram
- For independent tests at α = 0.05 the chance of a false positive is 1 minus 0.95 to the power m.
- That chance is 0.2262 for five tests and 0.6415 for twenty.
Why it is asked: it is the most common real-world way to fool yourself with p-values. Read the full explanation: A/B testing interview questions.
How does the Bonferroni correction work, and what does it cost?
Bonferroni divides α by the number of tests m: 0.05 divided by five tests is 0.01, and by twenty is 0.0025, which keeps the family-wise rate at or below α (MedCalc manual, accessed October 2026).1 The cost is power: a stricter per-test cutoff makes real effects harder to detect, because lowering α raises β (Penn State power analysis, accessed October 2026). Controlling the false discovery rate, the expected share of false positives among your discoveries, is a less strict alternative.
Why it is asked: it tests whether you know the fix and its price. Read the full explanation: probability interview questions.
Why can't you stop an A/B test the moment the p-value dips below the threshold?
Each look at the data is another test, so checking repeatedly and stopping at the first small p-value pushes the false-positive rate above α, by the same logic as multiple comparisons; Armitage, McPherson and Rowe (1969) analyzed this inflation for repeated significance tests on accumulating data. Fix the sample size or a stopping rule in advance, or use a method built for repeated looks.
Why it is asked: peeking is the classic experiment mistake in product teams. Read the full explanation: A/B testing interview questions.
What is the difference between Fisher's p-value approach and Neyman–Pearson testing?
Fisher treated the p-value as graded evidence against the null. Hypothesis tests came later, with the 1933 paper by Neyman and Pearson, which introduced Type I and Type II errors and formalized power (Lew, 2019). In that workflow you choose α and a sample size before the experiment and decide by a threshold. Textbooks blend the two traditions.
Why it is asked: it shows depth beyond the textbook recipe. Read the full explanation: probability interview questions.
Follow-up questions
What hypothesis testing questions are asked in statistics interviews?
The twenty-one above cover the recurring ones: hypotheses, p-values, error types, power, sample size, z and t tests, test assumptions, chi-square tests and multiple comparisons. A common follow-up is to explain a p-value to a non-expert and then to critique a result. The statistics interview guide places them among the other topics of a data or machine learning round.
How do I prepare for hypothesis testing questions?
Practice saying the definition, the assumption it needs and the failure mode in three sentences, then redo the z-test arithmetic with a calculator. Pair it with probability interview questions, since tests rest on distributions and conditional probability.
What if the interviewer asks for a Bayesian view of a test?
A Bayesian answer gives the probability of a hypothesis given the data, which a p-value does not. It needs a prior belief, and Bayes' rule combines it with the likelihood. The Bayes theorem questions walk through that calculation.
Which distribution does a z-test rely on?
A z-test compares a standardized statistic with the standard normal distribution, which is why the normal curve matters; see what a Gaussian distribution is.
- A p-value is the probability of data at least this extreme under the null model; it is not the probability the null is true.
- Type I error is rejecting a true null (α); Type II is missing a false one (β); power is one minus β.
- With a fixed sample size, lowering α lowers power; more data lowers both errors.
- Statistical significance is not practical significance, and fail to reject is not proof of the null.
- Many tests inflate false positives; Bonferroni divides α by the number of tests and costs power.
Sources
- The ASA Statement on p-Values: Context, Process, and Purpose — American Statistical Association / The American Statistician (Wasserstein and Lazar), 7 Mar 2016
- ASA Statement on p-Values with commentary (Yale-hosted copy) — American Statistical Association, 7 Mar 2016
- NIST SP 800-22 Rev 1a, A Statistical Test Suite — National Institute of Standards and Technology, 1 Jan 2010
- S.3 Hypothesis Testing — Penn State Eberly College of Science, 26 Oct 2026
- 5.4 Further Considerations for Hypothesis Testing (STAT 800) — Penn State Eberly College of Science, 26 Oct 2026
- S.5 Power Analysis — Penn State Eberly College of Science, 26 Oct 2026
- Type I and II Errors and Significance Levels — University of Texas at Austin (M. Sheehy), 1 Jun 2011
- Multiple testing — MedCalc manual, 11 Aug 2026
- A reckless guide to P-values (arXiv 1910.02042) — arXiv (Michael J. Lew), 1 Oct 2019
- Repeated significance tests on accumulating data — Journal of the Royal Statistical Society, Series A (Armitage, McPherson and Rowe), 1 Jan 1969
- NIST/SEMATECH e-Handbook of Statistical Methods — National Institute of Standards and Technology, 1 Jan 2012
