In this article
  1. Before you start
  2. Why interviewers ask this
  3. The t test from first principles
  4. Worked example
  5. Trade-offs and edge cases
  6. How to answer it in an interview
  7. Common mistakes
  8. Follow-up questions
  9. Sources

Before you start

This page is part of the probability and statistics guide.

Why interviewers ask this

The t test is a common first question in a statistics or experimentation round because one answer shows whether you understand sampling noise, not only a formula. A weak answer recites "if p is below 0.05, the result is significant." A strong answer says what is being compared, what the null hypothesis claims, why the gap is divided by a standard error, and what the p-value does not mean.

The question comes in many forms: compare the old and new checkout page, test whether a model's accuracy changed, explain a p-value to a doctor, or say which error is worse in a diagnosis. All of them rest on the same five steps below. Data science, analyst and machine learning roles use it, because an experiment result without an error estimate cannot support a decision, and the interview loop overview shows where such a round sits.

The t test from first principles

Null hypothesis
The null hypothesis is the default claim a test tries to find evidence against, usually that there is no difference or no effect, such as "the two population means are equal".
Standard error
The standard error of a mean difference is the typical size of the gap between two sample means that sampling luck alone would produce; it shrinks as the groups get larger.
Degrees of freedom
The degrees of freedom of a t test are the number of independent pieces of information left for estimating spread; they set which t distribution, and so which cut-off, the test uses.

In plain words

Imagine two buckets of apples from two orchards. The first bucket weighs a little more on average. Is that orchard better, or did you just happen to pick heavier apples? A t test answers by asking how big a gap luck alone could make with buckets of this size and this much variation among apples. The analogy breaks in one place: a t test says nothing about why one orchard differs, only whether the gap looks bigger than chance.

Stated precisely, the test compares a sample mean with a value, or two sample means. It divides the gap by an estimate of its noise. William Sealy Gosset published it in 1908 in Biometrika (Student, 1908) under the pseudonym "Student", while working for the Guinness brewery in Dublin (Wikipedia, read October 2026). The "t" is the name of the statistic.

Step 1: State the null hypothesis and the alternative

A test starts by writing down two competing claims. The null hypothesis says the population means are equal, written as μ1=μ2. The alternative says they differ, or, for a one-sided test, that one is larger (NIST e-Handbook, two-sample t-test, read October 2026). You also choose the significance level before looking at the data, most often 0.05, which is the false-alarm rate you accept. Choosing it afterwards is how people talk themselves into results.

For the running example, an old checkout page against a new one, the null is "the average load time is the same on both pages" and the alternative is "it differs". You now know what the test can reject and what it cannot: it can reject equality, and it cannot say why the gap exists; random assignment of users to pages is what lets you read the gap as caused by the new page.

Step 2: Measure the gap and its standard error

The gap is the difference of the two sample means. In the illustrative example, 10 page loads of the old page averaged 12.36 seconds, and 10 loads of the new page averaged 11.38 seconds, a gap of 980 milliseconds.1 The standard error says how large such a gap is by luck alone. For two groups with standard deviations s1 and s2 and sizes n1 and n2, it is the square root of the whole sum (s1 squared over n1) plus (s2 squared over n2) (NIST e-Handbook, read October 2026):

SE=s12n1+s22n2

A variance is a standard deviation squared, and each term inside the root is the variance of one sample mean, the group variance divided by its size. Here the group variances are 0.4027 and 0.2351 (the standard deviations 0.6346 and 0.4849, squared), so the two terms are 0.4027 over 10 = 0.04027 and 0.2351 over 10 = 0.02351, and the standard error is the square root of 0.06378: 252.5 milliseconds.1 That is why the same gap is noise at 10 page loads per group and evidence at 160.

Step 3: Divide the gap by the standard error to get t

The t statistic is the gap divided by its standard error, so it counts how many standard errors apart the two means are:

t=x‾1−x‾2SE

In the example, 980 divided by 252.5 gives t = 3.881.1 A t near 0 means the gap is about what luck produces; a t far from 0 in either direction means it is not. Dividing is what makes the number comparable across experiments: a gap of one second means little on its own, but a gap of four standard errors means the same thing whether the units are seconds, dollars or clicks. This is the whole idea of the test. The remaining steps only translate the ratio into a probability and a decision. You now have one number to judge.

Step 4: Find the degrees of freedom and read the p-value

If the null is true, the t statistic follows a t distribution (for Welch's version, approximately), a bell-shaped curve with heavier tails than the normal curve, and its exact shape depends on the degrees of freedom. For the unequal-variance version of the test, called Welch's t test, the degrees of freedom come from the Welch-Satterthwaite formula, which the National Institute of Standards and Technology (NIST) says holds up when sample sizes or variances differ (NIST e-Handbook, read October 2026). Write v1 for s1 squared over n1 and v2 for s2 squared over n2, the two variances of the means from Step 2:

ν=(v1+v2)2v12n1−1+v22n2−1

Here v1 is 0.04027 and v2 is 0.02351, so (0.04027 + 0.02351) squared over (0.04027 squared over 9 plus 0.02351 squared over 9) gives 16.84.1 The p-value is the chance of a t at least as far from 0 as 3.881, in either direction, if the null were true: 0.0012.1 The American Statistical Association (ASA) states that p-values do not measure the probability that the studied hypothesis is true (ASA, March 2016).

Step 5: Decide, and report the size of the effect

Compare the p-value with the significance level chosen in Step 1. Equivalently, reject the null when the absolute value of t exceeds the critical value for the chosen level and degrees of freedom (NIST e-Handbook, read October 2026). Here the critical value is 2.111 and t is 3.881, and p of 0.0012 is below 0.05, so the null is rejected.1 Then report more than the verdict. The 95 percent confidence interval for the gap, a range of gaps that the data do not rule out, is the gap plus or minus the critical t times the standard error:

(x‾1−x‾2)±tcrit×SE

Here that is 980 plus or minus 2.111 times 252.5 milliseconds, which is 447 to 1,513 milliseconds.1 The ASA adds that a p-value does not measure the size or importance of an effect, so a decision needs the gap and its interval, not only p.

A t test turns a difference of means into a p-value in five stepsFive stages in a row: state the hypotheses, measure the difference of means, divide by the standard error to get t, find the degrees of freedom and the p-value, then decide. The checkout example has a difference of 980 milliseconds, t of 3.881 and p of 0.0012.Hypothesesnull: meansequalDifferencegap andstandard errort statistic3.881p-value0.0012Decisionreject at 0.05Diagram: Interview Clozo, CC BY 4.0 ·interview.clozo.ai/blog/t-test.html

The t statistic is the difference of means measured in standard errors; the p-value converts it into a probability.CC BY 4.0 — source: What is a t test, and how do you run one?
Step-by-step description
  1. Null: the means are equal.
  2. Gap 980 milliseconds; standard error 252.5 milliseconds.
  3. t is 980 over 252.5, so 3.881.
  4. 3.881 on 16.84 degrees of freedom gives p of 0.0012.
  5. p is below 0.05: reject the null.
  • In the illustrative checkout example the difference of means is 980 milliseconds and the standard error is 252.5 milliseconds.
  • That gives t = 3.881 on about 16.84 degrees of freedom, with a two-sided p-value of 0.0012.

Knowing it is half the job. Saying it is the other half.

Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.

Start Free Trial →

Worked example

The data are illustrative: 10 page-load times in seconds for each of two checkout pages, invented for this page. Old page: 12.1, 11.4, 13.0, 12.6, 11.8, 12.9, 13.4, 12.2, 11.7, 12.5. New page: 11.2, 10.8, 12.0, 11.5, 11.1, 12.3, 10.9, 11.6, 11.4, 11.0. The standard deviations are sample standard deviations, which divide by n minus 1 (dividing by n would give 602 milliseconds for the old page, not 635). The script computes the test by hand from the formulas above and with scipy, and the two agree.1

The two checkout pages differ by 980 milliseconds, which is 3.881 standard errors, so the null of equal means is rejected at the 0.05 level.
QuantityValue
Old page: mean and standard deviation (n = 10)12.36 seconds and 635 milliseconds (illustrative)
New page: mean and standard deviation (n = 10)11.38 seconds and 485 milliseconds (illustrative)
Difference of means980 milliseconds
Standard error of the difference252.5 milliseconds
t statistic, Welch3.881
Degrees of freedom, Welch16.84
Two-sided p-value0.0012
95 percent confidence interval for the gap447 to 1,513 milliseconds
Critical t at 0.05 (two-sided, 16.84 degrees of freedom)2.111

The pooled version, which assumes equal variances, gives exactly the same t of 3.881 here, because with equal group sizes its standard error is algebraically the same as Welch's; it differs only in the degrees of freedom, 18 instead of 16.84, so its p-value is 0.0011 instead of 0.0012.1 The scipy documentation says its default assumes equal population variances and that setting equal_var=False runs Welch's test (SciPy manual, read October 2026). NIST says Welch's version holds up when sample sizes or variances differ, and with equal group sizes, as here, the two versions give the same t, which is why defaulting to Welch's costs little.

The cut-off depends on sample size. A small sample needs a larger t, because the t distribution has heavier tails until the degrees of freedom grow. The two-sided 5 percent critical t values are 4.303 at 2 degrees of freedom, 2.571 at 5, 2.228 at 10, 2.042 at 30 and 1.984 at 100, and the normal-distribution value at the same level is 1.96.1

The t cut-off falls toward the normal 1.96 as the degrees of freedom growBars of the two-sided 5 percent critical t value for 2, 5, 10, 30 and 100 degrees of freedom: 4.303, 2.571, 2.228, 2.042 and 1.984, against the normal-distribution value 1.96.Critical value (t)012345251030100NormalDegrees of freedom4.3032.5712.2282.0421.9841.96Diagram: Interview Clozo, CC BY 4.0 ·interview.clozo.ai/blog/t-test.html
Small samples need a larger t to reach significance; by 100 degrees of freedom the cut-off is almost the normal one.CC BY 4.0 — source: What is a t test, and how do you run one?
Text description of the diagram
  • Two-sided 5 percent critical t values: 4.303 at 2 degrees of freedom, 2.571 at 5, 2.228 at 10, 2.042 at 30 and 1.984 at 100.
  • The normal-distribution value at the same level is 1.96.

Sample size also changes the verdict for a fixed gap. Holding a gap of 200 milliseconds and a standard deviation of 700 milliseconds in both groups, the p-value is 0.53 with 10 per group, 0.21 with 40, 0.011 with 160 and 0.00000037 with 640 page loads per group.1 The gap never changed; only the standard error did. The standard error of the 200-millisecond gap is 313 milliseconds at 10 page loads per group (t = 0.64) and 78 milliseconds at 160 (t = 2.56).1

Trade-offs and edge cases

The right t test depends on how the data were collected, and picking the wrong one is a frequent design error. One group against a fixed value uses the one-sample test, with n minus 1 degrees of freedom (NIST e-Handbook, one-sample test, read October 2026). Its statistic is the sample mean minus the fixed value, divided by the sample standard deviation over the square root of n:

t=x‾−μ0s/n

Here μ0 is the fixed value and s the sample standard deviation. The same people measured twice, or matched pairs, use the paired test, which analyses each pair's difference. Two separate groups use the two-sample test, with Welch's version when variances may differ (NIST e-Handbook, two-sample t-test, read October 2026).

The way the data were collected decides which t test to runDecision flow: ask how the data were collected. One group against a known value uses the one-sample test, the same units measured twice uses the paired test, and two separate groups use Welch's two-sample test.one grouppairstwo groupsHow were the datacollected?One-sample ttestone group vs afixed valuePaired t testsame unitsmeasured twiceWelch t testtwo separategroupsDiagram: Interview Clozo, CC BY 4.0 ·interview.clozo.ai/blog/t-test.html
Match the test to the design: one group, matched pairs or two independent groups.CC BY 4.0 — source: What is a t test, and how do you run one?
Text description of the diagram
  • One group compared with a fixed value uses the one-sample t test with n minus 1 degrees of freedom.
  • Matched pairs use the paired t test on the differences; two separate groups use the two-sample test, with Welch's version when variances may differ.

Treating paired data as two groups throws away the pairing. Take ten visitors whose page-load time is measured before and after a change (illustrative: they range from 6.1 to 15.3 seconds, and each is roughly 600 milliseconds faster afterwards). The paired test on the 10 differences gives t = 10.47 on 9 degrees of freedom, p = 0.0000024. Wrongly treating the same numbers as two groups gives t = 0.43, p = 0.67, because each visitor's own baseline stays in the noise.1 The checkout example above is different: its two groups are different page loads, so only the two-group test applies there.

Edge cases: the test assumes independent observations and sample means that are roughly normal (NIST e-Handbook, two-sample t-test, read October 2026), which the central limit theorem supports for large samples (Wikipedia, read October 2026). With a handful of heavily skewed values, a rank-based test or a resampling method is safer. Running many tests and keeping the best p-value breaks the 5 percent promise. In a simulation of 20,000 comparisons with no true difference, 4.6 percent were significant at 0.05, close to the promised 5, but that holds for one pre-planned test, not for twenty: with twenty independent tests at 0.05 the chance of at least one false alarm is about 64 percent.1

How to answer it in an interview

Interviewer asks

What is a t test, and how would you use it to compare two groups?

A strong answer

A t test asks whether a difference in means is bigger than sampling noise. I state the null that the means are equal, compute the gap, divide it by its standard error to get t, and convert that to a p-value using the degrees of freedom. I would use Welch's version so unequal variances do not distort the test, and I would report the confidence interval for the gap, not only whether p is below 0.05, because a tiny gap becomes significant with enough data.

The spoken structure: the question it answers, the ratio, the p-value meaning, then the caveat about effect size.

Common mistakes

  1. Reading p as the chance the null is true. (wrong) "A p-value of 0.03 means a 3 percent chance we are wrong." (right) "If there were no difference, data at least this extreme would turn up about 3 percent of the time." The ASA states that p-values do not measure the probability that the hypothesis is true.
  2. Equating significant with important. (wrong) "It is significant, so the effect is big." (right) "It is significant; the gap is 200 milliseconds, and I would check whether that matters." The p-value shrinks with sample size alone.
  3. Accepting the null. (wrong) "p is 0.4, so there is no difference." (right) "We failed to find evidence of a difference; the sample may be too small."
  4. Using a two-group test on paired data. (wrong) Comparing before and after on the same users as two groups. (right) Test each user's difference with the paired test.
  5. Choosing the level after seeing the result. (wrong) "It missed 0.05, so call it 0.10." (right) Fix the level before the data arrive.

Follow-up questions

What do you understand by p-value?

A p-value is the probability of data at least as extreme as the observed data if the null hypothesis and the model's assumptions were true. The ASA lists it as one measure of how incompatible the data are with a specified model, and warns against deciding on the threshold alone (ASA, March 2016).

If we fail to reject the null hypothesis, do we then accept the alternative hypothesis?

No, and we do not accept the null either. Failing to reject means the data did not provide enough evidence against the null at the chosen level; the sample may have been too small to see a real gap.

How would you explain a p-value to a doctor with no statistics training?

Say: "Suppose the new treatment did nothing. The p-value is how often a difference as large as ours would still appear by chance. A small number means chance is a poor explanation, but it does not tell you how much the treatment helps."

What is the difference between Type I and Type II errors?

A Type I error rejects a true null, a false alarm, and its rate is the significance level, 0.05 in the example. A Type II error keeps a false null, a missed effect, and its rate falls as the sample grows. A t test controls the first directly and the second through sample size. In cancer screening, with the null "no cancer", a missed cancer is a Type II error and a false alarm is a Type I error; which is worse depends on the costs, usually the missed cancer in screening, and the false alarm where a positive leads to risky treatment. Related reading: correlation versus causation, Bayes theorem questions and the classical machine learning guide.

When do you use a t test instead of a z test?

A z test assumes the population standard deviation is known, or that the sample is large enough for the normal cut-off to fit. A t test estimates the standard deviation from the sample, so it uses the heavier-tailed t distribution: the two-sided 5 percent cut-off is 2.228 at 10 degrees of freedom against 1.96 for the z test.1 The difference fades as the degrees of freedom grow, to 1.984 at 100.1

What is a confidence interval?

A 95 percent confidence interval is a range built by a method that captures the true gap in 95 percent of repeated experiments; it holds every gap that a two-sided test at 0.05 would not reject. In the example it is 447 to 1,513 milliseconds. With four times as many page loads per group it would be about half as wide, because the standard error falls with the square root of the group size.

Key takeaways
  • A t test divides a difference of means by its standard error and turns that ratio into a p-value.
  • Use Welch's version for two groups unless you know the variances match, meaning the two groups have the same spread.
  • A p-value is not the probability the null is true, and it does not measure the size of an effect.
  • Report the gap and its confidence interval; failing to reject is not accepting.
  • Match the test to the design: one sample, paired, or two independent groups.

Sources

  1. 1.3.5.3. Two-Sample t-Test for Equal Means, NIST/SEMATECH e-Handbook of Statistical Methods — National Institute of Standards and Technology (page shows no date; read 2026-10-24), 24 Oct 2026
  2. 1.3.5.2. Confidence Limits for the Mean and one-sample t-test, NIST/SEMATECH e-Handbook of Statistical Methods — National Institute of Standards and Technology (page shows no date; read 2026-10-24), 24 Oct 2026
  3. scipy.stats.ttest_ind (SciPy v1.18.0 manual) — SciPy project (page read 2026-10-24), 24 Oct 2026
  4. American Statistical Association releases statement on statistical significance and p-values — American Statistical Association, 7 Mar 2016
  5. The probable error of a mean, Biometrika 6(1):1-25 (doi:10.1093/biomet/6.1.1) — Biometrika (Student, i.e. W. S. Gosset), 1 Mar 1908
  6. Student's t-test — Wikipedia (date = read date; pointer to Gosset, Biometrika 1908), 24 Oct 2026