What does your confidence level mean when building a confidence interval?

In this article

1. Before you start
2. Why interviewers ask this
3. Confidence level from first principles
4. The Staff-level answer
5. The rubric
6. Three variants of this question
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page belongs to the statistics interview questions learning path.

Why interviewers ask this

The question, part of the wider set of statistics interview questions, separates candidates who can compute an interval from candidates who can say what it means. It is reported by candidates in data and machine learning interviews as a quick check on statistical judgment, because the wrong reading leads to wrong decisions in experiments and model evaluation.

A weak answer says "I am 95% sure the true value is in my interval", which sounds right and is not what the number states. A strong answer says the 95% belongs to the procedure: repeat the whole sampling-and-building process many times and about 95% of the intervals catch the true value. A Staff-level answer adds the trade-off, that raising the level widens the interval, and names what actually narrows it: more data or less noise per observation.

Confidence level from first principles

Confidence level

The confidence level is the long-run proportion of confidence intervals, built by one fixed method on repeated random samples from the same population, that contain the true parameter. It is chosen before the data are collected, and it describes the method's reliability rather than any single interval.

Confidence interval

A confidence interval is a range of values, computed from one sample as an estimate plus or minus a margin, that is meant to bracket an unknown population quantity such as a mean. Its width shows how precise the estimate is, and its level states how often the method succeeds.

In plain words

Think of a ring-toss game. The stake is fixed and you cannot see it. You throw a ring of a set size. The rule is built so that 95 throws in 100 (illustrative) land around the stake. Before you throw, your chance of success is 95%. After the ring lands, it either circles the stake or it does not. The analogy breaks in one place: in a real study you never see the stake, so you cannot check any single throw.

In one precise sentence, the confidence level is a property of the interval-building method over repeated samples, not a probability attached to one computed range.

Step 1: Build the interval from an estimate and a margin

A confidence interval for a mean is the sample mean plus or minus a margin, and the margin is a critical value times the standard error. The standard error is the standard deviation of the sample mean, estimated as the sample standard deviation divided by the square root of the sample size. A critical value is the multiplier, read from a table or code, that sets how many standard errors the margin spans for the chosen level. NIST gives the known-spread form with a normal critical value.

Take an illustrative sample of 25 values with mean 50 and standard deviation 10. The standard error is 2, the 95% critical value from the t distribution with 24 degrees of freedom is 2.064, and the interval is 45.87 to 54.13.¹ You now have an interval; the next step is what its 95% means.

Step 2: What the 95% level promises and what it does not

The 95% level promises something about the method over many repeats, and nothing about the one interval in your hand. In a simulation of 10,000 samples of 25 values from a population whose true mean is 50 (illustrative), 94.89% of the 95% intervals contained 50.¹ NIST describes the same idea: with repeated sampling the intervals bracket the true parameter in about 95% of cases.

Once your sample is drawn, the interval 45.87 to 54.13 is a fixed pair of numbers. The true mean is also fixed. Either it lies inside or it does not, so the 95% cannot be a probability that this range holds the truth. The 95% was a statement about how reliable the recipe was before the data arrived. The paper by Greenland et al. (2016) exists because such misreadings of confidence intervals remain widespread.

Step 3: A higher level buys certainty with a wider interval

A higher confidence level gives a wider interval from the same data, because the method must reach further to succeed more often. For the same illustrative sample, the 90% interval is 46.58 to 53.42, 6.84 wide; the 95% interval is 8.26 wide; the 99% interval is 44.41 to 55.59, 11.19 wide.¹ In the simulation the 90%, 95% and 99% methods contained the true mean in 90.34%, 94.89% and 98.84% of samples.¹

The 99% interval is 11.19 wide, against 8.26 at 95% (illustrative sample).¹ So a higher level is not simply better: it is less precise. The levers that narrow an interval without losing confidence are a larger sample and lower variance per observation, because both shrink the standard error. The variants below show the sample-size effect.

1 · Build one interval2 · Repeat the method3 · Raise the levelReplay

Step-by-step description

1. Estimate plus or minus a margin.
2. About 95% of repeats cover the truth.
3. A higher level widens the interval.

• The 95% interval from one illustrative sample is 45.87 to 54.13.
• In 10,000 repeated samples, 94.89% of the 95% intervals contained the true mean.
• The 99% interval is 11.19 wide, against 8.26 at 95%.

Text description of the diagram

• The interval is 6.84 wide at the 90% level.
• The interval is 8.26 wide at 95% and 11.19 wide at 99%.

The Staff-level answer

What makes it Staff-level: it separates the procedure from the result, states the cost of a higher level, and names the real levers.

The rubric

Three variants of this question

Does a 95% confidence interval contain 95% of the data?

No. The interval is about the population mean, not about individual values. With a standard deviation of 10 and a standard error of 2 in the illustrative sample, the interval is much narrower than the spread of single observations.¹ For a bound on how many single values sit within a given distance of the mean, see Chebyshev's inequality.

Can you say the true mean is in your interval with 95% confidence?

You can say it only as shorthand for the method: the procedure that built your interval captures the true mean in about 95% of repeats (NIST). It is not a probability for the one interval you hold, which either contains the true mean or does not.

What changes if the sample is four times larger?

The standard error halves, so the interval roughly halves in width at the same level. In the illustrative case, 100 values instead of 25 turn the 95% interval of 8.26 wide into 48.02 to 51.98, 3.97 wide.¹ The level itself, the 95%, does not change.

Common mistakes

• "There is a 95% probability the true mean is in this interval." (wrong) The true mean is fixed and the interval is already drawn. (right) "The method captures the true mean in about 95% of repeated samples."
• "95% of the data lie inside the interval." (wrong) The interval describes the mean, not single values. (right) "The interval estimates the population mean."
• "I chose 99% because higher confidence is always better." (wrong) It widens the interval. (right) "I traded precision for a lower miss rate, and I can regain precision with more data or less noise per observation."
• "I used the normal value because it is the standard one." (wrong) With an estimated spread and a small sample, the interval is too narrow. (right) "I used the t value with n minus 1 degrees of freedom." In the simulation, a 95% interval using the normal value 1.96 with an estimated standard deviation contained the true mean in only 93.81% of samples at a size of 25.¹

Follow-up questions

How do you build a 95% confidence interval for a mean?

Take the sample mean, divide the sample standard deviation by the square root of the sample size to get the standard error, and add and subtract the critical value times that standard error. NIST writes the known-spread version as the mean plus or minus a normal critical value times the standard deviation over the square root of the sample size.

When do you use the t distribution instead of the normal value?

Use the t distribution when the spread is estimated from the sample, which is the usual case. The t critical value is larger than the normal one, so at the same level the interval is wider when the standard deviation is estimated from limited data (NIST); see the t test page.

How is a confidence interval linked to a hypothesis test?

NIST describes a confidence stated at the one-minus-alpha level as the inverse of a significance level alpha. In practice, a value outside a 95% interval for a mean is a value that a two-sided test at the 5% significance level would reject.

What does a confidence interval tell you in an A/B test?

It gives the range of lifts consistent with the data under the method's reliability, so a wide interval says the experiment cannot separate small effects. See A/B testing interview questions.

Is there an interval that does carry a probability statement?

A Bayesian credible interval does, because it treats the parameter as uncertain given a prior; the usual confidence interval does not. See Bayes theorem interview questions and the wider probability interview questions.

Sources

1. NIST/SEMATECH e-Handbook of Statistical Methods, 7.1.4 What are confidence intervals? (accessed 2026-10-26) — National Institute of Standards and Technology, 26 Oct 2026
2. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations — European Journal of Epidemiology (Greenland et al.), 21 May 2016