Probability & Statistics

How would you design an A/B test for a new ranking algorithm?

By Interview Clozo · Published 27 October 2026 · Last verified 26 October 2026 · 12 min read

How would you design an A/B test for a new ranking algorithm? You randomly assign users to a control group that sees the current ranker and a treatment group that sees the new one, then compare one primary metric chosen in advance. The follow-up that separates levels is what you check before reading that metric: whether the split itself is what you designed.

In this article

Before you start
Why interviewers ask this
A/B test for a ranking algorithm from first principles
The Staff-level answer
The rubric
Three variants of this question
Common mistakes
Follow-up questions
Sources

Before you start

This page belongs to the statistics interview guide. The A/B testing interview questions page covers the wider family of experiment prompts.

Why interviewers ask this

A badly designed ranking experiment ships a worse ranker or kills a better one, and the dashboard looks confident either way. Interviewers use the question to see whether a candidate can turn a vague goal, "is the new ranker better?", into a plan that survives scrutiny.

A weak answer says "split users 50/50 and compare click-through rate." A strong answer names the unit of randomization, the primary metric and the guardrails, the sample size and the stopping rule. A Staff-level answer adds what to check before reading any result and what to do when many candidate rankers are waiting for a test slot. The rubric below shows the difference line by line.

A/B test for a ranking algorithm from first principles

Sample ratio mismatch (SRM): Sample ratio mismatch is a gap between the group sizes an experiment was designed to produce and the sizes it actually produced, large enough that chance does not explain it. It signals a bug in assignment or logging, so the results cannot be trusted.
Interleaving: Interleaving is a test design that shows each user one blended list built from two rankers and credits every click to the ranker that contributed the clicked item. It measures preference inside each user instead of comparing two groups.

In plain words

Imagine a café that wants to know whether a new menu order sells more coffee. It shows the old menu to a random half of customers and the new one to the other half. Then it compares sales. That is an A/B test: a randomized experiment that gives two groups different versions and compares one outcome.

The analogy breaks in one place. Customers pick once. Search users come back every day and learn, so who sees what, and for how long, matters far more.

In one precise sentence, a ranking A/B test estimates the causal effect of the new ranker on a chosen metric by comparing randomly assigned groups. Randomization is what lets you read the difference as cause, not just correlation.

Step 1: Randomize by user, then check the split before reading results

Randomize by user so each person sees one ranker. Assign each account once, using a stable hash of its identifier, and keep the assignment for the whole test. Randomizing per request lets the same person see both rankers, so the effect of one leaks into the other, and heavy users count unevenly.

Before looking at any metric, run a split check. Suppose an experiment is designed to split 100,000 accounts evenly and the result is 50,500 and 49,500. A chi-squared test on those counts gives a statistic of 10 and a p-value of about 0.0016 (illustrative counts).¹ That is too unlikely to be chance, so you hunt for the bug first: a crash in one arm, bot traffic, lost logs.

You now know the experiment is sound or not. Only then do the metrics mean anything.

Step 2: Fix one primary metric and the guardrails before launch

The overall evaluation criterion (OEC) is the single metric, or one fixed combination of metrics, that decides whether the new ranker ships. Choose it before the test starts so no one picks the flattering number afterward.

Clicks alone mislead. A ranker can raise click-through rate by making results worse, because users click more to find what they wanted. A sturdier primary metric counts successful sessions per user, for example sessions that end in a long watch or a completed purchase.

Then list guardrail metrics: latency, complaint rate, revenue, and the diversity of what is shown. Write down the decision rule too: ship if the primary metric rises by at least the minimum worthwhile amount and no guardrail drops beyond its limit. A written rule stops post-hoc storytelling.

Step 3: Size the test and run whole weeks

Statistical power is the chance that a test detects a real effect of a chosen size, and the minimum detectable effect is the smallest change worth detecting. Together with the baseline rate and the significance level, they set how many users each arm needs.

Size the test on a per-user yes/no metric, so each account is one trial. Take a baseline where 10% of users have a successful session, a 5% significance level and 80% power (all illustrative). Detecting a relative lift of 10% needs about 14,700 accounts per arm. A 5% lift needs about 57,800, and a 2% lift about 356,300.¹ Halving the lift needs about four times the users, so the smallest lift worth shipping sets the cost.

Halving the relative lift roughly quadruples the users each arm needs, from about 14,700 to about 57,800.
Relative lift to detect	Users with a successful session (baseline 10%)	Users per arm (about)
10%	11.0%	14,700
5%	10.5%	57,800
2%	10.2%	356,300

If the metric is a ratio such as clicks per impression while you randomize by user, the variance must be computed per user (the delta method or a per-user metric), or the test looks more precise than it is.

Divide the required users by daily eligible users, then round the duration up to whole weeks, because behavior differs between weekdays and weekends. The sizing rests on the normal approximation from the Gaussian distribution.

A smaller lift needs far more users per arm, so the smallest worthwhile lift sets the test length.CC BY 4.0 — source: How would you design an A/B test for a new ranking algorithm?

Text description of the diagram

At a 10% relative lift the test needs about 14,700 accounts per arm.
At a 2% relative lift the test needs about 356,300 accounts per arm.

Step 4: Read the result once, at the planned end

Read the result once, at the planned end, because every extra look raises the false-positive rate. If you test five times during a run and stop at the first "significant" result, a change with no effect at all is declared a winner in about 14% of runs, not 5%.¹

Fix the sample size and the end date in advance and report at that point. If the business needs early looks, use a sequential method built for repeated checks and decide the rule before launch.

At the end, report the estimated effect with its confidence interval, not only a pass or fail. Check every guardrail. A t test or a similar test supplies the interval for a mean metric; for per-user metrics where a few heavy users dominate the mean (see mean vs median), cap extreme values or use a per-user variance-reduction method such as controlled-experiment using pre-experiment data (CUPED). Then decide by the rule you wrote in Step 2.

Step 5: Use interleaving first when many rankers wait

Interleaving can prune many candidate rankers faster than a standard A/B test. It is an optional stage that runs before the A/B test, not after it. A Netflix Technology Blog post (November 2017) describes a two-stage process. The first stage is a fast pruning step with interleaving. The second is a traditional A/B test on the few survivors, which measures longer-term behavior. The same post says the first stage finishes in a matter of days. The method follows Chapelle et al..

In team-draft interleaving, the rankers draft in rounds. In each round a coin toss decides which ranker picks first, and each ranker contributes its highest-ranked item not yet in the list. Illustrative example: ranker A orders items a, b, c, d and ranker B orders b, e, a, f. If A wins both tosses, the blended list is a, b, c, e, and the contributors are A, B, A, B. If the user clicks b, c and e, B gets two credits and A gets one.¹

The coin toss matters because lower positions get fewer clicks, so each position should be equally likely to come from either ranker. Interleaving measures preference, not retention, so the survivors still need an A/B test.

Five decisions, in order, turn a ranking change into a result you can trust.CC BY 4.0 — source: How would you design an A/B test for a new ranking algorithm?

Step-by-step description

Randomize by user, then check the split.
Primary metric and guardrails come first.
The smallest worthwhile lift sets the size.
Read at the planned end only.
Many rankers? Interleave before step 1.

Randomize by user so each person sees one ranker.
Check the split before reading any result.
Read the result once at the planned end.

Knowing it is half the job. Saying it is the other half.

Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.