How do you design an experimentation platform?

In this article

1. Before you start
2. Why interviewers ask this
3. An experimentation platform from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one design topic in the machine learning (ML) system design interview path.

Why interviewers ask this

An experimentation platform is where product decisions, machine learning (ML) models and statistics meet, so the prompt tests breadth. Candidates report variants of it at AI labs and large tech companies: a general platform, one for large language model (LLM) features, one that balances short-term engagement against retention, and one that rolls back bad models automatically.

A weak answer describes a feature-flag service that splits traffic and stops there. A strong answer covers the full loop: consistent assignment, exposure logging, metrics, statistical checks, and a way to stop harm. It also says why that loop exists. Google's engineers describe experimentation as practically a mantra: "we evaluate almost every change that potentially affects what our users experience" (Tang et al., 2010). A platform that serves that many changes needs rules that keep the results believable.

An experimentation platform from first principles

Experimentation platform

An experimentation platform is a shared system that assigns users to experiment variants, records which variant each user saw, computes metrics for each variant, and reports whether a change caused a real difference, so many teams can test changes safely and consistently.

Sticky assignment

Sticky assignment is a rule that gives a user the same experiment variant on every visit, usually by computing the variant from the user id with a hash function instead of drawing a new random number each time.

In plain words

Picture a taste test at a shop. Every customer gets a numbered ticket. Even tickets get recipe A and odd ones get recipe B. A clerk writes down who got what and how much they bought. A manager checks the two piles are about the same size before comparing sales. If recipe B sales look bad, the clerk can pull it. The analogy breaks in one place: real platforms run hundreds of tests on the same customers, so tests must not disturb each other.

In one precise sentence, the platform maps a user and an experiment to a variant, logs exposure, estimates each effect, and checks validity first.

The same pipeline serves user interface changes and ML changes. The Google paper says its changes include "different machine learning algorithms that might affect ranking or content selection" (Tang et al., 2010), so a platform must accept a model version as a variant just as it accepts a button color.

Step 1: Hash the user id so each user keeps one variant

Assignment by hashing gives every user the same variant on every visit, because the variant is a pure function of the user id and an experiment salt. A salt is a string unique to the experiment, which makes every experiment shuffle users independently.

def bucket(user_id, salt, buckets=1000):
    return stable_hash(f"{salt}:{user_id}") % buckets

The function stable_hash must return the same value across servers and restarts. A bucket range then maps to a variant: in an illustrative 10/10/80 split, buckets 0 to 99 are control, 100 to 199 are treatment, and the remaining 800 are not in the experiment.¹

The common error is drawing a random number per request. A user who makes ten requests, each drawn 50/50, then sees both variants with probability about 99.8 percent,¹ so the groups are blurred and the measured effect shrinks toward zero. Assignment hashes the user id so each user keeps one variant.

The input is a user id and a salt; the output is one variant. You now know why assignment must be a function, not a coin flip. The unit you hash (user, device, session or account) should be the unit among which effects spread.

Step 2: Use layers so concurrent experiments do not collide

Layers let concurrent experiments share traffic without interfering. A layer is a slice of traffic in which each user is in at most one experiment, so experiments in different layers overlap while experiments in the same layer never do.

Google describes an overlapping experiment infrastructure built to run more experiments, produce better decisions and run them faster (Tang et al., 2010). The layering rule below is the general design idea. Put changes that cannot coexist, such as two versions of the same ranking model, in one layer so they are mutually exclusive. Put independent changes, such as a ranking change and a color change, in different layers, each hashed with its own salt, so both can run on the same users.

Input: a user and the list of layers. Output: at most one variant per layer. The platform records every assignment, so analysts can ask which experiments overlapped. Interacting changes placed in different layers can still affect each other, so the owner must decide where each experiment belongs.

Step 3: Log exposure and compute metrics in one shared place

Exposure logs record who actually saw each variant. An exposure event is written when the variant is applied, not when the user was assigned, because a user can be assigned and never reach the changed screen. Analyzing only exposed users avoids diluting the effect with people who never experienced it. Control must log the same exposure event at the same point, recording that the user would have seen the change (a counterfactual exposure), so both arms are filtered identically. If the treatment changes who reaches that point, analyze all assigned users instead (intent-to-treat), and expect the split check to catch an asymmetric trigger.

Metrics come from one shared pipeline with fixed definitions, so two teams cannot define "active user" differently. Each experiment names one primary metric, which is the metric the decision rests on, and a few guardrail metrics. A guardrail metric is a metric that must not get worse, such as error rate or page load time, even when the primary metric improves.

The Google paper makes a point that applies here: infrastructure alone is insufficient, so it also discusses "the associated tools and educational processes required to use it effectively" (Tang et al., 2010). Shared definitions, templates and review are part of the platform. Input: exposure and event logs. Output: one row per user, per metric, per variant.

Step 4: Check the split first, then read the metric

Analysis checks the split before it reads any metric. If a design promised an even split and the observed counts differ by more than chance allows, that is sample ratio mismatch, and the experiment's results are not trusted until the cause is found. Fabijan et al. (2019) diagnose its causes; typical ones are a crash that drops treatment users from the logs, or a filter applied after assignment.

Once the split passes, the platform estimates the difference per metric with a confidence interval. Sample size is fixed in advance from the smallest effect worth detecting, because looking at the p-value every day and stopping when it dips below the threshold inflates false positives, and sequential methods exist to make early looks valid (Johari, Pekelis and Walsh, 2015).

Variance reduction uses pre-experiment data. In the method called CUPED (controlled-experiment using pre-experiment data), each user's outcome is adjusted by the same user's earlier behavior, which leaves a variance of the original times one minus the squared correlation (Deng et al., 2013).¹ A smaller variance means fewer users per arm. Input: exposed-user metrics. Output: effect estimate, interval and a validity verdict.

Step 5: Add guardrails, automatic rollback and long-term holdouts

Guardrails stop a harmful change automatically. The platform watches every guardrail metric for every running experiment, compares treatment against control, and ramps traffic in stages so a bad change reaches few users before an alarm. A breach pauses the experiment or switches the variant off without a human in the loop; a person is paged afterward.

With thousands of concurrent experiments, run one monitor per guardrail metric over all experiments rather than one custom monitor per experiment, and use anomaly detection with sensible thresholds. With that many metric-experiment pairs, some alarms are false by chance, so control the false discovery rate or set a stricter alert threshold. The anomaly-detection design page covers that machinery.

Short experiments measure short-term engagement, which is not the same as retention. A holdout is a small group of users kept on the old experience for a long period, so a later comparison shows lasting effects that a short test cannot. For retention-style outcomes, see how churn and customer value are modeled. Input: live metric streams. Output: automatic pauses and a long-term comparison.

1 · Assign2 · Layer3 · Log exposure4 · Analyze5 · GuardReplay

Step-by-step description

1. Hash the user id so each user keeps one variant.
2. Concurrent experiments share traffic without interfering.
3. Record who actually saw each variant.
4. Check the split before reading any metric.
5. Stop a harmful change automatically.

• Assignment hashes the user id so each user keeps one variant.
• Layers let concurrent experiments share traffic without interfering.
• Exposure logs record who actually saw each variant.
• Analysis checks the split before it reads any metric.
• Guardrails stop a harmful change automatically.

Worked example

Suppose a product has a baseline conversion rate of 10 percent (illustrative) and the team wants to detect a change of 0.5, 1 or 2 percentage points. For a yes/no outcome the variance of one user is p times one minus p, which is 0.09.¹ A common rule of thumb for 80 percent power and the usual two-sided false-positive rate of 0.05 is n = 16 × variance ÷ effect², where n is users per arm (Kohavi et al., 2009).¹ The 16 comes from twice the squared sum of two standard-normal cut-offs, about 15.7, rounded.

That gives 57,600 per arm for a 0.5-point change, 14,400 for one point and 3,600 for two points.¹ Halving the effect to detect quadruples the traffic. An A/B/n test with three arms needs three times 57,600, which is 172,800 in total for the smallest effect, assuming the same per-arm n and before any correction for several comparisons.¹

Now apply variance reduction with a correlation of 0.7 (illustrative) between a user's earlier and later behavior. The variance left is 1 − 0.49, which is 0.51, so the sample size falls to 51 percent of the unadjusted figure: 7,344 per arm at the one-point effect.¹

For the split check, suppose a 50/50 design over 100,000 visitors shows 50,700 control and 49,300 treatment (illustrative). Each arm is expected to hold 50,000, so the chi-square statistic is 700² ÷ 50,000 twice, which is 19.6.¹ The usual cut-off at the 0.05 level with one degree of freedom is 3.841, and the p-value is below 0.001, so this is sample ratio mismatch.¹

Text description of the diagram

• Sample size per arm rises as the effect to detect shrinks.
• Adjustment with a correlation of 0.7 cuts the sample size to 51 percent of the unadjusted figure.

Sample size per arm rises as the effect to detect shrinks. Adjustment with a correlation of 0.7 (illustrative) cuts the sample size to 51 percent of the unadjusted figure.

Trade-offs and edge cases

A plain feature flag service decides who sees a change. An experimentation platform also logs exposure, checks the split and judges the result. Treating the two as one is the commonest scoping mistake in this interview, and the Google paper's point that tools and process are needed beyond infrastructure applies directly.

Text description of the diagram

• A feature flag service decides who sees a change.
• An experimentation platform also logs exposure, checks the split and judges the result.

Edge cases to name:

• Interference. If users share a limited resource, treatment can change what control sees. In a marketplace, a cheaper price for treatment buyers can drain supply that control buyers wanted, which biases a user-level split; see dynamic pricing. Randomize by region or by time block instead, and accept fewer independent units.
• LLM features. A prompt, model or retrieval change is a variant, so log its full configuration with each exposure. Outputs vary run to run, so judge by many samples. A model that grades outputs is a metric to calibrate against human labels and against online behavior, never to trust alone.
• Multi-stage systems. A change to retrieval alters what the ranker sees, so test whole funnels, not one stage; the retrieval, ranking and re-ranking funnel shows why.
• Speed against rigor. Central checks slow teams a little but stop wrong launches. A novelty effect can also fade, so long tests or holdouts matter for habits.
• Abuse. Bots and fraud distort metrics, so exclude them before analysis, as in real-time fraud detection.
• Unit mismatch. When users are randomized but a metric is per page view or per session, the observations of one user are correlated, so a naive test understates the noise. Use the delta method, a standard way to estimate the variance of a ratio metric, or analyze at the user level.

How to answer it in an interview

The spoken structure: say assignment first, then logging, then analysis with the split check, then guardrails, and finish with the trade-off.

Common mistakes

• "I assign a random variant on each request." (wrong) A user then sees both variants. (right) "I hash the user id with an experiment salt so the variant is sticky."
• "The platform is a feature-flag service." (wrong) That covers assignment only. (right) "Assignment is one stage; logging, analysis checks and guardrails are the rest."
• "I check the p-value daily and stop when it is significant." (wrong) Repeated looks inflate false positives. (right) "I fix the sample size in advance or use a sequential method."
• "A large effect on the primary metric means ship." (wrong) A guardrail may have broken. (right) "I ship only if guardrails hold and the split check passed."
• "Two weeks of engagement shows retention." (wrong) Short-term and long-term effects differ. (right) "I keep a holdout and compare retention later."

Follow-up questions

How would you design an A/B testing platform for LLM features with trustworthy metrics?

Treat prompt, model and retrieval version as the variant and log them with every exposure. Combine online behavior with a calibrated grader checked against human labels, and keep guardrails for safety and cost.

How do you measure short-term engagement and long-term retention in an A/B/n test?

Run the test for the planned sample size and read engagement from it. Keep a holdout on the old experience for a long period, and compare retention there. Do not decide retention from the short test.

How do you detect regressions and roll back models across thousands of concurrent experiments?

Monitor every guardrail metric across all experiments with shared detection jobs, ramp traffic in stages, and switch a variant off automatically when treatment breaches its threshold against control.

How do you scale evaluation for real-world and A/B test scenarios?

Share the assignment, logging and metric pipelines across teams, precompute common metrics, and reserve full tests for changes that passed cheaper checks on held-out data first.

What does experimentation platform architecture look like at a glance?

Five stages in a line: assignment, layers, exposure and metric logs, analysis with validity checks, and guardrails with rollback. Each stage guards against a different way to get a wrong answer.

Sources

1. Overlapping Experiment Infrastructure: More, Better, Faster Experimentation — Google Research (Tang, Agarwal, O'Brien, Meyer; KDD 2010), 25 Jul 2010
2. Diagnosing Sample Ratio Mismatch in Online Controlled Experiments — ACM KDD 2019 (Fabijan et al.), 4 Aug 2019
3. Always Valid Inference: Bringing Sequential Analysis to A/B Testing (arXiv 1512.04922) — arXiv (Johari, Pekelis, Walsh), 15 Dec 2015
4. Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data — Microsoft Research (Deng, Xu, Kohavi, Walker; WSDM 2013), 4 Feb 2013
5. Controlled experiments on the web: survey and practical guide — Data Mining and Knowledge Discovery (Kohavi et al.), 1 Feb 2009