LLM Evaluation

How would you measure whether a new UI feature is improving engagement?

By Interview Clozo · Published 27 October 2026 · Last verified 26 October 2026 · 10 min read

Measure a new user interface (UI) feature with a randomized A/B test: split users at random, compare one goal-based metric, watch guardrails. That answers "how would you measure whether a new UI feature is improving engagement?" The follow-up that separates levels is how many users you need and how long to run, and the answer is not "until it looks significant".

In this article

Before you start
Why interviewers ask this
Measuring a UI feature's engagement from first principles
The Staff-level answer
The rubric
Three variants of this question
Common mistakes
Follow-up questions
Sources

Before you start

You can read this page with no statistics background; each term is explained where it first appears.

The same logic applies when you are evaluating large language model (LLM) features, and an A/B test is one kind of online evaluation.

Why interviewers ask this

The question separates candidates who can turn a vague goal into a decision from candidates who list metrics. "Engagement" means nothing until you say which user behavior, in which group, over which time. Interviewers use the question to see whether you ask what the feature is for, whether you know a comparison needs randomization, and whether you know a result can mislead.

A weak answer says "track clicks on the new feature and see if they go up." A strong answer names a goal-based primary metric, a control group, guardrails and a sample size. A Staff-level answer adds what goes wrong: peeking at results early, a broken traffic split, and a short-lived novelty bump. Reported in interviews at AI labs and large tech companies, the question comes with a product the interviewer names; the structure below holds for any product.

Measuring a UI feature's engagement from first principles

Primary metric: A primary metric is the single measure that decides whether a feature worked, chosen from the product goal before any data is viewed, so the experiment has one clear success test instead of many chances to find a lucky result.

In plain words

Imagine you add a new button to a shop and sales rise the next week. Maybe the button worked. Maybe it was payday. A fair test shows the button to a random half of shoppers and not to the other half, in the same week. Any difference then comes from the button.

The analogy breaks in one place: shoppers differ a lot from each other, so a small difference needs many shoppers before you can trust it. Precisely, the answer is a randomized experiment on one metric chosen from the goal, sized and timed in advance. Everything below follows from those few ideas: a fair comparison, one metric, a fixed size and a single reading.

A sound answer fixes the metric and the sample size before launch and reads the result once.CC BY 4.0 — source: How would you measure whether a new UI feature is improving engagement?

Step-by-step description

Choose the primary metric from the goal.
Randomize users and size the test.
Check the split, then read at the planned time.
Give the lift with an interval and guardrails.

The primary metric is chosen from the product goal.
Users are randomized and the sample size is fixed before launch.
The result is read once after the traffic split is checked.
The lift is reported with an interval and the guardrails are read.

Step 1: Choose the primary metric from the goal, not from the log

A primary metric is the one number that decides whether the feature worked, and you choose it from the product goal before you look at any data. The Google Research paper on user-centered metrics for web applications describes a process for mapping product goals to metrics. Rodden, Hutchinson and Fu (CHI 2010) introduce the Happiness, Engagement, Adoption, Retention and Task success (HEART) framework, which works from goals, to signals, to metrics. Illustrative examples for a UI feature: happiness = a survey rating; engagement = actions per active user per week; adoption = the share of active users who use the feature; retention = the 7-day return rate; task success = the share of started tasks that finish.

Ask what the feature is for. If it is a faster way to finish a task, the metric is tasks completed per active user, not clicks on the feature, since clicks can rise while task success falls. Write the metric as a proportion with a clear window and add two or three guardrails. Next you decide how many users the test needs.

Step 2: Randomize users and size the test before launch

Randomization assigns whole users, not page views, to control or treatment, so one person always sees one version. Sample size comes from four choices: the baseline rate, the smallest lift worth detecting, the false positive rate (usually 5%, as in Kohavi, Tang and Xu, 2020) and power (usually 80%, the chance of detecting a real lift of that size).

In an illustrative case, the baseline rate is 30% and the smallest lift worth shipping is 1.5 points, to 31.5%. A two-proportion power calculation then needs 14,853 per group, 29,706 in total.¹ If 12,000 eligible visitors arrive each week, that is three weeks, which also covers whole weekly cycles.¹ Fix these numbers before launch.

Step 3: Look once, at the planned time, after checking the split

Peeking breaks the false positive rate. In a simulation of an experiment with no real effect, one planned look at the end declares a winner 4.84% of the time, close to the intended 5%; five looks, stopping at the first significant one, do so 13.92% of the time.¹

Checking five times and stopping at the first significant result nearly triples the false alarm rate.CC BY 4.0 — source: How would you measure whether a new UI feature is improving engagement?

Text description of the diagram

One planned look at the end gives a false positive 4.84% of the time.
Stopping at the first significant of five looks gives a false positive 13.92% of the time.

Before reading any effect, check the split. Suppose the plan was 50/50 and the data show 50,000 in control and 51,200 in treatment, against an expected 50,600 in each. A chi-square check gives a statistic of 14.229 and a p-value of 0.0002, so the split is suspect and the result should not be trusted until the cause is found.¹ That failure is sample ratio mismatch. Because the check runs on every experiment, practitioners use a strict threshold such as p < 0.001 rather than 0.05, as in Fabijan et al. (Knowledge Discovery and Data Mining (KDD) 2019); the book by Kohavi, Tang and Xu also covers this check.

Step 4: Report the lift with an interval, and read the guardrails

The lift is the treatment rate minus the control rate, reported with a confidence interval, a range of effects consistent with the data. Say each group holds 15,000 people. Control has 4,500 successes (30.0%) and treatment 4,740 (31.6%): a lift of 1.6 points, a p-value of 0.0027 and an interval from 0.555 to 2.645 points.¹

Now read the guardrail. Complaints go from 2.0% to 2.2%, a difference of 0.2 points with an interval from -0.125 to 0.525 and a p-value of 0.2271.¹ That is not proof of safety: the interval still allows a harmful rise. The decision weighs a likely gain against a possible cost, with the goal in mind.

Knowing it is half the job. Saying it is the other half.

Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.