In this article
Before you start
This page is one step of the machine learning (ML) system design interview path.
Why interviewers ask this
The consequence of a vague metric is that nobody can tell whether the system worked. Candidates report this follow-up arriving right after they propose a success metric in a machine learning design round, and it separates people who name a number from people who could build it.
A weak answer repeats the name: "activation rate, the share of users who activate." A strong answer writes the formula: who is counted, what event counts, over what window, and who is left out. A Staff-level answer goes one step further and says how the number could be gamed and which second number would catch that. The interviewer is checking whether your metric could be handed to an analyst and computed without a meeting.
Metric definition from first principles
- Metric definition
- A metric definition is a written rule that says exactly how one number is computed: the entity counted, the numerator, the denominator, the time window, what is included or excluded, and which timestamp is used, so that any two people reproduce the same value from the same data.
In plain words
Think of a recipe. "Make it tasty" is a name. A recipe lists ingredients, amounts and cooking time. Two cooks following it get the same dish. A metric definition is the recipe for a number. The cooking time in the recipe maps to the time window of the metric, and the list of ingredients maps to who and what is counted.
The analogy breaks in one place. A cook has no stake in the dish being scored. People who read a metric are often paid on it, which is why Step 4 exists.
In one precise sentence: a metric is any number a system reports, and a metric definition is the rule that makes that number reproducible.
Step 1: Name the entity and the counted event
Google's Rules of Machine Learning defines a metric as "a number that you care about" and an objective as "a metric that your algorithm is trying to optimize". An objective is a metric that a learning system tries to optimize, so every objective is a metric but most metrics are not objectives.
Start with the entity: the thing one row of data describes, such as a user, a workspace, a session or a request. Then name the event that is counted. The same rules advise a first objective that is simple, observable and attributable, such as whether a ranked link was clicked.
Input: the goal in plain words. Output: one entity and one event. A definition that counts events but never names whose events they are is not finished. The event is a proxy metric for the goal, so say what it stands in for.
Step 2: Fix the denominator and the window
The denominator is the eligible population, and the window is how long each entity has to produce the event. Both change the number. A rate names both a numerator and a denominator, and a time frame belongs in the metric (Analytics Made Simple).
Step-by-step description
- Say whose events are counted.
- Fix who is eligible and for how long.
- Remove bots; pick the clock.
- Pair the metric with a guardrail.
- A metric definition names an entity and the event that is counted.
- The denominator and the time window change the number.
- Exclusions and the event timestamp complete the rule.
- A guardrail metric guards against gaming.
Take an illustrative cohort of 1,000 signups. 400 activate on day 1, 200 on day 3, 100 on day 12, and 300 never do, of which 60 are bots.1 The same cohort gives 0.6, 0.6383, 0.4255 or 0.7447 depending on the definition.1 A seven-day window with bots removed gives 0.6383.1
| Definition | Numerator | Denominator | Rate |
|---|---|---|---|
| All signups, 7 days | 600 | 1,000 | 0.6 |
| Bots removed, 7 days | 600 | 940 | 0.6383 |
| Bots removed, 1 day | 400 | 940 | 0.4255 |
| Bots removed, no time limit | 700 | 940 | 0.7447 |
Text description of the diagram
- The same cohort gives 0.6, 0.6383, 0.4255 or 0.7447 depending on the definition.
- A seven-day window with bots removed gives 0.6383.
A second trap is how a rate is averaged. Optimizely's ratio metrics documentation sums the numerator across all qualifying units, sums the denominator across the same units, and divides the totals (a pooled ratio). With three illustrative accounts that have 9 of 10 seats active, 1 of 2 and 0 of 1, the pooled ratio is 0.7692, while the mean of the three per-account ratios is 0.4667.1 The two answer different questions, so say which one you mean.
Step 3: Write the inclusion rules and the event timestamp
The inclusion and exclusion rules say which rows are removed before counting: bots, internal test accounts, refunds, duplicate events. The timestamp rule says which clock assigns an event to a window: when the event happened, or when the system received it.
A practitioner spec template covers the grain, the formula with its numerator and denominator, include and exclude filters, identity rules, the event timestamp used, the time zone and a late-arriving data policy (Analytics Made Simple). It is practitioner consensus, not a standard. In our own voice, add the entity and the gaming risk to that checklist.
Exclusions and the event timestamp complete the rule. The late-data policy matters because a number read on Monday can change by Friday; say whether you freeze it or recompute it. Input: the rate from Step 2. Output: a rule that does not depend on who runs it.
Step 4: Name the gaming risk and add a guardrail
Goodhart's law is the observation that a measure that becomes a target ceases to be a good measure (Thomas and Uminsky, 2020). The mechanism is simple: when a measure rewards performance, people have an incentive to manipulate it in order to receive the reward (Goodhart's law analysis, 2022).
Kohavi, Tang and Xu note that revenue alone is an insufficient Overall Evaluation Criterion (OEC), the quantitative measure of an experiment's objective, because it could lead to plastering the site with ads, so Bing weighs revenue against user-experience metrics such as sessions per user (chapter 1 excerpt). A guardrail does the same job from outside the main metric, so a guardrail metric guards against gaming.
An illustrative ranking test shows it: variant Y lifts click-through rate (CTR), clicks divided by impressions, from 0.12 to 0.15, but next-day return falls from 0.3 to 0.26.1 If you stop at the optimized number, you ship the worse variant.
A metric definition names an entity and the event that is counted. The denominator and the time window change the number. Exclusions and the event timestamp complete the rule. A guardrail metric guards against gaming.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →The Staff-level answer
And how would you define that metric itself?
Take first-week activation rate. The entity is a workspace. The numerator is workspaces that finish the core workflow within seven days of creation. The denominator is all eligible workspaces created in the cohort, with bots and internal test accounts removed. I assign events by when they happened, in one fixed time zone, and I recompute once late data lands. Two analysts get the same number from that. Then I name the gaming risk: a team paid on this number can shrink the workflow, so I pair it with thirty-day retention as a guardrail and write down the baseline and the target.
What makes it Staff-level: it closes with the gaming risk and a named guardrail, not just a formula.
The rubric
| Dimension | Weak answer | Strong answer | Staff answer |
|---|---|---|---|
| Correctness | Names the metric only | Gives numerator and denominator | Adds window, exclusions and timestamp rule |
| Depth | "Engagement" or "accuracy" | Defines the eligible population | Shows how each choice changes the number |
| Trade-offs | None | Notes one alternative definition | Chooses a definition and says what it hides |
| Communication | Vague wording | One clear formula | Sayable in under a minute, with an example |
| Follow-up readiness | Surprised by "why?" | Can justify the denominator | Names the gaming risk and the guardrail |
Three variants of this question
How would you define accuracy precisely?
Accuracy is the number of correct predictions divided by the total number of predictions (Google glossary). A complete answer adds on which data set it is computed and warns that it hides class imbalance.
How would you define the metric for a fraud model?
Name the entity (a transaction or an account), the counted event (a confirmed fraud label), the label delay and the window, and keep a business guardrail such as blocked good customers. The real-time fraud detection design shows the surrounding system.
Which metric would you optimize and which would you only monitor?
Optimize one simple, observable metric and monitor the rest as guardrails. Google's rules note that indirect effects such as next-day visits suit A/B tests and launch decisions better than a first objective.
Common mistakes
- "Engagement went up, so it worked." (wrong) The name has no formula. (right) "Engagement is sessions per active user per week, bots removed."
- "The rate is conversions over visits." (wrong) The population and window are missing. (right) "Conversions within seven days of first visit, over unique eligible visitors."
- "I'll average the ratios across accounts." (wrong) Averaging ratios without saying so mixes two questions: small accounts count as much as large ones. (right) "I report the pooled rate for the business view and the per-account mean for the typical customer, and I say which one is the headline."
- "Pick the best metric and optimize it forever." (wrong) The proxy drifts from the goal once it is a target. (right) "I optimize one metric, watch guardrails and revisit the choice after launch."
Follow-up questions
What is the difference between a metric and an objective?
A metric is any number the system reports; an objective is the metric the algorithm optimizes (Google).
How do you stop a team from gaming the metric?
You cannot remove the incentive, so you add a guardrail that the gaming would damage, and you review the definition when the number moves without a product change.
Why does the choice of denominator matter?
It decides who is counted as eligible. In the toy cohort, removing 60 bots moves the seven-day rate from 0.6 to 0.6383.1
Should the model optimize the same metric the business reports?
Often not directly. The model needs a simple, observable target, as in the retrieval, ranking and re-ranking funnel, while the business reports slower outcomes such as retention.
How does the definition change for a delayed outcome?
State the label delay and the maturity window, as the churn, long-term value and credit-risk design does. A rate read before outcomes mature is biased low. The delay is a moderation appeal outcome in harmful-content detection and the actual arrival time in arrival-time prediction.
- A metric definition names the entity, numerator, denominator, window, exclusions and timestamp.
- The same data gives different numbers under different denominators and windows.
- Pooled ratios and averaged ratios differ; say which one you report.
- Goodhart's law means a target metric needs a guardrail metric beside it.
Sources
- Rules of Machine Learning: Best Practices for ML Engineering — Google for Developers (Zinkevich), 26 Oct 2026
- Machine Learning Glossary: Metrics — Google for Developers, 26 Oct 2026
- Trustworthy Online Controlled Experiments, Chapter 1 excerpt — Kohavi, Tang and Xu (book site), 1 Jan 2020
- The Problem with Metrics is a Fundamental Problem for AI (arXiv 2002.08512) — arXiv (Thomas and Uminsky), 20 Feb 2020
- Goodhart's Law — CNA, 1 Sep 2022
- Definitions and metric specs — Analytics Made Simple (practitioner page), 26 Oct 2026
- Ratio metrics — Optimizely documentation, 26 Oct 2026
