Product & Analytics Cases

How would you improve data quality on a product like Siri?

By Interview Clozo · Published 27 October 2026 · Last verified 26 October 2026 · 13 min read

How would you improve data quality on a product like Siri? Name the quality dimensions first, then score each one against human-labelled ground truth from a small reviewed sample, and fix the weakest dimension at its source. The mistake that costs the point is tuning the wake word and the transcription metric as if they were independent, and the second step shows why a trigger change moves the transcription numbers with no model change.

In this article

Before you start
Why interviewers ask this
Data quality on a voice assistant from first principles
The Staff-level answer
The rubric
Three variants of this question
Common mistakes
Follow-up questions
Sources

Before you start

This page is one question in the product and analytics case interviews path, alongside data science interview questions. It needs three small ideas, each explained in a sentence below.

Why interviewers ask this

The question separates candidates who treat "data quality" as a slogan from candidates who can turn it into measurements. A product-sense round for engineers can ask it about any machine-learning product, and questions of this kind are reported by candidates in interviews at large tech companies.

A weak answer says "collect more data and clean it". That answer names no failure, no metric and no way to tell whether quality improved. A strong answer splits quality into parts, says who labels each part and how many cases they label, and names a trade-off. A Staff-level answer adds the effect most people miss: the data you collect depends on the earlier stages of the product, so changing one stage silently changes the data that every later stage is measured on. Siri is a good example because its pipeline has clear stages, and Apple has published how it reviews them.

Data quality on a voice assistant from first principles

Data quality dimension: A data quality dimension is a named property of data, such as accuracy, completeness, consistency, validity, timeliness or uniqueness, that lets a team measure how fit the data is for one purpose. It turns "quality" from a feeling into several separate numbers, each of which can be tracked and fixed on its own.
False accept: A false accept is a wake-word error in which the assistant wakes although the user did not mean to call it, for example on another person's voice or on a similar phrase. Audio and a transcript of a non-request then enter the data, and the product may act on them.
False reject: A false reject is a wake-word error in which the assistant stays asleep although the user did say the wake phrase. The request never reaches the speech recogniser, so it leaves no transcript, no log line and nothing for a reviewer to examine.
Ground-truth label: A ground-truth label is a correct answer written by a trained human reviewer for one example, such as the exact words spoken or the intent behind a request. A metric compares the system's output against it, so the metric is only as good as these labels.

A quality fix starts with separate labels, respects the wake trigger's effect on the logs, measures on a sample and ends with a re-measure.CC BY 4.0 — source: How would you improve data quality on a product like Siri?

Step-by-step description

Wake, words and task need separate labels.
A trigger change alters what reaches the logs.
A small reviewed sample gives the error rate.
Fix the weakest dimension, then measure again.

Wake-word errors, transcription errors and task failures need three different labels.
A change to the wake trigger changes which requests reach the transcription logs.
A small human-reviewed sample gives the ground truth for the error rate.
Fix the weakest dimension at its source, then re-measure.

In plain words

Think of a kitchen that serves thousands of dishes a day. The head chef cannot taste every dish. She tastes a small random sample and scores each one on separate things: temperature, seasoning, look. Then she fixes the worst of those. Data quality on Siri works the same way. The analogy breaks in one place: a kitchen sees every dish it makes, but Siri never sees the requests it failed to wake for. So the sample you review is always a sample of what got through, and you must say so when you report a number.

In one precise sentence, you improve data quality by defining separate measurable dimensions, estimating each from human-labelled samples, and acting on the weakest one.

Step 1: Split quality into three kinds of label

Quality on Siri is at least three problems with three kinds of label. Apple says its grading reviewed a small sample of audio from Siri requests, less than 0.2 percent, with computer-generated transcripts, to measure how well Siri was responding (Apple Newsroom, August 2019). Contractors told The Guardian (26 July 2019) that they graded whether the activation was deliberate or accidental, whether Siri could help and whether its response was appropriate.

Those reported criteria suggest three labels. Wake intent on logged requests finds false accepts. False rejects never reach the logs, so they need a separate labelled test set (Step 2). Hearing accuracy gives a transcript to compare against, which yields a word error rate. Appropriate response gives a task-success label.

Standards such as the International Organization for Standardization (ISO) and International Electrotechnical Commission (IEC) standard 25012 define data quality through characteristics like accuracy and completeness. Dagster's 2026 explainer lists accuracy, completeness, consistency, validity, timeliness and uniqueness. Accuracy is how correctly data represents the real event. Completeness is whether all required data is present, so false rejects are a completeness failure.

Step 2: Fixing the wake trigger changes the data you collect

The wake trigger decides which audio ever becomes data, so changing it changes every later metric. A very small speech recogniser runs all the time and listens for the wake phrase, and only then does the rest of Siri parse the speech (Apple, 2017). Unintended activations come from similar phrases or other people's voices (Apple). Apple chose the balance between false alarms and false rejects experimentally, and its word error rates are directly influenced by trigger performance (Apple, HomePod research).

Misses leave no transcript and are more likely in noisy places, so measure them on a scripted, labelled evaluation set recorded in quiet and noisy conditions; the HomePod research reports false-reject rates on evaluation data. In live logs, near-threshold trigger scores and repeat requests within seconds are proxy signals.

In this illustrative case, 10,000 requests are spoken, 2,000 in noisy rooms. Cutting the noisy miss rate from 10 to 4 percent adds hard audio to the logs, and a more sensitive trigger would also add false accepts, which this model leaves out.¹

With the recogniser unchanged, making the trigger more sensitive in noisy rooms lifts the measured word error rate from 9.67% to 9.92% in this illustrative case.
Quantity (illustrative)	Before	After
Noisy-room requests logged	1,800	1,920
Measured word error rate	9.67%	9.92%

The recogniser did not get worse; the logs got harder. Report quality by condition and keep trigger and recogniser dashboards separate.

Step 3: Measure each dimension on a reviewed sample

Humans label a small random sample, and the sample gives the error rate for the whole stream. Word error rate (WER) is the Levenshtein distance between the recognised words and the human reference transcript, divided by the number of words in the reference; the distance is the fewest substitutions, insertions and deletions that turn one word string into the other (arXiv 2211.16112). Automatic speech recognition (ASR) is the part of the system that turns speech into text, and WER is how it is scored.

The reference "set a timer for ten minutes" heard as "set a time for ten minutes" scores 16.67%, and a second example with one deletion and one insertion scores 50%.¹
Reference	Heard	Edits	Reference words	WER
set a timer for ten minutes	set a time for ten minutes	1 substitution	6	16.67%
call mum at six	call at six thirty	1 deletion, 1 insertion	4	50%

Take 1,000,000 requests a day (illustrative). Reviewing under 0.2 percent, the share Apple reported, means about 2,000 requests.¹ For a yes/no rate such as false-accept share, 2,000 reviews give a worst-case margin of error of about 2.19 points at 95 percent confidence, shrinking with the square root of the sample size.¹ WER can exceed 100 percent and its errors cluster within utterances, so bootstrap over sampled utterances for its interval.

The margin of error falls with the square root of the sample, so a few thousand reviewed requests give a usable error rate.CC BY 4.0 — source: How would you improve data quality on a product like Siri?

Text description of the diagram

At 2,000 reviewed requests the worst-case margin of error is about 2.19 points.
The margin shrinks with the square root of the sample size.

Do not stop at WER. The arXiv paper notes that no single definition serves every purpose, and WER ignores how serious each error is: mishearing "timer" costs more than dropping "the".

Step 4: Fix the weakest dimension at its source, then re-measure

Fix the weakest dimension at its source, then re-measure. If noisy rooms drive the errors, the source is coverage: collect and label more noisy audio and re-score the same sample design. If false accepts drive them, the source is the trigger, and the trade-off against false rejects must be tuned and judged together.

Data collection has privacy limits that shape the fix. Apple says that by default it no longer retains audio recordings of Siri interactions and continues to use computer-generated transcripts to help Siri improve (Apple Newsroom). The consequence is that old audio cannot be re-transcribed by a better model, so ground-truth review has to rely on consented samples. Apple later made manual review opt-in and brought audio grading in-house, and its Siri Speech Study app lets participants explain what they meant when Siri misheard (TechCrunch, 2021). Apple also says personalised speech recognition runs on the device, which cuts what is sent to servers (Apple Legal). Test a fix with an experiment, as in reading an A/B test like a product owner.

Knowing it is half the job. Saying it is the other half.

Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.