MLOps

How do you build a data flywheel from production feedback?

By Interview Clozo · Published 27 October 2026 · Last verified 26 October 2026 · 13 min read

A data flywheel is a repeating loop you build from production feedback: it logs requests, corrects and labels the feedback, trains a candidate, gates it on a held-out evaluation and releases it, so each release produces the next loop's data. The error to avoid is treating feedback as an unbiased sample, and the feedback section shows a thumbs-down share far above the true error rate.

In this article

Before you start
Why interviewers ask this
Data flywheel from first principles
The Staff-level answer
The rubric
Three variants of this question
Common mistakes
Follow-up questions
Sources

Before you start

This page is one answer in the machine learning operations (MLOps) interview questions path, and it leans on the basics of machine learning algorithms.

Why interviewers ask this

The question separates people who have only trained a model once from people who have kept one useful after launch. A model that ships and is never updated slowly decays, and the data that would fix it is already sitting in production logs. Interviewers want to hear whether you can turn that raw material into a safe, repeatable release cycle.

A weak answer says "collect thumbs-down and retrain." A strong answer names each stage (logging, feedback, selection, training, a release gate, monitoring) and says what can go wrong in each. A Staff-level answer adds the failure that is easy to miss: the feedback you receive is not a random sample of your traffic, so training on it naively teaches the model the wrong lesson, and the loop can reinforce its own mistakes.

Data flywheel from first principles

Data flywheel: A data flywheel is a repeating process that uses data exhaust from a live system to improve the model behind it, so that a better model attracts more use and produces more exhaust. It can improve accuracy, latency and cost, not accuracy alone.
Data exhaust: Data exhaust is the by-product data a running application produces anyway: prompt and response logs, user feedback signals, and labels added by human experts. Nobody collects it for training in advance, which is why it must be logged on purpose.

In plain words

Picture a restaurant that keeps notes on every dish sent back and every plate returned empty. Each week the chef studies the notes and adjusts the recipes. The chef tastes the new dish against the old one before it reaches the menu. Better food brings more diners, and more diners mean more notes. The analogy breaks in one place. Diners who complain loudly are not a fair picture of all diners, and a flywheel must account for that.

In one precise sentence: a data flywheel is a loop of log, learn from feedback, evaluate, release and monitor, where each release produces the data for the next loop. NVIDIA's data flywheel blueprint describes it as using production data exhaust to raise accuracy and cut the latency and cost of generative AI systems. Its README says the project is no longer actively maintained and is retained for reference only. Treat it as a description of the pattern, not a product to adopt.

Step 1: Log every request with enough context to replay it

Logging is the base of the loop: record the input, the model's output, the model version, the settings and a request identifier for every request. Without the identifier you cannot join a later thumbs-down to the exact response it judged.

Google's Rules of Machine Learning advises saving the features used at serving time and piping them to a log for use in training. The reason is training-serving skew, which is a difference between how a model performs in training and in serving, caused for example by different data handling in the two pipelines. If you train on features rebuilt differently from those the live model saw, the lessons do not transfer. A feature store and request tracing (LLM observability) are the usual homes for this record. Input: a live request. Output: one replayable log row.

Step 2: Collect feedback and remember it is a biased sample

Feedback comes in several kinds. Explicit feedback is a thumbs-down or a rating. Implicit feedback is behavior, such as a user rephrasing the question or abandoning the page. Expert feedback is a reviewer's label. NVIDIA's README lists logs, end-user feedback and expert labeling as the exhaust, so thumbs buttons are only one source.

Selection bias is the error that appears when the chance of an example being observed depends on the outcome you want to measure, so the observed examples do not represent the whole. Take an illustrative day of 1,000,000 requests in which 6% of responses are truly bad. Users of a bad response leave a thumbs-down 3.5% of the time. Users of a good response leave a thumbs-down about 0.1% of the time (900 in all) or a thumbs-up 1.5% of the time. You receive 3,000 thumbs-down, of which 70% come from truly bad responses, and 14,100 thumbs-up.¹ Thumbs-down are then 17.5% of all feedback, while the true error rate is 6%.¹ Feedback covers only 1.71% of traffic.¹

Correcting the biased feedback

A random audit estimates the error rate directly. Label 400 randomly chosen requests and the error rate comes with a margin of about 2.3 percentage points at 95% confidence.¹ The audit is too small to measure the bias itself: at a 3.5% rate, about one of the roughly 24 bad responses in it would have left feedback.¹

To correct metrics computed on the much larger feedback pool, estimate the feedback propensities, the chance that a response of each kind gets feedback, from labeled samples stratified across feedback and no-feedback traffic. Then weight each feedback item by the inverse of its class's propensity, estimating the class from the labeled samples because thumbs-down are only 70% precise. In this scenario the weighted estimate is 6%, recovered by construction because the rates are known.¹ Output: feedback joined to the log, with weights.

Raw feedback overstates the error rate; with known feedback rates, weighting recovers the true figure by construction.CC BY 4.0 — source: How do you build a data flywheel from production feedback?

Text description of the diagram

Thumbs-down are 17.5% of feedback while the true error rate is 6%.
The weighted estimate is 6%, recovered by construction because the rates are known.

Step 3: Select and label the examples worth learning from

You cannot label everything, so choose. Three pools are useful: examples with negative feedback, examples where the model was uncertain, and a random slice that keeps the training data representative. Suppose a team can label 500 examples a day. A random draw finds about 30 bad responses, while drawing from thumbs-down (70% of them truly bad, as in Step 2, illustrative) finds about 350.¹

Targeted sampling is more efficient, but it is not representative, so keep a random slice and record which pool each label came from. Labels come from human reviewers (added on purpose, unlike the by-product exhaust), from users' own corrections, or from a stronger model that proposes a label a person spot-checks. Input: the weighted feedback pool. Output: a labeled set with its source recorded. Take care with personal data: strip or mask it before it enters a training set, and check that the product's terms allow reuse.

Step 4: Train a candidate model on the new data

Training turns the labeled set into a candidate. Fine-tuning is continuing to train an existing model on new examples so it adapts without starting from scratch; for a language model the examples can be corrected responses or pairs of a preferred and a rejected response. Always mix the new examples with the older data, otherwise the candidate may fix today's failures and forget earlier skills.

The candidate is a new version with its own identifier, trained from a frozen, versioned data snapshot so that you can reproduce it. Training runs as a scheduled job on shared infrastructure. Input: a snapshot of labeled data plus the current model. Output: one candidate, not yet released.

Step 5: Gate the candidate on an evaluation before release

The gate compares the candidate with the live model on held-out examples the training never touched, ideally including a random slice from production. A candidate is promoted only if it clears fixed rules, for example no drop on any critical category.

The held-out set must be big enough to see a difference. At 90% accuracy, 250 examples give a margin on one model's accuracy of about 3.7 percentage points, 1,000 examples about 1.9 and 4,000 about 0.9, all at 95% confidence (illustrative).¹ The gate compares the candidate and the live model on the same examples with a paired test, such as McNemar's test, on the examples where they disagree, so the gain it can detect depends on how often the two models disagree. Release gradually: send a small share of traffic to the candidate first and compare it with the live model. Output: a promote or block decision.

Step 6: Release, monitor and feed the next loop

After promotion, monitor the same signals that fed the loop: feedback rate, quality sampled from a random audit, latency and cost. Watch for model drift, and keep the previous version ready for rollback.

The loop can also be a feedback loop unwanted: Google lists a feedback loop between the model and the algorithm as a source of training-serving skew. If the model decides what users see, keep a small random exploration share. Output: a live version that produces tomorrow's data exhaust.

Each loop starts with logging every request and ends with a monitored release, and the release produces the next loop's data.CC BY 4.0 — source: How do you build a data flywheel from production feedback?

Step-by-step description

Record every request with its version and identifier.
Collect feedback and correct its bias.
Select and label the examples worth learning from.
Train a candidate on a versioned snapshot.
Promote only if it passes the held-out evaluation.
Release gradually, monitor, and feed the next loop.

Each loop starts with logging every request.
A candidate model is released only after it passes a promotion gate.
The monitored release produces the next loop's data.

In the illustrative scenario, feedback overstates the error rate nearly threefold, and the loop pays back its training cost in about eight days.
Quantity	Value (illustrative scenario)
Requests per day	1,000,000
True error rate	6%
Thumbs-down share of all feedback	17.5%
Weighted estimate of the error rate	6% (recovered by construction when the rates are known)
Feedback coverage of traffic	1.71%
Bad responses found by 500 random labels	about 30
Bad responses found by 500 targeted labels	about 350
Daily cost before, in cost units	1,000,000
Daily cost after routing 70% to a small model	370,000
Days to repay a one-off training cost of 5,000,000 units	7.9

The cost rows use relative units: a large model costs 1 unit per request and a small one 0.1, and the loop trains a small model good enough for 70% of traffic.¹ That is a 63% saving, against the vendor-reported best case of "up to 98.6%" in NVIDIA's own tests. NVIDIA reports that figure for one internal use case, tool calling in a human resources (HR) chatbot, where a tuned llama-3.2-1b-instruct replaced llama-3.1-70b-instruct. It is a per-use-case model swap, not a whole-system saving, so read it as a ceiling for that kind of swap, not a typical result.

Knowing it is half the job. Saying it is the other half.

Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.