Recommender feedback loops: training data is yesterday's output

In this article

1. Before you start
2. Why interviewers ask this
3. The feedback loop from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

Why interviewers ask this

Candidates reported this theme in system design rounds on feeds, restaurant search, advertising and writing tools, so it is worth having a clean answer. The question sounds like a data question but tests whether you see that the model changes the data it later learns from.

A weak answer treats clicks as a fair vote: "we collect interactions and retrain daily." A strong answer asks which items the user could even see, says that an unclicked item that was never shown is missing data rather than a rejection, and adds a mechanism that gathers fresh evidence on purpose. A Staff-level answer also separates the fixes: exploration to collect new data, and weighting to correct data you already have. It then says how to measure the loop, with a holdout that the model does not control.

The feedback loop from first principles

Feedback loop

A feedback loop in a recommender is a cycle in which the deployed model decides which items users see, the users' reactions to those items are logged, and the logs become the training data of the next model, so the model partly trains on its own earlier decisions.

Exposure bias

Exposure bias is the skew in logged data that arises because only items the previous model chose to show could receive clicks, so items that were rarely shown look unpopular for lack of evidence rather than lack of appeal.

In plain words

Imagine a teacher who grades only the essays of students in the front row. She then picks next term's front row from those grades. Students at the back never get graded, so they never get promoted, whatever they could write.

The analogy breaks in one place. A teacher could walk to the back row any day. A recommender learns about the back only if it deliberately shows those items.

In one precise sentence: a recommender learns click rates only for items it exposes. Its estimate for an unexposed item stays wherever it started.

Step 1: Yesterday's model decides what is exposed

The loop starts with a ranking model that scores every candidate and puts the highest scores in the visible slots. Input: a user and a candidate list. Output: an ordered list in which only the top few items are seen. The model's scores are therefore not just predictions; they are decisions about which evidence will exist tomorrow.

A reranker or a candidate-retrieval stage makes the same kind of decision earlier in the pipeline: anything it drops is never even scored. Jiang et al. (2019) describe the general effect: the system's decisions influence user beliefs and preferences, which in turn affect the feedback the learner receives, and this can give rise to echo chambers and filter bubbles. You now know where the data's shape comes from: the slots.

Step 2: Only exposed items can be clicked

A user cannot click an item the page never displayed. Input: the ranked list. Output: a log row per displayed item, with a click or no click. Items below the visible slots produce no row at all, so the log has no entry that says "the user would have liked this."

This is why an unclicked item can mean two different things: shown and skipped, or never shown. A shown-and-skipped item is weak evidence of low interest. A never-shown item is no evidence either way. Treating them the same, for example by labelling every unseen item a negative, bakes the old policy into the labels. You now know the log is a record of exposure first and taste second.

Step 3: Retraining on those logs locks the gap in

The next model trains on the log. Input: yesterday's logged rows. Output: new scores. Items that were shown get more rows, so their estimated click rates sharpen toward their true rates. Items that were not shown get no new rows, so their estimates stay at whatever early, noisy value they had. If that value was low by bad luck, the item stays below the item that is shown, is never shown, and never gets the chance to correct itself. In the worked example the better item stays at 0.05 against a true 0.12.¹

The McMahan et al. (2013) paper from Google's ads team names the inherent feedback loop of such a system and notes that, without strong extra assumptions, it blocks theoretical results about the effect of calibration. Calibration, there, means the average predicted rate matches the observed rate on a slice of data.

Step 4: Exploration and weighting weaken the loop

Two repairs work on different parts of the problem. Exploration collects new evidence by routing a small share of impressions to items the model would not choose; the cost is a few worse impressions now. Weighting corrects data you already have: if you know the chance that each logged item was shown, you can divide each click by that chance so that rarely shown items count for more.

Propensity is the chance that an item was shown to a user in a given slot, and weighting by its inverse is the correction. Joachims et al. (2016) use this for position bias, the effect that a result's slot strongly changes how many clicks it receives, and note that training directly on raw clicks gives sub-optimal rankings.

Weighting only works for items the old policy could have shown: an item with a zero chance of exposure has no rows to weight. A greedy ranker gives every unshown item that zero, so exploration is also what makes propensities non-zero and known.

1 · Model decides2 · Only seen items3 · Logs lock in4 · Break the loopReplay

Step-by-step description

1. Yesterday's model picks the items users will see.
2. Only exposed items can receive clicks.
3. Retraining on the logs freezes unexposed items' estimates.
4. Exploration adds new evidence to the logs.

• Yesterday's model decides which items are exposed.
• Only exposed items can be clicked.
• Retraining on those logs locks the gap in.
• Exploration and weighting weaken the loop.

Worked example

Two items, A and B, compete for one visible slot. All values here are illustrative. A has a true click rate of 0.10 and B has 0.12, so B is better. Each starts with 100 prior impressions, but A has 10 clicks (0.10) and B only 5 (0.05) by bad luck. The slot receives 1,000 impressions a day for 30 days, and each day the item with the higher estimated rate takes the top slot, apart from an exploration share split equally between the two items.¹

In the greedy run item A takes the top slot every day, B receives nothing, and B's estimate stays at 0.05.¹ With 10% exploration, B gets 50 impressions a day from exploration alone, which lifts its estimate to 0.1025 by the end of day 6, just above A's 0.10. From day 7 B holds the top slot, and its estimate reaches 0.1183 by day 10.¹ With 10% exploration the estimate reaches 0.1197 on day 30, close to the true 0.12.¹ A smaller share still works, only slower: at 2% the lead comes on day 26.¹

Text description of the diagram

• With no exploration item B's estimate stays at 0.05.
• With 10% exploration the estimate reaches 0.1197, close to the true 0.12.

Position weighting is the second repair. Suppose, as an assumption and not a measurement, that the chance of a slot being noticed is 1.0, 0.5, 0.33, 0.25 and 0.2 for slots one to five. An item with a true rate of 0.2 once seen then shows raw click rates of 0.2, 0.1, 0.066, 0.05 and 0.04. Dividing each click by its slot's chance gives weights of 1, 2, 3.03, 4 and 5 and recovers 0.2 in every slot.¹

Trade-offs and edge cases

Exploration is not free. Each exploration impression may be worse than the model's first choice, and the cost is felt as lost clicks in the weeks you pay it. A larger share learns faster but costs more, and the right share depends on how fast items arrive and how fast tastes change; that is a decision to measure, not a constant to quote.

Weighting has its own limit: it works only when the propensities are known or well estimated. A mistaken propensity model moves the bias rather than removing it, and a very small propensity produces a huge weight and noisy training. Joachims et al. report robustness to misspecified propensities, but only within the settings they tested. A deterministic policy has propensities of 0 or 1, so log the serving probabilities from a randomised policy.

Text description of the diagram

• Exposure bias means items the model does not show collect no data.
• Position bias means top slots collect more clicks.
• Propensity weighting divides each click by the chance it was seen.

The table states four forms of one problem. Exposure bias means items the model does not show collect no data. Position bias means top slots collect more clicks. Popularity bias means items that already have many impressions collect still more. New items face the cold start problem: no history, so no exposure, so no history. Calibration helps with the scale of predictions, not with which items get seen, and multi-task ranking adds more signals but does not by itself add exposure. Reported practice differs on how much traffic to reserve for exploration, so give a rationale and not a number.

How to answer it in an interview

The spoken structure: say the data is the old policy's output, say why unseen is not negative, then give exploration, weighting and a holdout.

Common mistakes

• "Items the user didn't see are negatives." (wrong) Unseen means no evidence; only shown-and-skipped carries weak negative signal. (right) "I treat unexposed items as unobserved, and model the chance they were shown."
• "More logged data will fix the bias." (wrong) More data from the same policy keeps the same exposure bias. (right) "I need data from a different policy, so I add exploration."
• "I'll evaluate the new model on last week's logs." (wrong) The logs reflect the old policy, so items it hid look bad. (right) "I evaluate with a randomised holdout or a controlled experiment, and I weight logged evaluation."
• "Cold start is a separate topic." (wrong) New items get no exposure, so they get no data; it is the same loop. (right) "Cold start is the loop's extreme case, and exploration is the shared fix."

Follow-up questions

Let's say a user sees the top five restaurants and chooses one. Should we mark the others as negative?

Only the shown ones, and only weakly. The four shown and skipped items are weak negatives, with slot position taken into account, while restaurants outside the visible list are unobserved and carry no label.

What are the problems for a newly registered restaurant?

It has no impressions, so it has no clicks, so the model gives it a low or default score and never shows it. Reserve exploration traffic for new items and pair it with content features; see the cold start problem.

How would you design a feedback loop to improve an AI writing tool over time?

Log what the tool suggested, what the user accepted, edited or rejected, and what was never offered. Because the tool only learns from suggestions it chose to make, vary suggestions on a small share of traffic and keep a holdout with a different policy, so accepted suggestions are not just the ones it already favoured.

How would you monitor a feedback loop for advertising bid models?

Compare predicted and observed rates on slices of data, which is the calibration check described by McMahan et al. (2013), and watch it over time because the traffic and the model both drift. Keep a small randomised holdout so you can tell model decay from a changed market. Candidates have reported this as an advertising question; the answer here is advice, not a known company practice.

Why is feedback-loop bias higher in a reels feed than on YouTube, where users can search?

Search, browse and subscriptions create exposure the ranker did not choose, so those logs hold some off-policy evidence. In a feed that only shows what the system serves, every logged row comes from the ranker's own choices. Exploration therefore matters more there, including at candidate selection, where a dropped item is never even scored. Candidates reported this framing in a feed design round.

How would you get users to give extra data to improve recommendations?

Ask for signals that do not depend on exposure, such as a "not interested" control or a short preference choice, and keep them rare and optional. Explicit answers are sparse but unbiased by the ranker's choices, so use them to check the click-based model. More logged rows from the same policy would repeat the same skew; fresh evidence needs exploration or a changed policy. The wider design context is in the machine learning system design guide.

Sources

1. Ad Click Prediction: a View from the Trenches — Google (McMahan et al., KDD 2013), 11 Aug 2013
2. Degenerate Feedback Loops in Recommender Systems (arXiv 1902.10730) — arXiv (Jiang et al.), 27 Feb 2019
3. Unbiased Learning-to-Rank with Biased Feedback (arXiv 1608.04468) — arXiv (Joachims, Swaminathan, Schnabel), 16 Aug 2016