In this article
Before you start
This page belongs to the machine learning system design interview path, so the usual round structure (clarify, data, model, evaluation, serving) applies here too.
Why interviewers ask this
Ranking is where product judgment meets machine learning, so one prompt tests many skills. Candidates report prompts such as ranking search results for Airbnb Experiences, ranking emails for a query and a user profile, or building a search ranking model for Bing. These are reported by candidates, not confirmed company policy.
A weak answer names a model ("gradient boosting on features") and stops. A strong answer starts with the objective, explains where labels come from and why clicks mislead, and says how a model that scores a handful of items per request differs from one that scores millions. A Staff-level answer adds the failure modes: position bias, feedback loops, a metric gain that does not survive an A/B test, and a latency budget that decides which model may run where. The sections below build that answer in the order you would say it.
Search ranking system from first principles
- Search ranking system
- A search ranking system is a pipeline that turns a query, and often a user, into an ordered list of results by retrieving candidates cheaply and then scoring them with trained models, so the most useful results appear first within a fixed latency budget.
- Candidate generation
- Candidate generation is the first stage of a ranking pipeline, a fast retrieval step that cuts a huge corpus down to a short list of plausible results so that slower, more accurate models only score that short list.
In plain words
Picture a company hiring for one job. A recruiter skims a thousand résumés in a few minutes and keeps a short pile. A panel then spends an hour on each person in that pile and ranks them. Nobody would give every résumé a full hour. The analogy breaks in one place: a search engine must answer in a fraction of a second, and it learns from how past visitors behaved, not from a panel's notes.
In one precise sentence, a search ranking system estimates a usefulness score for each candidate result given the query and the user, then sorts by that score.
Step 1: A funnel retrieves first and ranks second
The first design decision is the funnel. Scoring every document with the best model is too slow, so the system retrieves a short list with a cheap method, such as keyword matching or a vector index, then ranks that list. Covington et al. (2016) describe this two-stage shape for YouTube: candidate generation first, ranking second.
With an illustrative corpus of 10,000,000 documents, the funnel retrieves 1,000, a light ranker keeps 200, and a heavy ranker orders those 200 and shows 10.1 Each stage takes a list and returns a shorter, better-ordered list; the retrieval, ranking and reranking funnel covers the shape in general. Input: the query and the corpus. Output: a short list. You now know why the stages exist; the next step is how the models in them learn. For the query-side companion, see the typeahead and query autocomplete design.
Step 2: Training labels come from clicks, but clicks must be debiased
A ranker needs labels that say which results were good. Two sources dominate: human raters, who grade relevance but are slow and costly, and user behavior such as clicks, which is plentiful but biased. Position bias means a result shown first is clicked more than the same result shown fourth, so a model trained on raw clicks learns to repeat the old ranking.
A standard correction is to weight each click by the inverse of how likely its position was to be examined, the propensity-weighting approach of Joachims, Swaminathan and Schnabel (2016), who use click models as the propensity estimator. With an illustrative examination chance of 1, 0.5 and 0.25 at ranks 1, 2 and 4, and an equal true click chance of 0.3, the observed click rates are 0.3, 0.15 and 0.075, and the weights are 1, 2 and 4.1 Another option is to randomize a small share of results to measure examination directly. Input: logged impressions and clicks. Output: labels corrected for where each result sat.
Step 3: A ranking model learns from features, with a pointwise, pairwise or listwise loss
The ranker takes features of the query, the result and the user, and outputs a score. Learning to rank is the practice of training a model so that its scores produce a good ordering rather than good individual predictions. Three loss families exist. A pointwise loss treats each result as its own example. A pairwise loss learns which of two results should win; pairwise is the name for training on pairs of results. A listwise loss looks at the whole ordered list and a list metric.
Haldar et al. (2018) report that Airbnb's early search gains came from a gradient boosted decision tree model, that the gains plateaued, and that the team then applied neural networks to break out of the plateau. A common starting point is therefore trees, with a neural model justified by what the trees no longer capture. Many systems also train one model with several output heads, for example click, purchase and return, and combine the head scores into one ranking score; this is multi-objective, or multi-task, ranking. Input: debiased labels and features. Output: a scoring function.
What features does a search ranker use?
A search ranker uses four groups of features: how well the result matches the query, how good the result is, who is asking, and the context of the request. Match features include a classic term-frequency keyword score and the similarity between a query embedding and a result embedding. Quality features include a result's age, its popularity and its past engagement.
User features include history and language, and context features include device, time and location. Position is a special case. A strong design uses it as a feature only at training time, so the model can separate the effect of the slot from relevance, and sets it to a constant at serving time, because the final position is not known until the ranking is done.
Step 4: Evaluate with a ranking metric, then with an A/B test
Batch evaluation on logged data comes first and is cheap, but it does not replace the online test. Precision at k is the share of the top k results that are relevant. Normalized discounted cumulative gain (NDCG) is a metric that rewards putting highly relevant results near the top, by discounting each result's gain by the log of its rank and dividing by the best possible ordering's score. Manning et al. (2008, chapter 8) define it with a gain of 2^R − 1 for a grade R and a discount of log₂(1 + m) at rank m.
A gain in NDCG on logged data is a reason to run an A/B test, not proof of a win. The test measures what users do, such as successful searches, and also watches latency and complaints. Input: a candidate model. Output: a ship or no-ship decision.
Where are features computed at request time, and what do you monitor?
Document features are computed ahead of time and stored for fast lookup, while query and user features are computed or fetched when the request arrives, so the heavy ranker pays only for what depends on the request. Precomputing document features keeps the latency budget for the model itself. In production, monitor four things: latency per stage against the budget, the freshness of stored features, drift in feature and score distributions compared with training, and the online metric. A common failure is a mismatch between the feature values seen in training and at serving, so log the features used at serving and train on those logs. See the real-time fraud detection design for monitoring streaming signals.
Step-by-step description
- Retrieve cheaply, then rank a short list.
- Correct clicks for position bias.
- Train a ranker on the labels.
- Check NDCG, then run an A/B test.
- A cheap retrieval stage narrows the corpus before heavy ranking.
- Click labels need a position bias correction before training.
- A gain on logged data is confirmed by an A/B test.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
Take one query with six results graded 0 to 3 for relevance (illustrative); a result counts as relevant for precision if it is graded 1 or higher. In the order the system returned them, the grades are 3, 2, 3, 0, 1, 2. The discounted gain at each rank is the gain 2^R − 1 divided by log₂(1 + m), which gives 7.0, 1.8928, 3.5, 0, 0.3869 and 1.0686; they sum to a DCG (discounted cumulative gain) of 13.8483.1 The ideal order, 3, 3, 2, 2, 1, 0, has a DCG of 14.5954, so NDCG at 6 is 13.8483 / 14.5954 = 0.9488.1
Now put the irrelevant result first: 0, 3, 2, 3, 1, 2. The DCG falls to 10.3867 and the NDCG to 0.7116, while precision at 3 drops from 1.0 to 0.6667.1
| Quantity | Value (computed or illustrative) |
|---|---|
| NDCG at 6, relevant result first | 0.9488 |
| NDCG at 6, irrelevant result first | 0.7116 |
| Precision at 3, relevant result first | 1.0 |
| Precision at 3, irrelevant result first | 0.6667 |
| Light ranker, 1,000 candidates at 0.02 milliseconds each | 20 milliseconds |
| Heavy ranker, 200 candidates at 0.5 milliseconds each | 100 milliseconds |
| Heavy ranker on all 10,000,000 documents | 5,000 seconds |
The per-document costs are illustrative: the two ranking stages cost 120 milliseconds in total, while scoring all 10,000,000 documents with the heavy model costs 5,000,000 milliseconds, about 41,667 times more.1 That ratio is the reason the funnel exists.
Text description of the diagram
- NDCG at 6 is 0.9488 with the relevant result first and 0.7116 with an irrelevant result first.
- Precision at 3 is 1.0 and 0.6667 for the same two orders.
Trade-offs and edge cases
Each loss family trades simplicity for fidelity to the final metric. Pointwise training is easy but ignores order. Pairwise training sees order, though the number of pairs grows quickly with list length. Listwise training targets the metric most directly but is harder to optimize, because list metrics such as NDCG are not smooth.
Text description of the diagram
- A pointwise loss treats each result as its own example.
- A pairwise loss learns which of two results should win.
- A listwise loss looks at the whole ordered list.
Edge cases worth naming in an interview:
- Feedback loops. The model's ranking decides what users see, which decides the next training data. Randomized exploration slices break the loop.
- Personalization. Adding user features helps repeat users and does little for new ones; the cold-start case needs a fallback to query-and-item features.
- Choosing K. A fixed retrieval depth wastes compute on easy queries and misses answers on hard ones; a design can choose K per request from query difficulty, within the latency budget.
- Fairness. If photo results for a query show poor demographic balance, ranking by clicks can reinforce it. Candidates report this prompt for image search; measure diversity as its own metric, and see the computer vision system design questions for the vision side.
- Calibration. An ads ranker multiplies a click estimate by a bid, so the estimate must be a true probability, not just an ordering.
- Neighboring problems. Ads, feed and notification ranking reuse this skeleton with different objectives (see the follow-up on how the design changes by surface); a harmful-content filter often sits before ranking, and an embedding stage can feed retrieval (multimodal embedding service design).
How to answer it in an interview
How would you design a search ranking system?
I would first agree the objective, for example successful searches, then build a funnel: cheap retrieval to a few hundred candidates, then a ranker over that short list. For labels I would combine rater grades with clicks, weighted to correct position bias. I would start with gradient-boosted trees using a ranking loss and judge them by NDCG, then confirm in an A/B test with a latency guardrail. The risk is a feedback loop, so I would keep a small randomized slice.
The spoken structure: say the objective, then the funnel, then the labels and their bias, then the model and metric, then the A/B test and the risk.
Common mistakes
- "I'll train on clicks." (wrong) Clicks reflect position as well as relevance. (right) "I'll weight or randomize to correct position bias before training."
- "One big model scores the whole corpus." (wrong) That breaks the latency budget, as the worked example shows. (right) "Retrieval cuts the corpus first, and the heavy ranker scores a short list."
- "Our NDCG went up, so we ship." (wrong) A metric on logged data is not user behavior. (right) "NDCG gates the A/B test; the test decides."
- "Accuracy is the metric." (wrong) Ranking cares about order near the top. (right) "I use NDCG at k and precision at k, and I track latency."
Follow-up questions
How would you design a search ranking model for Bing?
Use the same funnel: retrieval from an index, a light ranker, then a heavier learning-to-rank model trained on rater grades and debiased clicks. Add query-understanding features and judge with NDCG plus an A/B test.
How do you design a personalized search ranking system for Airbnb Experiences?
Frame the objective (bookings that guests rate well), add user and listing features, correct position bias in the click logs, and compare on logged data before an A/B test. Haldar et al. (2018) are the closest public account of a marketplace ranker, covering Airbnb stays search.
How do you rank emails for a query and a user profile?
Retrieve candidate emails by query match within the user's mailbox, then rank with features such as sender, recency and past interaction. Personalization is the follow-up: it enters as user-specific features, with care for privacy.
How do you choose K, the number of candidates, for each request?
Train a small model on query features to estimate how deep the relevant results lie, then set K per request within the latency budget. Judge it by recall at the chosen K against the cost saved.
How does the design change for ads, feed and notification ranking?
The objective, the candidate source and one extra constraint change (the ads ranking system design goes deeper). Ads ranking multiplies a click probability by a bid, so that probability must be calibrated (check predicted against observed click rates in score buckets); feeds blend engagement signals and need diversity limits; notifications decide whether to send at all and need volume caps. The funnel, debiased labels, ranker and A/B test stay the same.
How do you reduce demographic bias in image search results?
Measure diversity of the top results per query, then change retrieval or re-rank to meet a diversity target without breaking relevance. Report both numbers in the A/B test.
- A search ranking system retrieves candidates cheaply, then ranks a short list with heavier models.
- Raw clicks carry position bias; correct them before training a ranker.
- NDCG at k rewards good order near the top; precision at k only counts relevant results.
- A gain on logged data earns an A/B test, not a launch.
- Name the risks: feedback loops, calibration for ads, and fairness in image results.
Sources
- Applying Deep Learning To Airbnb Search (arXiv 1810.09591) — arXiv (Haldar et al., Airbnb), 24 Oct 2018
- Unbiased Learning-to-Rank with Biased Feedback (arXiv 1608.04468) — arXiv (Joachims, Swaminathan, Schnabel), 16 Aug 2016
- Introduction to Information Retrieval, chapter 8: Evaluation in information retrieval — Cambridge University Press (Manning, Raghavan, Schutze), 7 Apr 2008
- Deep Neural Networks for YouTube Recommendations — Google Research (Covington et al., RecSys 2016), 15 Sep 2016
