How do you design an ads ranking system?

In this article

1. Before you start
2. Why interviewers ask this
3. Ads ranking system from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one topic of the machine learning (ML) system design interview guide. It assumes only three ideas.

Why interviewers ask this

The question joins four things a machine learning (ML) design round tests: a probability model, an economic objective, a data pipeline and a serving budget. Candidates report variants of it for Meta and other companies, including a version that names calibration; those are candidate reports, not public statements of what any company asks.

The reason it carries weight is the money path. He et al. (2014) note that advertisers bid and pay for measurable responses such as clicks, so click prediction is central to most online advertising systems. At that time Facebook had over 750 million daily active users and over 1 million active advertisers (as of 2014).

A weak answer says "train a model to predict clicks and sort by score." A strong answer ranks by value, not by click rate alone, and explains why the number must be a true probability. A Staff-level answer adds the funnel, the training-data distortion and budget pacing.

Ads ranking system from first principles

Effective cost per mille (eCPM)

Effective cost per mille is a score that states how much an ad is expected to earn per thousand impressions, computed as bid times estimated click rate times one thousand, so ads with different bids and click rates can be sorted on one scale.

Negative downsampling

Negative downsampling is the practice of keeping only a small fraction of the no-click examples in training data so a rare-event model trains faster, which raises the model's predicted rates and so requires a correction before the numbers are used.

In plain words

Think of a market stall auction where each seller says what they will pay per sale. A stall that pays a lot but sells almost nothing can lose. A stall that pays less and sells often can win. To compare them, the auctioneer multiplies what each pays by how often a visitor buys. The analogy breaks in one place: the auctioneer must guess "how often" before the visitor arrives, using a model.

Said precisely, an ads ranking system scores each eligible ad by expected value, using a calibrated probability of the paid action. It shows the highest scorers within each advertiser's budget.

Step 1: Rank by expected value, not by bid or by click rate

The ranking score is the bid multiplied by the estimated probability of the paid action, which for clicks is eCPM when scaled by one thousand. Input: a bid and a click estimate per ad. Output: one comparable number.

Ranking by bid alone shows a high-paying ad nobody clicks. Ranking by click rate alone gives away the advertiser's willingness to pay. Real platforms add terms such as ad quality or user experience, and the exact formula is platform-specific, so state your assumption in the interview ("score equals bid times estimated action rate, plus a quality term if needed") instead of claiming a company's formula. You now know the objective. The rest of the design exists to make the click estimate trustworthy.

Step 2: Calibrate the click estimate so the auction compares fairly

The click estimate must be a true probability, because the auction multiplies it by money. A uniform scale error leaves the ad-against-ad order unchanged: doubling every probability keeps every bid-times-estimate product in the same order. Calibration matters because real errors differ by ad and by slice, which flips winners; in the worked example one ad overstated twofold wins a slot and costs 10 score points.¹ It also matters because the absolute value meets reserve prices, organic content and budgets. Area under the receiver operating characteristic curve (AUC), which measures ranking quality only, cannot see either problem.

He et al. (2014) judge models with calibration, the ratio of the average estimated CTR to the empirical CTR, next to normalized entropy. Normalized entropy is the average log loss divided by the log loss of always predicting the average CTR; lower is better; it reacts to both ranking and calibration. Output: a mapping from score to a matching probability, checked by deciles. Common fixes are a Platt-style sigmoid or isotonic regression (Niculescu-Mizil and Caruana, 2005).

Step 3: Build labels and correct for negative downsampling

Clicks are rare, so training data is lopsided: almost every impression is a no-click. Teams keep every click and only a fraction of the no-clicks to cut cost, and that is negative downsampling. The side effect is that the model sees clicks far more often than the real world does, so its outputs run high.

The correction follows from the keep rate $w$. If $q$ is the model's output on downsampled data, the true rate is $p=q/(q+(1-q)/w)$. The worked example shows the number going out and coming back exactly.¹ Also decide the label window: a click arrives in seconds, but a purchase can arrive days later, so late conversions look like negatives unless you wait or model the delay. Input: logged impressions and actions. Output: labeled training rows and a known keep rate to undo.

Step 4: Narrow millions of ads to a few in stages

A heavy model cannot score every ad for every request. The funnel has three layers: retrieval, which uses targeting rules, budgets and cheap similarity search to cut the pool to a few thousand; a light ranker, which scores that set with a fast model; and a heavy ranker, which scores the short list with rich features and the calibrated click model. The same funnel idea appears in search ranking.

Input: all active ads and one request. Output: a short ranked list with eCPM scores. The trade-off is recall against cost: whatever retrieval drops, the heavy model never sees. He et al. (2014) report that a high-value feature, or the right model to begin with, improves results much more than tuning data freshness, learning rate or sampling, which tells you where to spend interview time.

In the heavy ranker, features come in families: user, ad, context, cross features that pair a user trait with an ad trait, and historical CTR counters. Models usually progress from logistic regression, to boosted trees feeding logistic regression, to deep models with embeddings, to multi-task heads for click and conversion. Judge a candidate before deployment with normalized entropy and calibration on held-out data, then confirm it with an A/B test on revenue, CTR and advertiser outcomes.

Step 5: Run the auction and pace the budget

The auction takes the ranked eCPM scores, fills each slot with the highest scorer, and charges according to its pricing rule (a design choice to state, such as a second-price style rule). Pacing is the control that spreads an advertiser's daily budget across the day so it does not run out in the first hour, usually by throttling how often the ad enters auctions or by scaling its bid.

Input: scored ads and remaining budgets. Output: winners, prices and updated budgets. Close the loop by logging every impression, click and price, because that log is both the training data for the next model and the evidence for the calibration check. Delayed feedback and exploration for brand-new ads, which have no history, are the next questions to expect.

1 · Value formula2 · Calibrate3 · Labels4 · Funnel5 · AuctionReplay

Step-by-step description

1. Rank by bid times estimated click rate.
2. Make the estimate a true probability.
3. Undo negative downsampling.
4. Narrow ads in stages.
5. Run the auction and pace budgets.

• The ranking score is the bid times the estimated click rate.
• The click estimate must be calibrated before the auction.
• The model sees clicks far more often than the real world does, so its outputs run high.

Worked example

Three ads compete for one slot. All numbers are illustrative, not measurements of any platform. Ad A bids 4 units and truly gets a click rate of 0.010, but the model says 0.020. Ad B bids 2.5 and truly gets 0.020, predicted 0.020. Ad C bids 1.5 and truly gets 0.025, predicted 0.025.¹

Ranking by bid alone also picks A. Ranking by the miscalibrated estimate picks A. Ranking by the true click rate picks B, so the auction gives up 10 score points on this slot.¹

Now the downsampling check. Take a base CTR of 0.001 and keep one negative in a hundred (illustrative). The training CTR becomes 9.1%.¹ Applying the correction gives back 0.001 exactly.¹

Finally the decile check on 20,000 simulated impressions with 531 clicks, from a model built to over-predict by a factor of 1.5. The mean predicted CTR is 0.0414 against an observed 0.0266, a calibration ratio of 1.558. In the top decile the model says 0.1299 and the data shows 0.084. The AUC is 0.7197, and it is exactly the same after every prediction is divided by 1.5, so AUC alone cannot reveal the fault. The chart merges the ten deciles into five groups of two.¹

Text description of the diagram

• The mean predicted CTR is 0.0414 against an observed 0.0266.
• The chart merges the ten deciles into five groups of two.

Trade-offs and edge cases

Calibration has to be maintained, not done once. Traffic shifts by hour, country and ad format, so teams check calibration per slice, not only overall, because a model can be calibrated on average and wrong in a segment.

Other edge cases a Staff reader expects:

• Cold-start ads. A new ad has no click history, so its estimate leans on features and an exploration budget; without it, the model never learns about new ads.
• Feedback loops. The model only sees clicks on ads it chose to show, so it can lock in its own early guesses.
• Delayed conversions. If the paid action is a purchase, recent impressions look unlabeled; wait, or model the delay.
• Ranking and metrics. The Facebook paper reports that combining boosted decision trees with logistic regression beat either alone by over 3% (He et al., abstract); judge such gains with a metric that sees calibration.

Text description of the diagram

• AUC is unchanged when every prediction is divided by the same number.
• The calibration ratio compares mean predicted and observed click rate.
• Normalized entropy reacts to both ranking and calibration.

The same pattern of a cheap first stage and an expensive second stage shows up in typeahead, and a policy filter before the auction connects to harmful-content detection. Fraudulent clicks bend both the labels and the budgets, which is the subject of real-time fraud detection, and sudden click spikes are caught with anomaly detection.

How to answer it in an interview

The spoken structure: state the value formula, say why calibration follows, give the funnel, then name downsampling, the check and pacing.

Common mistakes

• "Highest bid wins." (wrong) That shows ads nobody clicks. (right) "I rank by bid times estimated click rate, plus a quality term if the platform uses one."
• "Our AUC is high, so the model is ready." (wrong) AUC ignores scale. (right) "AUC checks order; I also check the ratio of predicted to observed rate by decile."
• "Downsampling negatives is free." (wrong) It raises every predicted rate. (right) "I undo it with the known keep rate before the auction."
• "One daily score is enough." (wrong) Budgets and traffic move within the day. (right) "I pace spend through the day and recheck calibration per slice."

Follow-up questions

Design an ads ranking system with calibration?

Rank by bid times a calibrated click estimate. Fit a mapping such as isotonic regression on fresh traffic, undo any negative downsampling, and track predicted against observed rate by decile and by slice.

Why is AUC not enough for an ads model?

AUC measures only whether clicked impressions score above unclicked ones, so a uniform scale error leaves it unchanged. Errors that differ by ad or slice flip winners, and the absolute value meets reserve prices, organic content and budgets. The decile check shows the miscalibration AUC cannot detect.

Design an ad-serving platform with targeting and analytics?

Add a targeting layer in retrieval (audience, location, budget filters), the ranking funnel above, and an event log that feeds reporting dashboards and the next training run. Keep the log schema identical for training and analytics.

Design query generation to maximize CTR?

It is the same click-model loop with a different action: the system proposes queries and learns from clicks. Generate candidate queries first, rank them with a click model, then apply a relevance guardrail, because optimizing clicks alone learns clickbait.

How do you handle delayed conversions?

Use a label window long enough for most conversions, or train a model of the delay. Treat very recent impressions as unlabeled instead of as negatives.

Sources

1. Practical Lessons from Predicting Clicks on Ads at Facebook — ACM, ADKDD 2014 (He et al.), 24 Aug 2014
2. Practical Lessons from Predicting Clicks on Ads at Facebook (open PDF copy) — ADKDD 2014 (He et al.), author-hosted copy, 24 Aug 2014
3. Predicting Good Probabilities With Supervised Learning — ACM, ICML 2005 (Niculescu-Mizil and Caruana), 7 Aug 2005