In this article
Before you start
This page is one step of the recommendation system design learning path.
Why interviewers ask this
A notification system can lose a user in a way a feed cannot: the user switches notifications off and the channel is gone. LinkedIn Engineering writes that repetitive, excessive and low-quality notifications can lead members to file complaints and disable notifications, and can make them less likely to use the app. That cost is why the question separates candidates.
Candidates report prompts such as "design a personalized notification ranking system" and variants about eligibility, order and suppression. A weak answer trains a click model and sorts by its score. A strong answer treats the problem as several decisions: who is eligible, how to score, how many to send, when and on which channel. A Staff-level answer adds the fatigue term, the volume budget and an evaluation plan that does not trust accuracy metrics alone.
The problem also differs from search. The user sees one notification from a candidate set, there is no query to show intent, and relevance decays with delay, so a notification that is timely now may be irrelevant if sent later. Notification ranking decides whether, when and on which channel to send, not only the order.
Text description of the diagram
- A user sees one notification per candidate set.
- There is no explicit query to show intent.
- Notification ranking decides whether, when and on which channel to send.
Design notification ranking from first principles
- Notification ranking
- Notification ranking is a decision system that takes the candidate notifications waiting for one user and decides which to send, in what order and when, using a score of expected value rather than click probability alone.
- Fatigue cost
- A fatigue cost is a penalty subtracted from a notification's expected value that stands for the harm of interrupting the user, such as complaints, disabled notifications or lower app use.
- Notification budget
- A notification budget is the maximum number of notifications a system may send one user in a period, such as a week, set per user from a model of how that user reacts to volume.
In plain words
Think of a personal assistant who may tap your shoulder to pass on messages. A good one skips trivia, waits for a calm moment and never taps ten times in an hour. The analogy breaks in one place. A human assistant knows your plans, while the system has only past behavior, and the user gives no explicit context about what they need right now.
In short, notification ranking keeps the candidates that are allowed. It gives each a score that weighs benefit against interruption cost. It sends only the best ones that fit the user's budget.
Step 1: Generate candidates and filter for eligibility
Eligibility removes candidates before any scoring. Candidates arrive from several sources, such as friend activity, product announcements and advertiser messages, and each source is a separate producer. Meta describes machine-learned models that decide who should get a notification, when to send it and what content to include, and LinkedIn adds channel choice among badge, push and email, volume optimization and message spacing.
The eligibility filter applies cheap, explainable rules: the user muted this type, notifications are turned off, the content was deleted, the same event was already sent, or a minimum gap since the last send has not passed. Rules keep the model from spending effort on candidates that must never go out, and they give you something to audit. Input: all candidates for one user. Output: the eligible list. The next step scores that list.
Step 2: Score each candidate by expected value minus fatigue cost
The score of a notification is its click probability times its value, minus its fatigue cost. Click-through rate (CTR) is the share of delivered notifications that users tap, and a click model estimates it per candidate. Sorting by that estimate alone ranks a cheap, tap-bait notification above a valuable one.
Value is a weighted sum of goals. LinkedIn's relevance layer combines three utilities: whether a member visits, their engagement on the notifications, and their overall engagement on LinkedIn, and it tunes the trade-off online. The fatigue term comes from the harm LinkedIn names: complaints, disabled notifications and less use. Several estimates feed the score, so a multi-task ranking model, which predicts several outcomes at once, is one natural model shape.
Training the click model can use a loss shaped for this setting: Yue et al. weight the pairwise loss between candidates by the expected regret of misordering the pair. The probabilities must be calibrated, meaning a predicted 0.2 should come true about one time in five, or the multiplication by value is meaningless.
Which features, labels and fatigue estimate feed the score?
The score needs defined features, labels and a fatigue estimate before any training. Feature groups are user activity and recency, time since the last notification, the user's open rate by notification type, item freshness and source, hour of day and device. The click label is a tap within an attribution window, for example 24 hours (illustrative); a tap after the window counts as a negative, so the window trades delayed clicks against how long you wait for training data.
Dismissals, disabling a type and unsubscribing are kept as separate negative labels. The fatigue cost is then estimated, not guessed: train a model for the probability that the user disables or unsubscribes after this send, and multiply it by a tuned weight. Pinterest likewise predicts a user outcome, activity next week, given the volume sent.
Step 3: Set a weekly budget per user
A per-user budget caps how many notifications go out. The budget is its own model, separate from the ranker. Pinterest describes budgeting as calculating the optimal number of notifications to send each person in a week, in two steps: utility prediction, then budget allocation. It trains a gradient boosting decision tree (GBDT) classifier, a model built from many small decision trees added one after another so each corrects the errors of the earlier ones, to predict whether the user will be active next week given a budget b this week.
Meta used causal inference to find highly active users who would see content organically anyway, and sent them fewer notifications while improving the overall experience. The budget answers how many, and the score answers which ones. A budget is also a guard against the fatigue that the score only prices one notification at a time.
Step 4: Decide send or suppress, channel and timing
A send decision combines the score, the threshold and the remaining budget. Send the highest-scoring candidates while budget remains and the score clears a minimum; suppress the rest. LinkedIn built Air Traffic Controller (ATC) as the ultimate decision maker for notifications; the design intent is that no producer bypasses the gate.
Timing matters because relevance decays. A notification that is timely now may be irrelevant if sent later, so the system either sends soon or drops the item. Channel choice and message spacing, as in the LinkedIn relevance layer, refine the decision. Meta's diversity-aware ranking reduces uniformity and, per the post, lowers volume while raising engagement rates. For sequences of decisions, LinkedIn reports a reinforcement learning (RL) policy for send-or-drop choices that is fully deployed.
Step 5: Evaluate with replay and an online experiment
An online experiment decides whether the policy ships. Accuracy metrics such as area under the receiver operating characteristic (ROC) curve (AUC) measure how well a click model orders examples, not whether the whole policy is better. Importance sampling is a method that reweights logged outcomes so they estimate how a new policy would have performed on the traffic the old policy produced. Prabhakar et al. use a one-step importance sampling method to evaluate trained policies before deployment and to choose which candidates go to an online test.
Then run a randomized experiment on real users; Yue et al. report gains in a production experiment. Track the primary engagement metric together with guardrails such as opt-outs, because a gain in taps that comes with more disabled notifications is a loss.
Step-by-step description
- Rules remove candidates before any scoring.
- Click probability times value, minus fatigue cost.
- A per-user budget caps volume.
- Send the best that clear the minimum score.
- An online experiment decides whether the policy ships.
- Eligibility removes candidates before any scoring.
- The score equals click probability times value minus fatigue cost.
- A per-user budget caps how many notifications go out.
- An online experiment decides whether the policy ships.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
One user has four eligible candidates and one ineligible one. Every number below is illustrative, not a measurement or a figure from any company.1 The user's weekly budget is four notifications and two are already sent, so two remain, and the minimum score to send is 0.05 (illustrative).1
| Candidate (illustrative) | Click probability | Value | Fatigue cost | Score |
|---|---|---|---|---|
| Friend activity | 0.30 | 1.0 | 0.10 | 0.20 |
| Advertiser message | 0.45 | 0.3 | 0.15 | -0.015 |
| Product announcement | 0.10 | 1.2 | 0.10 | 0.02 |
| Group update | 0.20 | 0.8 | 0.10 | 0.06 |
| Muted friend post | 0.25 | 1.0 | 0.10 | not scored (ineligible) |
The eligibility filter drops the muted post before scoring.1 Sorted by click probability, the order is advertiser message (0.45), friend activity (0.30), group update (0.20) and product announcement (0.10), so a two-slot budget sends the advertiser message and friend activity, with a total score of 0.185 (illustrative).1
Sorted by score, the order is friend activity (0.20), group update (0.06), product announcement (0.02) and advertiser message (-0.015). The two sent are friend activity and group update, with a total score of 0.26.1 The product announcement falls below the 0.05 minimum and the advertiser message has a negative score, so both are suppressed. The advertiser message has the highest click probability and the worst score, because its value is low and it carries the highest fatigue cost.
Now add delay. Suppose friend activity has a click probability of 0.3 when it is sent at once and that it halves every two hours (illustrative). Click probability halves every two hours in this illustrative example. It falls from 0.3 now to 0.0375 after six hours, and the score drops from 0.2 to -0.0625, so the item should be suppressed once its score falls below the minimum, which here happens just after two hours, where the score is exactly 0.05 and then drops below the minimum.1
Text description of the diagram
- Click probability halves every two hours in this illustrative example.
- It falls from 0.3 now to 0.0375 after six hours.
Trade-offs and edge cases
Notification ranking trades short-term taps against long-term retention. Each approach below optimizes something different, and the right one depends on how much logged data and engineering the team has.
| Approach | Optimizes | Main risk | Use when |
|---|---|---|---|
| Sort by click probability | taps | fatigue, opt-outs | a quick baseline |
| Score minus fatigue plus budget | value per send | weights need tuning | the standard design |
| Sequential policy (RL) | long-term value | hard evaluation | enough logged data |
Edge cases matter in the interview. New users have no history, so fall back to type-level averages and a conservative budget; see the cold start problem. Logged data exists only for notifications the old policy sent, so the next model learns from yesterday's decisions, and you need exploration or reweighting; the feedback loop page explains why. Friend-activity candidates may come from systems like people you may know.
A language model may write the message text, which is a separate decision covered in large language models (LLMs) inside a recommender. A team can choose rules plus a tuned score, as in the LinkedIn relevance layer whose decision function is tuned online, or a sequential policy, as LinkedIn also reports with reinforcement learning, so state your choice and why.
How to answer it in an interview
Design a personalized notification ranking system.
I would split it into gates. First, an eligibility filter removes muted types and recent duplicates. Then a model scores each candidate as click probability times value, minus a fatigue cost, with calibrated probabilities. A per-user weekly budget model caps volume, and I send the top candidates that clear a minimum score, choosing timing and channel because relevance decays. I would evaluate with replay estimates first, then an online experiment that watches opt-outs as a guardrail. Ranking by click rate alone raises taps but burns the channel.
The spoken structure: name the gates, give the score formula, add the budget, then say how you would test it and what guardrail you would watch.
Common mistakes
- "I rank by predicted click rate." (wrong) Click rate ignores fatigue and long-term value. (right) "I score expected value minus a fatigue cost."
- "More notifications mean more engagement." (wrong) Meta reduced volume and still raised engagement rates. (right) "Volume is a budget that I tune per user."
- "This is the same as search ranking." (wrong) Only one item is seen per set, there is no query and relevance decays. (right) "It is a send-or-suppress problem with a time dimension."
- "Good AUC means a good system." (wrong) AUC measures ordering, not the policy. (right) "I check replay estimates and an online experiment with opt-out guardrails."
Follow-up questions
How would you batch notifications into a digest or choose a channel?
LinkedIn's relevance layer selects the channel among badge, push and email and spaces messages by state-aware delivery timing. In your design, treat channel and timing as a decision after the score, so a candidate that misses the push threshold but is still valuable can be held for a digest or a lower-interruption channel. That holding rule is a design assumption, not a reported practice.
How would you design a personalized notification ranking system for Apple devices?
The ranking logic stays the same; the platform adds constraints. Apple's notification interruption levels are passive, active, time-sensitive and critical (as of October 2026), so the design can map a candidate's urgency to a level instead of treating every send alike. As an assumption, reserve time-sensitive for candidates whose value decays fast and keep critical out of ranked social or marketing sends. User-controlled Focus and notification settings are hard eligibility constraints: a blocked candidate is removed before scoring, not scored low. Privacy expectations on Apple devices favor on-device features, so state the assumption that sensitive signals are computed on the device and only coarse values reach the server. Confirm each point in the developer documentation.
How do you decide which notifications are eligible, which to show first and which to suppress?
Eligibility is rule-based and runs first. Order eligible candidates by score, and suppress any below the minimum score or beyond the budget. Reported prompts of this kind come from candidates, and the same three-gate answer fits them.
How do you evaluate a notification ranking model?
Use replay estimates such as importance sampling to pick candidates, then run an online experiment on the primary engagement metric plus opt-out and complaint guardrails. Prabhakar et al. describe the importance-sampling step for choosing online test candidates.
Why does calibration matter for notification ranking?
The score multiplies a probability by a value and compares across notification types. If one type's probabilities run high, it wins unfairly, so check calibration per type and recalibrate.
Why not just rank notifications by click-through rate?
Click-through rate rewards tap-bait and ignores the cost of interrupting. LinkedIn combines several utilities instead, and the fatigue term keeps users from disabling the channel.
- Notification ranking decides whether, when and on which channel to send, not only the order.
- Score each candidate as click probability times value minus fatigue cost, with calibrated probabilities.
- A per-user budget caps volume, and eligibility rules run before any scoring.
- Relevance decays with delay, so send soon or suppress.
- Test with replay estimates, then an online experiment with opt-out guardrails.
Sources
- Learning to Rank For Push Notifications Using Pairwise Expected Regret (arXiv 2201.07681) — arXiv (Yue et al.), submitted January 2022, 19 Jan 2022
- A New Ranking Framework for Better Notification Quality on Instagram — Engineering at Meta, 2 Sep 2025
- Improving Instagram notification management with machine learning and causal inference — Engineering at Meta, 31 Oct 2022
- Air Traffic Controller: Member-First Notifications at LinkedIn — LinkedIn Engineering, undated post, accessed 2026-10-26, 26 Oct 2026
- Near Real Time AI Personalization for Notifications at LinkedIn — ACM (WSDM 2022), 21 Feb 2022
- UNNotificationInterruptionLevel — Apple Developer Documentation, undated, accessed 2026-10-26, 26 Oct 2026
- User state-based notification volume optimization — Pinterest Engineering, undated post, accessed 2026-10-26, 26 Oct 2026
- Multi-objective Optimization of Notifications Using Reinforcement Learning — ACM (Prabhakar et al., KDD 2022), 14 Aug 2022
