In this article
Before you start
This page is one step of the recommendation system design learning path.
Why interviewers ask this
A candidate who sorts one list of titles and stops has answered a smaller problem than the one asked. Candidates report several wordings of this prompt for Netflix design rounds: a dynamic recommendation system, the Top Picks feature, a real-time homepage with a tight latency budget and cache invalidation, and the design of an experiment for a new recommendation surface.
Interviewers use it to see whether you treat the homepage as page construction. The homepage is where discovery happens: Gomez-Uribe and Hunt (2015) report that two of every three hours streamed on Netflix are discovered there.
A weak answer names one model and one metric. A strong answer separates candidate rows, row rankers and page assembly, and says how a change is tested. A Staff-level answer adds the tensions the page must balance and the traps in the experiment, such as the novelty effect.
Netflix's homepage from first principles
- Page generation
- Page generation is the process that turns a pool of candidate rows and videos into one finished, ordered and deduplicated homepage for a member on a device. It sits above the individual row rankers and decides which rows appear, in what order and with which videos, balancing relevance with diversity, freshness and layout limits.
- Novelty effect
- The novelty effect is a temporary change in behavior caused by a change being new rather than better. In a recommender test it can show up as members playing previously undiscovered titles, which makes the result look better than the medium-term effect will be.
- Effective catalog size
- The effective catalog size (ECS) is a measure that shows how widely a member's plays spread over the catalog. With titles sorted by plays and p_i the share of plays of title i, ECS = 2 × (sum of p_i × i) − 1. It is 1 when every play hits one title and N when plays spread evenly over N titles.
In plain words
Think of a newsstand where every shelf has a theme. A stocker picks which shelves to show you. The stocker also picks their order and the items on each. The analogy breaks in one place: every member gets a different newsstand, built in real time.
In one precise sentence, the homepage is a grid of rows, each row is a ranked list of videos, and page generation chooses and orders the rows.
Alvino and Basilico (2015) describe the homepage as videos organized into thematically coherent rows in a two-dimensional layout. A row is a themed, ranked list of videos, and each row usually comes from a single algorithm. Gomez-Uribe and Hunt (2015) stress that the recommender is "a collection of different algorithms serving different use cases", not one model.
Requirements to clarify first
Ask what the member does and what the page is for. Functional needs: show rows of titles per profile on each device, let a member resume what they were watching, and handle new titles and new members. Non-functional needs: a latency budget the interviewer states, freshness after a member acts, and scale across regions.
Then list signals. Wang et al. (2026) list explicit signals such as play, add to My List and thumbs-up, implicit signals such as trailer views or details-page visits, and request context such as time of day, day of week and device. A model learns from what members did, not from star ratings: Gomez-Uribe and Hunt describe stars as long past as the focus, with ranking built on viewing context. Adaptive bitrate selection is a separate video-delivery system and is out of scope here.
Step 1: Generate candidate rows
Page generation starts from a pool of possible rows for the member. Gomez-Uribe and Hunt (2015) say a typical member has tens of thousands of rows that could go on the homepage, which makes the computation hard to manage. Alvino and Basilico (2015) describe the first move as building candidate groupings of videos before any final page exists.
Input: the member, profile, device and context. Output: a set of candidate rows, each with a candidate video list. In an interview, say that the pool is large, so you cannot fully rank every row for every request; you prune or precompute. The sources do not publish per-row candidate counts, so label any figure you use as your assumption. You now have the raw material; the next step makes each row good.
Step 2: Filter, then rank videos inside each row
Each candidate group is filtered, then ranked by an algorithm suited to that row. Alvino and Basilico (2015) say the filters handle concerns like maturity rating and removing some previously watched videos, and that videos are ranked "according to a row-appropriate ranking algorithm".
Gomez-Uribe and Hunt (2015) name several. The personalized video ranker (PVR) is a general-purpose ranker that orders the whole catalog or a subset for a member and is blended with popularity; because it is used so widely, its rankings must work across the catalog, which limits how personalized it can be. The Top-N ranker is optimized only for the head of the ranking and drives the Top Picks row.
Trending Now uses short-term trends, and Continue Watching ranks recently viewed titles by the estimated chance of resuming or abandoning. The sims algorithm is an unpersonalized list of similar videos for every video, though which Because You Watched rows appear is personalized. Ranking on several goals at once is covered in multi-task ranking.
Text description of the diagram
- The personalized video ranker is a general-purpose ranker blended with popularity.
- The Top-N ranker is optimized only for the head of the ranking and drives the Top Picks row.
Step 3: Select and order rows for relevance and diversity
Row selection decides which rows make the page and in what order, and it is where a naive design fails. Alvino and Basilico (2015) call the simplest approach, treating rows as items in a ranking problem, a row-ranking approach, and warn that it "would lack any notion of diversity". Rows with the highest scores can all be near copies of each other.
Their progression is a rule-based template, then row ranking, then a stage-wise greedy method that adds diversity as it picks rows, then full-page optimization. The last is out of reach: a page layout is defined in a discrete space, so directly optimizing a page-quality function is a computationally prohibitive integer programming problem.
The page also balances competing goals: discovery against continuation, accuracy against diversity, and freshness against stability, plus device constraints. Say these out loud in an interview; they show you see a page, not a list.
Step 4: Assemble the page and format it for the device
Assembly makes the page consistent and fits it to the screen. Alvino and Basilico (2015) say that as the page is assembled there is extra filtering such as deduplication to remove repeat videos, and rows are formatted to the right size for the device. A video that qualifies for two rows should appear once.
Gomez-Uribe and Hunt (2015) report about 40 rows on a typical homepage, depending on the device, and up to 75 videos per row. The arXiv paper on the value of recommendations (arXiv 2511.07280) also describes the billboard: one highlighted title per user that spans the width of the screen.
Serving speed matters here. Wang et al. (2026) note that the homepage is often generated incrementally, a few rows at a time, so recommendations respond to in-session behavior, and that latency is a primary engineering constraint.
Step 5: Choose the artwork with a bandit
The last choice is how each title is shown. Chandrashekar et al. (2017) personalize artwork with contextual bandits. A contextual bandit is an online learning method that picks one option per request from the context and keeps some randomness so it keeps learning; it is designed to minimize regret.
Randomizing is needed because, in their words, "if a member plays a title it can only come from the image that we decided to present to that member." Logging what was shown and with what probability lets them correct for skewed selection and evaluate models without bias. They also say the system must respond quickly when a title launches, a cold-start case. For scale, they note that personalized selection for each asset means a peak of over 20 million requests per second with low latency.
Step-by-step description
- A member has tens of thousands of possible rows.
- Each row is ranked by its own algorithm.
- Row selection balances relevance and diversity.
- Duplicates are removed and rows fit the device.
- A bandit picks the image and keeps exploring.
- A member has tens of thousands of candidate rows.
- Each row is filtered and ranked by its own algorithm.
- Row selection balances relevance and diversity.
- Page assembly removes duplicate videos and formats rows for the device.
- Artwork is chosen by a contextual bandit that keeps exploring.
A member has tens of thousands of candidate rows. Each row is filtered and ranked by its own algorithm. Row selection balances relevance and diversity. Page assembly removes duplicate videos and formats rows for the device. Artwork is chosen by a contextual bandit that keeps exploring.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
A page holds at most about 3,000 slots: about 40 rows times up to 75 videos per row.1 A member, though, "loses interest after perhaps 60 to 90 seconds of choosing, having reviewed 10 to 20 titles" (Gomez-Uribe and Hunt, 2015). Ten to twenty titles are 0.33% to 0.67% of the slots.1 The first screen must therefore carry the best of the page, which is why row order matters as much as row content.
Take 30,000 candidate rows (illustrative; the paper says only tens of thousands). Keeping 40 means keeping 0.13% of them.1
Testing a change is the costly part. The paper's decision rule calls a win when the measured gap is at least 1.96σ, where σ² = pc(1−pc)/nc + pt(1−pt)/nt for control and treatment retention rates and cell sizes. At retention of 50% in both cells with equal sizes, σ² = 0.5/n, so n = 0.5 × (1.96 ÷ gap)², with the gap written as a fraction (0.001 for 0.1 percentage points). These samples are the point at which a true gap just reaches 1.96σ, which is about a coin-flip chance of a win. A true gap of 0.5 percentage points just reaches 1.96σ at a sample of 76,832 per cell.1
A true gap of 0.2 percentage points does so at 480,200, and a true gap of 0.1 percentage points just reaches 1.96σ at a sample of 1,920,800 per cell.1 An 80% chance of a win needs the factor (1.96 + 0.84)² in place of 1.96², about twice as many: 3,920,000 per cell for 0.1 percentage points.1 The paper says "we need roughly 2 million members per cell to measure a retention delta of 50.05% to 49.95%=0.1% with statistical confidence." That is close to our coin-flip figure of 1,920,800.
| Quantity | Value (computed or illustrative) |
|---|---|
| Slots per page, 40 rows × 75 videos | 3,000 |
| Titles reviewed (10 to 20) as a share of slots | 0.33% to 0.67% |
| Candidate rows kept, 30,000 illustrative to 40 | 0.13% |
| Sample per cell, 0.5 percentage point gap, 1.96σ bar | 76,832 |
| Sample per cell, 0.2 percentage point gap, 1.96σ bar | 480,200 |
| Sample per cell, 0.1 percentage point gap, 1.96σ bar | 1,920,800 |
| Sample per cell, 80% power, 0.1 percentage point gap | 3,920,000 |
| Sample over both cells, 0.1 percentage point gap, 1.96σ bar | 3,841,600 |
| Toy ECS, 100 titles, even plays (illustrative) | 100 |
| Toy ECS, 100 titles, plays proportional to 1/rank (illustrative) | 37.56 |
Text description of the diagram
- A true gap of 0.5 percentage points just reaches 1.96σ at a sample of 76,832 per cell.
- A true gap of 0.1 percentage points just reaches 1.96σ at a sample of 1,920,800 per cell.
The toy ECS rows show the metric's meaning: with plays concentrated on a few titles the score falls far below the catalog size.
Trade-offs and edge cases
Accuracy against diversity, freshness against stability
Alvino and Basilico (2015) list competing page-level goals: discovery against continuation, accuracy against diversity, and freshness against stability. A page that updates on every action feels responsive but jumpy; one that never changes feels stale. Popularity helps too: Gomez-Uribe and Hunt say PVR works better with a healthy dose of popularity. Because rankers feed on what they showed, read how recommender feedback loops work.
Replay on logged data against randomized tests
Gomez-Uribe and Hunt say replay on logged data prunes variants, but it assumes members would have behaved the same way had a new algorithm made the recommendations, and it is not highly predictive of A/B outcomes. Their tests run randomized cells for typically 2 to 6 months. On existing members the novelty effect can inflate a result, because the change exposes previously undiscovered titles. Their remedy is to test on new members. They also choose retention and medium-term engagement over raw hours.
One model against a pipeline
Wang et al. (2026) describe the traditional homepage as a multi-stage pipeline with separate candidate generation and ranking at row and entity levels. GenPage replaces page construction with one model: each entity and each row is a single token, the vocabulary updates daily, and the reward is the sum over all impressed entities, so a binged show scores above a movie watched for 10 minutes. They report significant gains on the launch metric and 20% lower end-to-end serving latency. That is one recent system, not all of Netflix personalization; for large language models (LLMs) inside a ranker see designing LLMs inside a recommender. Notifications share signals but are a separate problem, covered in notification ranking.
How do you serve the homepage within a tight latency budget and keep the cache fresh?
Serving a homepage quickly means doing the heavy work before the request and only assembling the page when it arrives. The sources I read do not publish Netflix's cache design or a latency target, so this is first-principles reasoning and every number is my assumption. Precompute in batch what changes slowly: candidate rows and row rankings per profile and device. Update on events, such as a play that changes Continue Watching. At request time assemble only the first screen and load lower rows lazily; this fits the incremental generation, a few rows at a time, that Wang et al. (2026) describe.
Keep a per-profile and device cache with a time to live, and invalidate an entry when an event changes recent history. If an entry is missing or a model call is too slow, fall back to a popularity-based page. Take a budget of 100 milliseconds that an interviewer might state. For scale, 200 million members loading the page twice a day is about 4,630 requests per second on average and 13,889 at a peak three times higher (illustrative).1
How to answer it in an interview
How would you design Netflix's homepage?
First I would clarify the goal: members finding something to watch, measured by retention, on many devices. The homepage is a grid of themed rows. I generate candidate rows, filter titles the member should not see, and rank videos inside each row with a row-specific ranker. Then I select and order rows for relevance and diversity, because sorting rows by score alone gives near copies. I remove duplicates, fit the device and personalize artwork with a bandit that keeps exploring. I judge changes in randomized tests lasting months, on new members to avoid novelty.
The spoken structure: goal and metric, then rows, row rankers, page selection with diversity, assembly, artwork, and last the experiment.
Common mistakes
- "I will predict star ratings and sort by them." (wrong) Netflix's engineers describe ranking built on viewing context, not stars. (right) "I rank with viewing behavior and context, and optimize retention and engagement."
- "One model outputs the whole page." (wrong) The recommender is a collection of algorithms. (right) "Many row rankers plus a page generator; a single-model generator is one recent approach."
- "I sort rows by score." (wrong) That lacks diversity. (right) "I select rows with a diversity term and page-level goals."
- "The A/B test won, so ship it." (wrong) Novelty can inflate a test on existing members. (right) "I check medium-term effects and test on new members."
- "Click-through is the objective." (wrong) The paper chooses retention and medium-term engagement. (right) "I optimize retention with engagement as support."
Follow-up questions
How would you design a dynamic recommendation system for Netflix?
Use the five stages: candidate rows, row rankers, row selection with diversity, assembly, and artwork. Clarify requirements and signals first, then say how you test changes.
How do you design the homepage for low latency and cache invalidation?
The sources I read do not describe Netflix's cache design, so reason from first principles. Precompute what changes slowly, such as candidate rows, and generate the page incrementally, a few rows at a time; GenPage reports 20% lower end-to-end latency than the multi-stage stack. Invalidate on a member action that changes continue-watching or recent history.
How do you design the experiment for a new recommendation surface and handle novelty effects?
Randomize members into cells, pick retention and medium-term engagement as outcomes, and size cells from the effect you must detect and the power you want: for a 0.1 percentage point gap, about 2 million members per cell is the coin-flip point and about 3.9 million gives 80% power. Run for months, prefer new members to avoid novelty, and check that engagement supports any retention win.
Would you use hover-over behavior as a signal?
The sources I read do not document hover. Netflix's GenPage post lists implicit signals such as trailer views and details-page visits. Treat hover as weak implicit feedback, log what was shown, and test whether it improves the outcome.
How do you handle a new title or a new member?
For members, Gomez-Uribe and Hunt say the cold-start approach evolved into a survey during sign-up. For titles, the artwork bandit must learn quickly at launch; see the cold start problem.
- The homepage is a grid of themed rows, and each row usually comes from its own algorithm.
- Page generation selects and orders rows for relevance and diversity; sorting rows by score alone lacks diversity.
- Artwork is personalized with a contextual bandit that keeps exploring and logs what it showed.
- Judge changes in randomized tests that run for months, with retention as the main outcome.
- A true 0.1 percentage point retention gap just reaches the 1.96σ bar at about 2 million members per cell; 80% power needs about twice that.
Sources
- The Netflix Recommender System: Algorithms, Business Value, and Innovation — ACM Transactions on Management Information Systems, DOI 10.1145/2843948 (Gomez-Uribe and Hunt), 1 Dec 2015
- Learning a Personalized Homepage — Netflix Technology Blog (Alvino and Basilico), 9 Apr 2015
- GenPage: Towards End-to-End Generative Homepage Construction at Netflix — Netflix Technology Blog (Wang, Pan, Baltrunas), 29 Jun 2026
- Artwork Personalization at Netflix — Netflix Technology Blog (Chandrashekar, Amat, Basilico, Jebara), 7 Dec 2017
- The Value of Personalized Recommendations: Evidence from Netflix (arXiv 2511.07280) — arXiv, 1 Nov 2025
