How do you design LLMs inside a recommender system?

In this article

1. Before you start
2. Why interviewers ask this
3. LLMs inside a recommender from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page extends the recommendation system design guide, which covers the basic stages.

Why interviewers ask this

The consequence of a careless answer is easy to see: a design that calls an LLM for every candidate of every request collapses under its own traffic. Candidates report prompts of this kind in interviews at short-video and AI-chat companies, for example designing LLM-enhanced recommendations for a mobile feed or suggesting starting questions in an AI chat app; these are prompts as reported by candidates, not a company's published practice.

The question separates people who name LLM features from people who place them. A weak answer says "use an LLM to understand users better." A strong answer names where the model sits (before serving, in the ranker, in retrieval or in a chat layer), what it costs per call and how often it runs, and what stays classical. A Staff-level answer adds how the system is evaluated and what happens when the model is wrong or slow.

LLMs inside a recommender from first principles

LLM-enhanced recommender

An LLM-enhanced recommender is a recommendation system that keeps its retrieval and ranking stages and adds a large language model at one or more points in the pipeline, for example to enrich item text, rerank candidates, generate item identifiers or converse with the user.

Item enrichment

Item enrichment is the batch use of a language model to turn an item's raw metadata into richer text, tags or embeddings, run once per item and stored, so no model call is needed when a user asks for recommendations.

LLM reranker

An LLM reranker is a language model that receives a user's history and a short list of candidate items in a prompt and returns them in a new order, run at request time over candidates that another model retrieved.

Semantic ID

A Semantic ID (identifier) is a short tuple of codewords that stands for one item and is built from the item's content, so items with similar content share parts of their identifier; a sequence model can then generate the identifier of the next item.

In plain words

Think of a record shop. The clerk who reads the sleeve notes of every new album and writes a one-line summary on a card does that work once, in the back room. The clerk at the counter, who must pick three albums for a customer standing there, has seconds. An LLM in a recommender can be either clerk, and the whole design question is which job you give it. The analogy breaks in one place. A shop clerk talks to one customer at a time. A recommender serves a very large number of requests concurrently, so counter work multiplies.

In one precise sentence, an LLM-enhanced recommender keeps the retrieve-then-rank pipeline. It places a language model where its text understanding is worth its cost per call. The survey of Lin et al. discusses "the roles that LLM could play in different stages of the recommendation pipeline, i.e., feature engineering, feature encoder, scoring/ranking function, user interaction, and pipeline controller."

Step 1: Batch enrichment turns item text into features

Item enrichment runs the model once per item, ahead of time. The input is raw metadata, such as a title, a caption or a transcript; the output is cleaner text, tags or an embedding that goes into the item store next to the existing features. At serving time nothing calls the LLM, so the request path is unchanged.

This is the safest placement, and it explains the cold-start benefit: an item with no clicks still has text, so enrichment gives the retrieval and ranking models something to work with while the cold start problem lasts. Enrichment of users works the same way, for example summarising a long history into a short profile that is refreshed on a schedule. The survey names feature engineering as the first stage where an LLM can play a role (Lin et al.). For a mixed-media feed, LLM or multimodal encoders turn video frames, audio transcripts and captions into item embeddings in batch. You now know the cheapest role; the next step shows why it is cheap.

Step 2: Cost and latency decide per item or per request

A latency budget is the time the whole recommendation request is allowed to take, split across its stages; any stage that does not fit its slice cannot run in the request path. The same model has two very different price tags, depending on how often it is called.

Use illustrative inputs: 10,000,000 items, 2% of them refreshed a day, 100,000,000 requests a day. Batch enrichment then needs 200,000 calls a day, about 2.31 calls a second. Reranking 50 candidates per request needs about 1,157 calls a second on average and about 3,472 at a peak of three times the average, and each call carries about 1,800 tokens against 200 for an item.¹

That is 4,500 times more tokens a day than the refresh.¹ A model that writes a 150-token ranked list at an illustrative 20 milliseconds per token needs 3,000 milliseconds, many times a 200 millisecond budget.¹ The budget gates only the per-request LLM work; retrieval always runs. Expensive, slow models belong per item, in batch; per request they must be small, cached or limited to a few candidates.

Step 3: In the request path the LLM reranks or generates candidates

When the budget allows, the LLM can work on the request itself, in two ways. As a reranker it takes the retrieved candidates and orders them. Hou et al. formalise this as a conditional ranking task, with the interaction history as the condition and items from other candidate generation models as the candidates. They find promising zero-shot ranking, but the model struggles to perceive the order of past interactions and "can be biased by popularity or item positions in the prompts"; they report the issues can be alleviated.

As a generative retriever it replaces the nearest-neighbour lookup. Rajput et al. give each item a Semantic ID and train a Transformer-based sequence-to-sequence model, which they call Transformer Index for GEnerative Recommenders (TIGER), to generate the Semantic ID of the next item a user will interact with. They report that it also improves retrieval for items with no prior interaction history. A cross-encoder reranker is the lighter classical cousin of the LLM reranker, and a multi-task ranking model is what the LLM must beat on the objectives you care about. Retrieval stays.

Step 4: A chat layer converses and explains

The fourth role faces the user. In conversational recommendation the user types what they want, the LLM turns it into a query or a filter, the ordinary retrieval stack returns items, and the LLM writes the reply. The survey names user interaction as one of the stages where an LLM can play a role (Lin et al.). A small LLM can also rewrite a typed query into intent tags before retrieval, gated by the same budget rule.

The same layer can write a short reason for each recommendation. The rule that keeps it safe is that the model chooses among items the retrieval stack returned and does not name items outside that set. Check every identifier it mentions against the catalogue, and drop or regenerate any that does not exist. For a chat product that suggests starting questions, the same pattern applies: a batch job generates candidate questions per user segment, a classical model ranks them and the LLM runs only when the user asks for something new.

1 · Enrich in batch2 · Check the budget3 · Retrieve and rerank4 · Chat and explainReplay

Step-by-step description

1. The LLM turns item text into stored features once per item.
2. Cost and latency gate only per-request LLM work.
3. Retrieval always runs; the budget picks the ranker.
4. A chat layer talks about retrieved items only.

• Item enrichment runs the LLM in batch, once per item.
• The budget gates only the per-request LLM work; retrieval always runs.
• A chat layer converses and explains using items that retrieval returned.

Worked example

The comparison below uses assumptions stated as illustrative: a catalogue of 10,000,000 items, 200 tokens of text per item, 2% of items refreshed a day, 100,000,000 requests a day, 50 candidates of 30 tokens each plus a 300-token history per request.¹ Nothing here is a measurement of a real system.

Batch enrichment: the one-off backfill processes 2,000,000,000 tokens, and the daily refresh 40,000,000 tokens, at about 2.31 calls a second.¹ Online reranking: each request carries 1,800 tokens, so the day needs 180,000,000,000 tokens, at about 1,157 calls a second on average and about 3,472 at peak.¹

The two ways to shrink the online line are fewer candidates (rerank 10, not 50) and cheaper calls (score candidates in one forward pass instead of generating text). Both trade ranking quality for cost, which is why the final design usually keeps the LLM in batch and uses a distilled or classical ranker online.

Text description of the diagram

• A daily batch refresh needs about 2.31 LLM calls per second in the illustrative example.
• Online reranking needs about 1,157 calls per second on average and about 3,472 at peak.

Trade-offs and edge cases

Each role has a different cost pattern and failure mode, and the right design is the cheapest set of roles that fixes your actual weakness. If the weakness is new items, use enrichment. If it is the quality of the final order on a short list, test a reranker. If it is a chat product, add the chat layer.

Text description of the diagram

• Item enrichment runs in batch, once per item.
• The reranker runs per request and can show position and popularity bias.
• The chat layer can name items that do not exist.

Edge cases follow. Enrichment text goes stale when an item changes, so refresh it when the metadata changes. Reranker bias is documented: position and popularity effects in the prompt can reorder the list, so shuffle candidate order during evaluation and compare against the classical ranker on the same candidates (Hou et al.). Generative retrieval can output invalid identifiers, so check each against the catalogue; it also needs a stable Semantic ID scheme, and adding many new items later means rebuilding or extending it.

An LLM that never sees the real-time click signal cannot replace a model trained on it, so keep fresh behavioural features in the classical ranker. The survey describes LLM roles at different pipeline stages, so the safe interview position is that the LLM helps the pipeline. For evaluation, judge the full system with an experiment on real traffic, because a better text understanding score does not prove a better click or watch outcome. The machine learning system design guide places this inside a full interview answer, and the Transformer overview covers the model family underneath.

How to answer it in an interview

The spoken structure: keep the pipeline, list the roles from cheapest to most expensive, say the cost rule, then say how you evaluate and guard it.

Common mistakes

• "The LLM replaces the recommender." (wrong) That drops retrieval, real-time behaviour signals and the latency budget. (right) "The LLM helps one or more stages of the pipeline; I keep retrieval and a fast ranker."
• "I call the LLM for every candidate of every request." (wrong) Calls scale with traffic times candidates. (right) "I run the heavy model per item in batch and keep the online model small or cached."
• "The LLM ranker is neutral." (wrong) Studies report position and popularity bias in prompts. (right) "I shuffle candidates in evaluation and compare to the classical ranker."
• "The model will only suggest real items." (wrong) A generator can name items that do not exist. (right) "I constrain it to retrieved items and validate every identifier against the catalogue."
• "Better text scores before launch prove success." (wrong) Better text understanding may not move clicks. (right) "I run an online experiment on the metric the product cares about."

Follow-up questions

Can an LLM replace the recommendation model?

Not safely in the usual large-scale setting: the survey of Lin et al. treats LLMs as additions across the pipeline. Generative retrieval such as TIGER replaces the nearest-neighbour step, not the whole system.

How do you use an LLM for the cold start problem?

Enrich the new item's text in batch and embed it, so ranking has content features before any clicks arrive. Combine this with the standard methods in the cold start guide.

How do you stop an LLM recommender from suggesting items that do not exist?

Let the model choose only among retrieved items, or generate identifiers and check each against the catalogue before showing it. Drop or regenerate any miss.

How would you recommend starting questions in an AI chat app?

Generate candidate questions in batch per user segment, rank them with a small model using click-through rate (CTR) history, and call the LLM live only for a fresh context. This keeps the per-request cost near zero.

Is an LLM ranker biased?

Yes in the setting studied: Hou et al. report that it can be biased by popularity or item positions in the prompt. Randomise candidate order when you evaluate it.

How would you cut the cost of an LLM reranker?

Rerank fewer candidates, score in one forward pass instead of generating text, cache results for repeated contexts, and move the work to batch where possible. The recommendation design guide covers the classical stages it competes with.

Sources

1. How Can Recommender Systems Benefit from Large Language Models: A Survey (arXiv 2306.05817 v6) — arXiv (Lin et al.), 9 Jul 2024
2. Large Language Models are Zero-Shot Rankers for Recommender Systems (arXiv 2305.08845 v2) — arXiv (Hou et al.), 24 Jan 2024
3. Recommender Systems with Generative Retrieval (arXiv 2305.05065 v3) — arXiv (Rajput et al.), 3 Nov 2023