GenAI system design interview walkthrough: design a Perplexity answer engine

In this article

1. Before you start
2. Why interviewers ask this
3. A Perplexity-style answer engine from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one worked problem in the generative AI system design interview guide. It describes a design you can defend in a round. It does not describe Perplexity's internal system beyond what Perplexity publishes about itself.

Why interviewers ask this

A search-plus-answer product forces every RAG decision into one design, so the prompt "design Perplexity" tests whether you can reason about a whole pipeline instead of one component. Candidates report this prompt, and close variants such as designing a real-time search engine built on a large language model (LLM) or a system that can answer about an event from the last half hour, in interviews at AI labs and large tech companies.

A weak answer says "embed the documents, find the nearest ones, give them to the model." A strong answer separates fresh data from the static index, rewrites the query, diversifies and reranks, sizes the context from a token budget and checks the citations. A Staff-level answer adds what fails: near-duplicate chunks, a rewrite that drifts, a model that misreads a source, and a latency budget that every stage spends from.

A Perplexity-style answer engine from first principles

Answer engine

An answer engine is a search system that returns a written answer with citations to its sources instead of a list of links, by retrieving passages first and then asking a language model to compose the answer only from them.

Context budget

A context budget is the number of tokens of retrieved text you allow into one model request, fixed in advance from the context window, the prompt overhead and the room reserved for the answer.

Maximal marginal relevance

Maximal marginal relevance (MMR) is a selection rule that picks results one at a time, scoring each candidate by its relevance to the query minus its similarity to results already chosen, so near-duplicates lose to new information.

Requirements and scale

State requirements before you draw boxes. Functional requirements: the user asks a question, can ask a follow-up, gets a written answer with a numbered citation for each claim, and gets sources from the last hour when the question is about recent events. Non-functional requirements: a time to first token, the wait before the first answer word appears, with an illustrative target of about 1,500 milliseconds, and a cost per query set mainly by the input tokens sent to the model.

Scale sets the index size. With 100,000,000 pages, 20 chunks per page and an embedding of 768 numbers at 4 bytes each, the index holds 2,000,000,000 chunks and about 6.14 terabytes of raw vectors, or 1.54 terabytes with 8-bit numbers, before the text itself and any search structure.¹ Every input here is illustrative; in a round, ask the interviewer for the real ones.

In plain words

Think of a research assistant with a library card. You ask a question. The assistant does not answer from memory alone. They look up the freshest sources, pick a few that do not repeat each other, read them, and write a short reply with footnotes. The analogy breaks in one place: a language model can still misread a source it was given, so the footnotes need checking.

In one precise sentence, an answer engine runs retrieval-augmented generation (RAG), a method from Lewis et al. (2020) that pairs a language model with a searchable store of documents, so answers can use knowledge that was not in the training data.

Step 1: The freshness path starts with a crawl at ingestion

A static index cannot answer a question about something that happened half an hour ago, so the design needs a freshness path. The ingestion path is the part of the system that runs ahead of requests and keeps the index current: a crawler keeps fetching pages, with a short revisit interval for sites that change often, and writes them to the index as chunks with their embeddings and the fetch time.

Input: web pages. Output: an index whose chunks are each stamped with when they were fetched. The stamp matters later, because the answer can show it and the ranker can use it. A crawl alone still lags the web by its revisit interval, so Step 3 adds a live search at request time. You now know where freshness starts: a separate, slower-to-build path, not a property of the model.

Step 2: The question is rewritten before it is searched

A raw user message is a poor search query. It may be a follow-up ("what about its price?") that only makes sense with the earlier turns, so the first model call rewrites it into a standalone query. The search stage can then run keyword search and vector search on the rewrite and merge both lists.

One known trick is Hypothetical Document Embeddings (HyDE): a model writes an imagined answer, and the system searches with that text's embedding. Gao et al. (2022) note that the imagined document captures relevance patterns but is unreal and may contain false details. HyDE itself samples several hypothetical documents and averages their embeddings, and can include the query's own embedding in that average (Gao et al., 2022, section three point two). For news after the model's training cutoff all the imagined answers can still be confidently wrong, so keep the raw query in the mix.

Input: the user turn and history. Output: one or more standalone queries. You now know the engine searches on a rewrite, not on what the user typed.

Step 3: Retrieval, reranking and diversity pick the passages

When a time-sensitivity classifier flags the rewritten query as being about recent events, the system also queries an external live search source and merges those fresh results with the index hits before reranking. The merged candidate list is large and noisy, so a reranker, a slower model that reads the query and one passage together and scores them, reorders the top few dozen. A reranker is too slow to run over the whole index, which is why it comes after the cheap retrieval.

The next problem is repetition. Ten news sites may carry the same wire story, and top-k, the rule that keeps the k highest-scoring items, then fills the window with copies. Maximal marginal relevance, introduced by Carbonell and Goldstein (1998), trades relevance against similarity to what is already picked. A cheaper fix is to cluster near-identical chunks and keep one.

Input: candidate chunks from the index and, when flagged, from live search. Output: a short, varied, ordered list. You now know the list is chosen for coverage, not only for score.

Step 4: The context budget decides how many passages the model reads

The number of passages comes from a token budget, not from a habit such as "top-5". Take the context window, subtract the system prompt and question, subtract the room for the answer, and divide what is left by the chunk size. A longer context also costs more per request and is not used evenly: Liu et al. (2023) find that performance can degrade significantly when the relevant information moves, and is often highest when it sits at the beginning or end of the input.

So put the strongest passages first and last, and drop the weakest rather than pad. Input: ranked passages and the budget. Output: the passages that go into the prompt. You now know why a bigger top-k is not automatically better: it spends budget and buries evidence.

Step 5: The model answers from numbered passages with citations

Each passage gets a number in the prompt, and the model is told to answer only from those passages and to cite the number after each claim. The reply is then checked: each cited number must exist, and where affordable, a second pass compares each claim with its passage and flags unsupported ones.

Citations make an answer inspectable, but they do not prove it is right. The model can attach a correct number to a misread sentence, so the product shows the source and fetch time and lets the user click through. Input: labelled passages. Output: a cited answer plus the source list. When the passages do not contain the answer, the right output is "the sources do not say", not a guess. For a design where the model chooses its own searches, see agentic RAG.

1 · Ingest2 · Rewrite3 · Retrieve and rerank4 · Context budget5 · Cited answerReplay

Step-by-step description

1. A crawler feeds the index ahead of requests.
2. The question becomes a standalone query.
3. Index and live search results are merged and reranked.
4. Tokens decide how many passages fit.
5. The model answers from numbered passages.

• A crawler keeps the index fresh at ingestion.
• The question is rewritten before it is searched.
• Live search results are merged with index hits before reranking.
• The context budget decides how many passages the model reads.
• The model answers from numbered passages with citations.

Worked example

This example sizes the context and shows diversity at work. Every input is illustrative: a window of 8,000 tokens, 250 tokens per chunk, 400 tokens for the system prompt and question, and 500 tokens reserved for the answer.¹

With those inputs, top-5 puts 1,250 tokens of passages in the prompt, 1,650 input tokens in all; with the 500-token answer reserve that is 2,150 tokens, or 26.9% of the window. Top-20 uses 5,000 passage tokens and 5,400 input tokens; with the reserve, 5,900 tokens, 73.8% of the window, and about 3.27 times¹ the input of top-5.

The largest k that fits is (8,000 − 400 − 500) / 250 = 28.4, rounded down to 28: top-28 uses 7,900 tokens with the reserve, 98.8% of the window. Top-40 would need 10,400 input tokens, 10,900 with the reserve, 136.2% of the window, so it does not fit at all.¹ The budget therefore caps k at 28 in this setup, and the cost and position effects argue for going lower.

For diversity, the toy uses five candidates described by angles from the query, so cosine similarity, a score from −1 to 1 for how closely two embeddings point the same way, is easy to compute.

Candidates A, B and C sit close to the query and to each other, so by relevance alone the top two are A (0.9848) and B (0.9781), with a similarity of 0.9994 between them: the same passage twice. With MMR weighted 0.3 on relevance, a lower weight than the 0.5 to 0.7 often used, chosen here only to make the effect visible, the second pick is candidate E, whose relevance is 0.4226 but whose similarity to A is only 0.5736.¹ That also shows the cost: too low a relevance weight admits a weak passage, so the weight is tuned, not set once.

The latency figures are illustrative stage times of 300, 150, 200 and 500 milliseconds that sum to 1,150 milliseconds before the first answer token appears.¹

Text description of the diagram

• Top-5 uses 26.9% of the window.
• Top-28 uses 98.8% of the window, the largest k that fits.
• Top-40 needs 136.2% of the window, so it does not fit.

Trade-offs and edge cases

Freshness against latency and cost. A live-search fallback finds new events but adds a network call and a source you do not control. Decide when to call it: a classifier on the rewritten query can flag time-sensitive questions, and everything else uses the index.

Recall against noise. A larger top-k raises the chance the right passage is present, and adds distractors, tokens and cost. Tune it on a labelled set of questions, measuring answer quality, not only retrieval hits.

HyDE helps and hurts. It can lift retrieval when the question is vague and the model knows the topic. It can hurt when the topic is new to the model, because the imagined text pulls the search toward its own errors. HyDE itself averages the embeddings of several sampled hypothetical documents with the query embedding (Gao et al., 2022, section three point two), which in effect dampens one bad imagined answer. Going further, skipping HyDE for recent-event questions or pairing it with keyword search is a design choice, not a result from the paper.

Evaluation and caching. Evaluate on a held-out set of labelled questions, scoring citation precision (does each cited passage support its claim), answer correctness and freshness hit rate (the share of recent-event questions answered from a source fetched within the allowed age). Caching repeated queries saves the cost and latency of the whole pipeline but conflicts with freshness, so give entries for time-sensitive queries a short time to live, or skip the cache for them.

Reliability. Show sources and fetch times, refuse when the passages are silent, and treat web pages as untrusted text: a page can contain instructions aimed at the model, so the prompt must mark passages as data. Reported practice differs on how much automatic claim checking is worth its latency, so state it as a choice. The LLM gateway questions cover the serving layer around this pipeline.

How to answer it in an interview

The spoken structure: name the pipeline, say where freshness comes from, give the budget rule, then end on the trade-off. In the round, clarify requirements first (which questions, what freshness, what latency), sketch the five stages, go deep on one, then close with trade-offs and failure modes. The order below keeps you from spending the round on one box.

Text description of the diagram

• Clarify requirements first.
• Sketch the pipeline, then go deep on one stage.
• Close with trade-offs and failure modes.

Common mistakes

• "I embed the documents and take the top results." (wrong) That is search, not an answer engine, and it ignores freshness and the context limit. (right) "Index with a fresh path, rewrite, rerank, fit a token budget, then cite."
• "A bigger top-k is always safer." (wrong) The window fills, cost rises and position effects bury evidence. (right) "I derive k from the token budget and test answer quality."
• "Similarity search returns varied results." (wrong) It often returns near-duplicates. (right) "I diversify with maximal marginal relevance or by clustering duplicates."
• "Citations mean the answer is correct." (wrong) The model can misread a cited source. (right) "Citations make it checkable; I verify claims against passages and refuse when sources are silent."

Follow-up questions

How does Perplexity AI work (architecture)?

Perplexity's own help centre says it searches the internet and that each answer includes numbered citations linking to the original sources (Perplexity Help Center, How does Perplexity work?, accessed October 2026). Its models, index and ranking are not documented in the sources this page cites, so treat any internal detail as inference. The pipeline above is a defensible design of that pattern: ingest and crawl, rewrite the question, retrieve from the index plus live search, rerank and diversify, fit a token budget, then answer with citations.

How do you choose top-k and the context budget for RAG, given recall, noise, and cost all pull against each other?

Start from the token budget (window minus prompt minus answer room), divide by chunk size for a ceiling, then tune k downward on a labelled question set until answer quality stops improving. Put the best passages first and last.

When does HyDE hurt retrieval, and what variants fix its failure modes?

HyDE hurts when the model does not know the topic, since the imagined document may contain false details. Named variants and fixes: sample several hypothetical documents and average their embeddings with the query's own, as HyDE does (Gao et al., 2022, section three point two); search with the raw query as well; pair it with keyword search; or skip it for recent-event questions.

Your RAG keeps retrieving near-duplicate chunks. How do you diversify results?

Apply maximal marginal relevance, or cluster near-identical chunks and keep one per cluster. In the worked example the similarity between the first two results drops from 0.9994 to 0.5736, at the cost of a weaker second passage.

How do you answer questions about events after the model's training cutoff while staying reliable and transparent?

Retrieve fresh sources at request time through the crawler and live-search path, answer only from them, and show each source with its fetch time. If no source covers the event, say so. The generative AI system design guide places this inside the wider set of design problems.

Sources

1. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv 2005.11401) — arXiv (Lewis et al.), 22 May 2020
2. Precise Zero-Shot Dense Retrieval without Relevance Labels (arXiv 2212.10496) — arXiv (Gao et al.), 20 Dec 2022
3. Lost in the Middle: How Language Models Use Long Contexts (arXiv 2307.03172) — arXiv (Liu et al.), 6 Jul 2023
4. How does Perplexity work? (Perplexity Help Center) — Perplexity, 26 Oct 2026
5. The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries — ACM SIGIR (Carbonell and Goldstein), 24 Aug 1998