How do you scale an AI search assistant to 10M articles?

In this article

1. Before you start
2. Why interviewers ask this
3. Scaling an AI search assistant from first principles
4. The Staff-level answer
5. The rubric
6. Three variants of this question
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This is a question from the machine learning (ML) system design interview.

Why interviewers ask this

A design question that opens with "ten million articles" tests whether you size the problem before you draw boxes. Candidates report this kind of question in machine-learning system-design rounds; no public source says which company asks it. Skipping the sizing produces a design that fails on memory, on tail latency or on stale content.

A weak answer says "put the articles in a vector database and call a big model". A strong answer separates retrieval from generation and says why. A Staff-level answer puts numbers on the index, names what breaks when articles are edited or deleted, and measures the retrieval stage apart from the final answer.

Scaling an AI search assistant from first principles

Tombstone

A tombstone is a marker that says a stored item is deleted, so searches skip it even though its vector still sits in the index until the next rebuild. It is a design technique, not a built-in feature of every index.

In plain words

Picture a huge library. Nobody reads every book to answer your question. A catalogue points to the few shelves that matter, several branches each hold part of the stock, and a head librarian picks the best few pages and reads them to you. The analogy breaks in one place: a real catalogue is exact, while the index here may miss a true match in exchange for speed.

Retrieval-augmented generation (RAG) is the pattern behind this. Lewis et al. (2020) pair a pre-trained generator with a dense vector index of Wikipedia reached by a neural retriever, because models struggle to give provenance and to update their knowledge. RAG is a design in which the language model reads retrieved text instead of relying on what it memorised. For a website, a new article becomes an index update, not a retraining run.

1 · Count chunks2 · Compress3 · Shard the index4 · Re-rank, then writeReplay

Step-by-step description

1. Articles are split into chunks, and each chunk becomes a vector.
2. Compressed codes shrink each vector so the index fits in memory.
3. Each shard holds a share of the vectors and answers every query.
4. A re-ranker keeps a few passages for the language model.

• Chunks, not articles, set the number of vectors in the index.
• Compressed codes shrink each vector so the index fits in memory.
• A shard holds a share of the vectors and answers every query.
• A re-ranker keeps a few passages for the language model.

Step 1: Count chunks, not articles, to size the index

A vector index holds one vector per chunk. Chunks, not articles, set the number of vectors in the index. Assume 10,000,000 articles, as the question gives, eight chunks each and 768-dimensional 32-bit floats; the chunk count and vector size are illustrative. That gives 80,000,000 vectors of 3,072 bytes, or 245.76 GB raw, against 30.72 GB if you count articles by mistake.¹

Exact search compares the query with every vector; a vector-index tutorial describes it as perfect search quality at the cost of slow search. Approximate nearest-neighbour (ANN) search checks only part of the vectors and may miss a true neighbour in exchange for speed. ANN search is the only practical choice at this size.

Hierarchical Navigable Small World (HNSW) graphs are a common ANN choice. HNSW is a layered graph of nearby vectors: search starts at the sparse top layer and descends, which Malkov and Yashunin report allows logarithmic complexity scaling. You now know the size and the search method; the next question is the memory bill.

Step 2: Compress the vectors

Compressed codes shrink each vector so the index fits in memory. The Faiss guide says all Faiss indexes are stored in random-access memory (RAM), so memory sets the design. It gives HNSW a cost of d * 4 + M * 2 * 4 bytes per vector, where d is the dimension and M the links per node: 3,328 bytes at M of 32, or 266.24 GB for 80,000,000 vectors.¹

Product quantisation (PQ) splits each vector into pieces and stores a short code for each, trading recall, the share of the true nearest neighbours the search returns, for memory. The guide writes the code length as M; this page calls it m, a different parameter from the HNSW link count. With m of 64 (64-byte codes) PQ needs 5.12 GB before overhead. The guide says RaBitQ, which keeps one bit per dimension plus a small overhead, takes d / 8 + 8 bytes per vector, or 8.32 GB.¹ OpenSearch notes that real indexes use more than the raw code size, so measure memory and recall.

An inverted-file (IVF) index partitions the vectors into clusters, so a query scans only the nearest few (set by nprobe). For 10 million to 100 million vectors the guide recommends IVF262144_HNSW32, which finds cluster centres with HNSW. The scale design is IVF with PQ codes.

Text description of the diagram

• HNSW with M of 32 needs 266.24 GB for 80,000,000 vectors.
• Product quantisation with 64-byte codes needs 5.12 GB before overhead.

Step 3: Shard the index

A shard holds a share of the vectors and answers every query. With an illustrative budget of 64 GB of index memory per machine, the HNSW index needs five shards, or ten machines with two copies of each shard (one replica). The 5.12 GB of PQ codes fit the memory of one machine, so with IVF-PQ you shard for throughput and growth, not memory; the table below shows the uncompressed HNSW alternative.

Copies are sized by traffic: every shard sees every query, so the copies of the shard set together must carry the peak. At an illustrative 1,500 queries per second and 800 per copy, two copies are needed, and a third keeps the peak covered if one machine fails.¹ A cache for repeated queries lowers that load; it is a design choice.

Because every shard must answer, the slowest one sets the latency: if each shard has a 1% chance of a slow answer (illustrative), five shards slow about 4.9% of queries.¹

Step 4: Re-rank to a few passages before the model writes

A re-ranker keeps a few passages for the language model. A re-ranker is a slower model that reads the query and each candidate together and reorders them; the retrieval, ranking and re-ranking funnel explains why this stage follows a cheap one. Suppose the shards return 100 candidates, the re-ranker keeps 6 and each passage is about 300 tokens (all illustrative). The model reads 1,800 tokens instead of the 15,000 that 50 passages would cost, 8.33 times fewer.¹

Longer context is not a substitute. Liu et al. (2023) find that performance is often highest when relevant information sits at the beginning or end of the input, and significantly degrades when it sits in the middle of a long context, even for explicitly long-context models. Keep the passages few and cite them, so the answer carries provenance.

Many systems also run keyword search next to the vectors so exact product names and error codes match; that is a common design choice. A typeahead box in front of the assistant is a separate low-latency problem, covered in designing typeahead autocomplete.

The Staff-level answer

What makes it Staff-level: it starts from a computed size, names the failure of each stage and says how each stage is measured.

The rubric

Three variants of this question

What if the corpus grows tenfold?

The chunk count grows tenfold to 800,000,000 vectors, and the HNSW index becomes 2,662.4 GB, or 42 shards at 64 GB each.¹ The better move is to compress and partition: for 100 million to a billion vectors the Faiss guide recommends IVF1048576_HNSW32, with PQ codes, and 64-byte codes come to 51.2 GB before overhead, so quantisation stops being optional.

What if each reader may see only some articles?

Apply the permission filter during retrieval, so passages the reader cannot open never reach the model. Filtering after the search can leave too few results, so over-fetch or filter inside the index; this is design judgment, not sourced here.

What if the first token must arrive in under a second?

Give each stage a budget and check the sum. The numbers below are an illustrative assumption, not a measurement.¹

Common mistakes

• "Ten million articles means ten million vectors." (wrong) Each article is split into chunks. (right) "I size the index by chunks, so my assumption of eight chunks per article makes it eight times larger."
• "A long context window replaces retrieval." (wrong) Liu et al. (2023) find that performance significantly degrades when relevant evidence sits in the middle of a long context, even for explicitly long-context models. (right) "I retrieve, re-rank and pass a few passages."
• "Raising efSearch needs more memory." (wrong) A vector-index tutorial says efSearch (the size of the candidate list kept during search) and efConstruction (the same list during index build) leave the memory footprint unchanged. (right) "It costs latency; M and the vector size set memory."
• "I delete an edited article by removing its vector from the HNSW index." (wrong) The Faiss guide says HNSW does not support removing vectors. (right) "I mark it with a tombstone and rebuild on a schedule."

Follow-up questions

Why not just use a bigger context window instead of retrieval?

Long-context models still use the middle of the input poorly: Liu et al. (2023) find performance is often highest when relevant information sits at the beginning or end of the input. Retrieval also keeps cost and latency bounded and lets you cite the source article.

How do you handle edited and deleted articles at this scale?

Write new chunks into a small fresh index and hide stale ones with tombstones, because the Faiss guide says HNSW supports no removal. Merge by rebuilding shards on a schedule. This is design judgment rather than a sourced recipe.

Does retrieval-augmented generation stop the model from stating unsupported things?

Not by itself. Lewis et al. frame provenance and updatable knowledge as the motivation, not the removal of errors. Cite the passages, check that the answer follows them, and add output checks such as harmful-content detection.

How do you know the retrieval stage is working?

Keep a set of questions with known answer articles and measure how often the right chunk appears among the candidates, separately from the quality of the final text. Track drift by re-running the same question set on a schedule; the retrieval funnel shows where each stage can slip.

What changes if the articles contain images?

Image content needs its own embeddings in the same vector space, which adds a model and a service; see designing a multimodal embedding model and its service.

Sources

1. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv 2005.11401) — arXiv (Lewis et al.), 22 May 2020
2. Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs (arXiv 1603.09320) — arXiv (Malkov and Yashunin), 30 Mar 2016
3. Guidelines to choose an index — Faiss wiki (facebookresearch/faiss), 26 Oct 2026
4. Lost in the Middle: How Language Models Use Long Contexts (arXiv 2307.03172) — arXiv (Liu et al.), 6 Jul 2023
5. Nearest Neighbor Indexes for Similarity Search — Pinecone Learning Center, 26 Oct 2026
6. Faiss product quantization — OpenSearch documentation, 26 Oct 2026