Embeddings & Vector Search

How do bi-encoders and cross-encoders differ in architecture and use cases?

By Interview Clozo · Published 27 October 2026 · Last verified 26 October 2026 · 12 min read

How do bi-encoders and cross-encoders differ in architecture and use cases? A bi-encoder encodes the query and each document separately into vectors, while a cross-encoder reads both together and outputs one relevance score. So bi-encoders search millions of documents and cross-encoders rerank a short list. The follow-up that separates levels is why a cross-encoder cannot replace the retriever: about 50 million pair scores for 10,000 sentences.

In this article

Before you start
Why interviewers ask this
Bi-encoder vs cross-encoder from first principles
The Staff-level answer
The rubric
Three variants of this question
Common mistakes
Follow-up questions
Sources

Before you start

This page is one step of the embeddings and vector search learning path.

Why interviewers ask this

A candidate who mixes up the two designs proposes a search system that is either too slow or not accurate enough. That is the cost of the mistake, and it is why the question appears in retrieval and recommendation design rounds.

The question separates people who know the vocabulary from people who know where the compute goes. A weak answer says "one makes embeddings, the other makes scores." A strong answer says where the two texts meet: outside the network for a bi-encoder, inside it for a cross-encoder. That single fact explains the speed, the accuracy and the production pattern. A Staff-level answer adds the recall ceiling of the first stage, a rough cost estimate, and middle-ground designs such as the poly-encoder and late interaction.

Bi-encoder vs cross-encoder from first principles

Retrieve-and-rerank pipeline: A retrieve-and-rerank pipeline is a two-stage search design in which a fast first stage, usually a bi-encoder or keyword search, pulls a shortlist of candidate documents from a large collection, and a slower second stage, usually a cross-encoder, rescores only that shortlist to produce the final order.
Poly-encoder: A poly-encoder is an architecture that encodes the candidate into one vector and compresses the context into a few learned code vectors through attention, then lets the candidate attend over those codes, so candidate vectors can still be precomputed. Humeau et al. report it as faster than a cross-encoder and more accurate than a bi-encoder.
Late interaction: Late interaction is a scoring design in which query and document are encoded separately into one vector per token, and the score adds up, for each query token, its highest similarity to any document token (an operation called MaxSim). Document token vectors are precomputed, as with a bi-encoder, but the index is larger. ColBERT, from Khattab and Zaharia (2020), is the standard example.

In plain words

Picture a library. A bi-encoder is a librarian who writes one catalogue card per book, once. When you ask for something, she writes one card for your request and compares cards. A cross-encoder is a librarian who sits down with your request and one book, reads them side by side, and gives a mark. Her mark is better. But she must repeat this for every book.

The analogy breaks in one place. Real catalogue cards are lists of numbers learned by a network, not hand-written summaries.

Now the precise version. A bi-encoder is a model that encodes two texts independently into two vectors so a simple formula can compare them. A cross-encoder is a model that reads two texts together in one pass and outputs a single score for the pair. Both are typically built from a transformer such as Bidirectional Encoder Representations from Transformers (BERT), so the difference is wiring, not model family (Sentence Transformers documentation).

Step 1: A bi-encoder encodes each text on its own

A bi-encoder sends the query and the document through the network separately. The cross-encoder documentation describes passing sentences A and B independently, which gives sentence embeddings u and v, and then comparing them with cosine similarity (many systems use a dot product instead). Reimers and Gurevych (2019) build this with siamese and triplet network structures, meaning the same network is applied to each side.

Cosine similarity is a score of how closely two vectors point the same way, from -1 to 1. Input: one text. Output: one fixed-size vector. The query and the document never see each other inside the network. You now know the key property: a document's vector does not depend on the query. Training is covered in the two-tower model page, where the two sides may share weights or keep separate ones.

Step 2: A cross-encoder reads both texts together, so every pair costs a pass

A cross-encoder passes the query and the document into the transformer at the same time. Inside the network, attention, the operation that lets every token weigh every other token, runs across both texts. The retrieve-and-rerank documentation says the network then outputs a single score between 0 and 1 for how relevant the document is to the query. That attention across both texts is why it is more accurate.

Input: one pair. Output: one score. It "does not produce a sentence embedding" (Sentence Transformers documentation), and you cannot feed it a single text. The price is the pair count. Clustering 10,000 sentences with a cross-encoder needs scores for about 50 million combinations and takes about 65 hours, per the same source. The exact count is 49,995,000 pairs.¹ The next step shows what that does to search.

Step 3: Only the bi-encoder lets you precompute the collection

A bi-encoder embeds every document once, ahead of time, and stores the vectors in an approximate nearest-neighbour index. An approximate nearest-neighbour index is a structure that checks only part of the collection and may miss a few true matches in exchange for speed. At query time only the query is encoded. The retrieve-and-rerank documentation says the retriever must be efficient for collections with millions of entries. The same 10,000 sentences take about 5 seconds with Sentence-BERT instead of about 65 hours (Reimers and Gurevych, 2019). For the index side, read similarity search.

The pass counts are the stable result; the wall-clock times are not. Spreading the 2019 figure of 65 hours over the 49,995,000 pairs implies about 4.7 milliseconds a pair,¹ a pessimistic figure for today. The list of pretrained cross-encoders reports about 1,800 documents per second for ms-marco-MiniLM-L-6-v2 on a V100 graphics card (as of October 2026), so 100 candidates take about 56 milliseconds.¹ The ratio is the same: 1,000,000 passes against 100.

On an illustrative collection of 1,000,000 documents, scoring every document needs 1,000,000 cross-encoder passes per query against 100 for reranking, which is 10,000 times fewer.
Quantity	Value
Cross-encoder pairs to cluster 10,000 sentences	49,995,000
Cross-encoder passes per query over every document (illustrative 1,000,000 documents)	1,000,000
Cross-encoder passes per query over 100 candidates	100
Ratio of passes, all documents against reranking	10,000
Implied cost of one pair, from the 2019 BERT figure	about 4.68 milliseconds
Time for 1,000,000 passes at that implied cost	about 4,680 seconds (78 minutes)
Time for 100 passes at that implied cost	about 468 milliseconds
Time for 100 passes at a small current reranker's documented throughput (as of October 2026)	about 56 milliseconds
Network passes per query, bi-encoder only	1 (plus the index lookup)
Raw vectors for 1,000,000 documents, 768 numbers each, 32-bit (illustrative)	3.072 GB

Illustrative: reranking needs 10,000 times fewer cross-encoder passes than scoring everything; the 101 and 1 bars are tiny, and passes differ in cost.CC BY 4.0 — source: How do bi-encoders and cross-encoders differ in architecture and use cases?

Text description of the diagram

A cross-encoder over every document of an illustrative 1,000,000-document collection needs 1,000,000 passes per query.
Retrieve then rerank with 100 candidates needs 101 passes: one query encoding and 100 cross-encoder passes.
A bi-encoder alone needs one pass per query plus an index lookup.

Step 4: Production systems retrieve first, then rerank

The standard pattern joins the two. The retrieve-and-rerank documentation describes a retrieval system that returns a list of, for example, 100 possible hits, which a cross-encoder then rescores. The shortlist can come from a bi-encoder, from keyword search, or from both, as covered in hybrid search. The reranking stage has its own page: cross-encoder rerankers.

Input: a query and a shortlist. Output: the same documents in a better order. The limit is recall. If the retriever never returns the right document, the reranker cannot rescue it. Middle-ground designs exist for cases where neither extreme fits: the poly-encoder and late interaction (ColBERT), both defined above.

The bi-encoder narrows a large collection to a shortlist, and the cross-encoder spends its extra cost only on that shortlist.CC BY 4.0 — source: How do bi-encoders and cross-encoders differ in architecture and use cases?

Step-by-step description

Query and document are encoded separately into vectors.
Query and document are read together and give one score.
Only bi-encoder vectors can be stored ahead of time.
The index returns candidates and the cross-encoder reorders them.

A bi-encoder encodes the query and each document separately.
A cross-encoder reads the query and a document together and outputs one score.
A bi-encoder embeds every document once, ahead of time, and stores the vectors in an approximate nearest-neighbour index.
The index returns a shortlist of candidates and the cross-encoder reranks them.

Knowing it is half the job. Saying it is the other half.

Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.