What is a cross encoder reranker? How it works

In this article

1. Before you start
2. Why interviewers ask this
3. A cross-encoder reranker from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

Why interviewers ask this

A retrieval system that returns plausible but badly ordered passages looks fine in a demo and fails in production, so interviewers use the reranker to test whether you understand why one model cannot do both jobs. A weak answer says "the reranker is more accurate" and stops.

A strong answer says why it is more accurate (joint attention over the pair), why that makes it too slow to run on everything (no precomputation), and how the two stages are sized against a latency budget. The question fits retrieval-augmented generation (RAG) design rounds, often as "the retriever returns twenty chunks but only five fit in the prompt, now what?" It is the cleanest test of two-stage thinking in the machine learning system design loop.

A cross-encoder reranker from first principles

Cross-encoder

A cross-encoder is a transformer that takes a query and a passage as one joined input and outputs one relevance score, so every word of the query can attend to every word of the passage.

Bi-encoder

A bi-encoder is a model that encodes a query and a passage separately into vectors, so relevance is a cheap vector comparison and the passage vectors can be computed and stored before any query arrives.

In plain words

Think of a librarian and an examiner. The librarian glances at the spine of every book in the building and pulls out the hundred that look right. The examiner then sits with each of those hundred, reads it next to your question, and ranks them. The librarian is fast because she never reads a whole book. The examiner is slow because he reads every one against your question. The analogy breaks here. A real examiner could skip a clearly wrong book, but a cross-encoder spends a full pass on every pair, relevant or not, and a longer pair costs more.

In one precise sentence: a bi-encoder scores relevance from two separately computed vectors, a cross-encoder scores it from one joint pass over both texts, and a search system uses the first to find and the second to decide.

Step 1: Retrieve a candidate list with a bi-encoder

The first stage retrieves a few hundred or fewer candidates from the whole corpus with a bi-encoder and a vector index. Every passage was turned into a vector once, at indexing time. At query time the system encodes the query once and asks the index for its nearest neighbours. The sentence-transformers retrieve-and-re-rank guide (as of October 2026) uses about 100 candidates in its example.

This stage must handle collections of millions of passages, so it trades accuracy for speed and may return irrelevant items. The output is a list of passage identifiers with rough similarity scores, nothing more. What you now know: stage one is cheap per query because the expensive work was done earlier. How the index finds neighbours fast is the subject of the vector search hub.

Step 2: Pair the query with each candidate

The system builds one input per candidate by joining the query text to the candidate's text. With a shortlist of N candidates there are N pairs, and each pair is a separate input to the model. This step is trivial in code and decisive in cost, because the model will run once per pair. It also fixes the unit of work: the cost of reranking is the number of candidates times the cost of one pair, so doubling the shortlist doubles the bill. Nothing in this step uses a vector. The pair is raw text, tokenised together with a separator token between the two parts.

Step 3: Read the pair jointly, in one pass

The cross-encoder runs one transformer pass over the whole pair, so attention can connect words in the query with words in the passage. This is the property that makes it accurate and the property that makes it expensive. A bi-encoder's passage vector is computed without ever seeing a query, so it can be stored. A cross-encoder's internal state depends on both texts at once, so no part of it can be computed before the query arrives.

The cost of this design shows in numbers: finding the most similar pair among 10,000 sentences with a joint Bidirectional Encoder Representations from Transformers (BERT) model needs about 50 million inference computations, roughly 65 hours, against about 5 seconds with separately computed embeddings, as the Sentence-BERT paper reports (Reimers and Gurevych, 2019). The arithmetic is simple: 10,000 sentences make 10,000 × 9,999 / 2 = 49,995,000 pairs.

Step 4: Turn the output into a score

A small output layer on top of the transformer converts the joint reading into one number per pair. In the MS MARCO model we ran with the sentence-transformers library, the score is a raw value that is higher for more relevant pairs; across our eight example passages it ranged from −8.65 to 4.22. Some documentation describes scores squashed to between 0 and 1, but either way the scale belongs to the model, so scores are only meaningful for comparing candidates of the same query. Nogueira and Cho (2019) trained this kind of setup, a BERT model that scores query-passage pairs.

Their abstract reports the top entry on the MS MARCO passage retrieval leaderboard, outperforming the previous state of the art by 27% (relative) in mean reciprocal rank (MRR) at ten results, written MRR@10, which averages one over the rank of the first correct answer in the top ten. The model is trained on labelled relevant and irrelevant pairs, so its quality depends on how close your text is to that training data.

Step 5: Sort and cut to the top few

The pipeline sorts the candidates by score and keeps the top few, typically five, for the user or for the language model's prompt. Only the top 5 re-ranked passages are shown or passed on. This final cut matters for two reasons. First, the prompt has limited room, so only the best few can go in.

Second, language models use information in the middle of a long input less reliably than information at the start or end (Liu et al., 2023), so the order of what you keep also matters. A reranker therefore improves answers twice: it picks better passages and it lets you put fewer, better ones in front of the model. What it cannot do is recover a passage the first stage never returned.

1 · Retrieve candidates2 · Pair them with the query3 · Read each pair jointly4 · Score5 · Sort and cutReplay

Step-by-step description

1. The bi-encoder embeds the query and the index returns about 100 nearest passages.
2. Each candidate is joined to the query text, giving one pair per candidate.
3. The cross-encoder reads query and passage in one pass; nothing here can be precomputed.
4. The output head turns the joint reading into one relevance score per pair.
5. Sort by score and keep the top 5 for the user or the language model.

• A first-stage retriever returns about 100 candidates in the sentence-transformers example.
• The cross-encoder scores one query-passage pair per candidate.
• Only the top 5 re-ranked passages are shown or passed on.

Worked example

We ran a real bi-encoder (all-MiniLM-L6-v2) and a real cross-encoder (ms-marco-MiniLM-L6-v2, 22.7 million parameters per its Hugging Face page) on eight passages we wrote ourselves for one question: can a person with a peanut allergy safely eat food made in a factory that also processes tree nuts? This is an illustration, not a benchmark.²

Both rankers put the shared-equipment advice passage first, the one that advises reading labels because shared equipment can leave traces. The difference is in the middle. The warning-label passage, about food "made in a facility that also processes tree nuts", is rank 4 for the bi-encoder and rank 2 for the cross-encoder, so the cross-encoder lifted it from fourth to second. The tree-nut allergy definition drops from rank 2 to rank 4.

Now the cost. The model card (as of October 2026) reports a throughput of 1,800 query-passage pairs per second on a V100 graphics processing unit (GPU), a score of 74.30 on normalized discounted cumulative gain (NDCG) at ten results on the Text REtrieval Conference (TREC) Deep Learning 2019 track, and an MRR@10 of 39.01 on the MS MARCO dev set. From that throughput we computed the best-case time per query.¹

Reranking 20 candidates takes about 11 milliseconds, 100 candidates about 56 milliseconds and 400 candidates about 222 milliseconds, so at that rate a latency budget of 200 milliseconds allows about 360 candidates. Treat that as a ceiling, not a plan: throughput counts pairs per second while the graphics card is kept full, which is not the wait for one query. The model card does not say what text it timed; its scores are on MS MARCO passages, which average 56 words (Thakur et al., 2021, Table 1), and a longer pair costs more, so longer chunks allow fewer.

When Thakur et al. timed this same model reranking the top 100 results of a keyword search over entries that average 50 words, one query took about 450 milliseconds on one V100, about eight times our 56 (Thakur et al., 2021, Tables 1 and 3). Time the reranker on your own hardware before you fix the shortlist size. Scoring a corpus of 1,000,000 passages would take 556 seconds, about 9.3 minutes, for one query.

Text description of the diagram

• At 1,800 pairs per second, 100 candidates take about 56 milliseconds.
• At that rate a 200-millisecond budget allows at most about 360 candidates.
• Scoring 1,000,000 passages would take about 556 seconds, roughly 9.3 minutes.

Trade-offs and edge cases

The reranker is bounded by the first stage: if the right passage is not among the candidates, the reranker cannot find it. Raising the candidate count raises the ceiling and the cost together, so teams measure recall of the first stage at the candidate count they can afford before tuning the reranker at all. Reranking and late-interaction models "on average achieve the best zero-shot performances" on the Benchmarking Information Retrieval (BEIR) benchmark of 18 datasets but "at high computational costs" (Thakur et al., 2021), which is the whole trade-off in one sentence.

Late interaction is the middle path. ColBERT stores one vector per passage token before queries arrive and matches them at query time, which its authors report is competitive in effectiveness with existing BERT-based models while two orders of magnitude faster and using four orders of magnitude fewer floating-point operations per query (Khattab and Zaharia, 2020). At query time each query token takes its best-matching passage token vector, and the matches are summed into the score (ColBERT calls this MaxSim). It gives up joint attention inside the model, which is why its typical job sits between finding and reranking.

Text description of the diagram

• A bi-encoder encodes query and passage separately and can store passage vectors in advance.
• A cross-encoder reads the pair jointly and stores nothing per passage.
• ColBERT keeps per-token passage vectors stored in advance and compares them at query time.

Two further edge cases bite in production. Long passages can be cut off at the model's input limit, so the part that answers the question may never be read; chunk first, then rerank. And a reranker trained on one kind of text can rank worse than its own first stage on another. In the BEIR study, a MiniLM cross-encoder trained on MS MARCO reranked the top 100 results of Best Matching 25 (BM25), a keyword-matching ranking function, and beat BM25 alone on 16 of 18 datasets, including all three biomedical ones, but scored lower on the two argument-retrieval sets, ArguAna and Touché-2020 (Thakur et al., 2021). Keep a held-out set of real queries and measure before and after.

How to answer it in an interview

Say it in this order: the problem, why two stages, the cost reason, then the failure mode.

Common mistakes

• "The cross-encoder is just a better embedding model" (wrong). It produces no embedding at all. It scores a pair, so you cannot index its output. (Right: "it outputs a score for a pair, so it can only reorder candidates.")
• "Rerank the whole corpus for best quality" (wrong). At 1,800 pairs per second a million passages take minutes per query. (Right: "retrieve a shortlist with a bi-encoder, then rerank it.")
• "A reranker fixes poor retrieval" (wrong). It reorders what it is given. (Right: "I'd measure recall at the candidate count first.")
• "Scores are probabilities I can threshold at 0.5" (wrong). In the run above the scores ran from −8.65 to 4.22. (Right: "scores compare candidates within one query; I'd calibrate a threshold on labelled data.")

Follow-up questions

What is reranking in an LLM pipeline?

Reranking is the step between retrieval and generation that reorders retrieved passages by a more accurate relevance score so the language model sees the best few first. It is the second stage of the pipeline in Step 5 above.

Why is reranking important?

Retrieval by vector similarity is fast but approximate, and the prompt has room for only a few passages. Reranking spends extra compute on a shortlist so those few passages are the right ones, and the order matters because models use the middle of a long context less reliably.

Why can't a cross-encoder precompute anything?

Its output depends on the query and passage jointly, so there is nothing query-independent to store. A bi-encoder's passage vector does not depend on the query, which is why it can be stored.

How many candidates should you rerank?

Enough that first-stage recall is high, and few enough to fit the latency budget. At the model card's 1,800 pairs per second, a 200-millisecond budget allows at most about 360 pairs on one V100 GPU, but a measured run of the same model took about 450 milliseconds to rerank 100 (Thakur et al., 2021, Table 3), so time it on your own hardware. Measure recall at 20, 50 and 100 before choosing.

Should I use a reranker or a bigger context window?

A bigger window lets you include more passages but does not reorder them, and models use the middle of long inputs less reliably (Liu et al., 2023). Reranking keeps the prompt short and ordered, so the two are complements; compare cost per query and answer quality on your own questions. For the neighbouring decoding and prompt-length questions see large language model (LLM) inference and the AI engineer question list.

Sources

1. Passage Re-ranking with BERT — arXiv (Nogueira and Cho), v5 revised 2020-04-14, 13 Jan 2019
2. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks — arXiv (Reimers and Gurevych), 27 Aug 2019
3. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT — arXiv (Khattab and Zaharia), SIGIR 2020, 27 Apr 2020
4. BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models — arXiv (Thakur et al.), v4 revised 2021-10-21, 17 Apr 2021
5. Retrieve and Re-Rank — Sentence Transformers documentation, 24 Oct 2026
6. cross-encoder/ms-marco-MiniLM-L6-v2 model card — Hugging Face, 24 Oct 2026
7. Lost in the Middle: How Language Models Use Long Contexts — arXiv (Liu et al.), v3 revised 2023-11-20, 6 Jul 2023