In this article
Before you start
The question names no case, so this page assumes training a two-tower model to retrieve passages for a query. It belongs to embeddings and vector search.
Why interviewers ask this
Picking examples decides what a retriever learns, so the question separates candidates who have trained one from candidates who only describe the architecture. Karpukhin et al. (2020) write that negative selection is often overlooked but could be decisive for learning a high-quality encoder, and a survey by Zhao et al. (2022) finds dense retrievers sensitive to the quality of negatives.
A weak answer says "the matching pair is positive, everything else is negative." A strong answer separates cheap in-batch negatives from mined hard negatives and says what each costs. A Staff-level answer adds the failure: a mined negative may be an unlabelled positive, and training on it teaches the model the wrong thing.
Choosing positive and negative examples from first principles
- Positive example
- A positive example is a query paired with an item that is known to be relevant to it, usually from a labelled dataset or from a behaviour such as a click, which a contrastive loss pulls closer together in the vector space.
- Hard negative
- A hard negative is an item the model currently scores as close to the query although it is labelled as not relevant, usually mined from the top results of a first-stage retriever, so it teaches the model more than a random item does.
- False negative
- A false negative is an item treated as a negative that is in fact relevant, usually because the labels are incomplete or a duplicate of the positive sits in the same batch, so training pushes it away when it should be pulled closer.
In plain words
Think of teaching a new librarian which books answer which questions. You hand over a question and the book that answers it, a positive. Then you hand over wrong books so the librarian learns the difference. Random wrong books are easy to reject; a tax book about the wrong year teaches more. The analogy breaks in one place: nobody has read every book, so the "wrong" book you picked might secretly also answer the question.
In one precise sentence, you pick positives from labels and negatives from items that look similar to the query but are not labelled relevant, and you check those negatives for hidden positives.
Step 1: Positives come from labelled pairs
Positives come from the training data itself. In question answering (QA), a positive is a passage known to hold the answer; in other settings it can be a click or a matching caption. A labelled pair supplies the positive. Input: a labelled pair. Output: one query and its positive; the data owner decides what counts as positive.
Everything else is only presumed irrelevant. Karpukhin et al. (2020) note that all other passages in the collection can be viewed as irrelevant by default. "Negative" therefore often means "not labelled positive", which is weaker than "truly irrelevant". The same paper reports that a retriever trained on only 1,000 examples already outperformed Best Matching 25 (BM25) (Best Match 25, a keyword-ranking formula). You now have the anchor and its positive; next come the negatives.
Step 2: Negatives come from the batch and from a retriever
An in-batch negative is another example's positive. Sentence Transformers documentation describes the in-batch negatives as all positives and hard negatives from other samples in the batch, and says the loss accepts plain pairs, so no triplets are needed. A batch of B pairs gives each query B minus one negatives, so 4 pairs give 3.1
In-batch negatives are mostly easy. Xiong et al. (2020) argue that uninformative negatives dominate mini-batch training, giving diminishing gradient norms and slow convergence. The fix is hard negatives: a retriever searches the corpus, and its top results that lack the answer become negatives. The method of Xiong et al., Approximate nearest neighbor Negative Contrastive Learning (ANCE), picks them from the whole corpus with an asynchronously refreshed approximate nearest-neighbour (ANN) index. Dense Passage Retrieval (DPR) found that its best model used gold passages from the same batch plus one BM25 negative.
Step 3: Filter false negatives out of the hard negatives
The common error is mining the top-ranked unlabelled passages and calling them negatives. Qu et al. (2020) examined the top-retrieved passages of 100 questions that were not labelled positive and found that about 70% were positives or highly relevant. Annotators label few passages, so the top of a ranking holds unlabelled positives. A filter removes suspected false negatives.
Training on them hurts. On MS MARCO (Microsoft Machine Reading Comprehension) benchmark, Qu et al. report a mean reciprocal rank at 10 (mean reciprocal rank (MRR)@10) of 32.39 with in-batch negatives and 33.32 with cross-batch negatives, which are negatives shared across graphics processing units (GPUs) so that each query sees many more of them. Adding hard negatives without filtering dropped it to 26.03, and adding filtered hard negatives raised it to 36.38; both hard-negative rows sit on top of cross-batch negatives. Their filter uses a well-trained cross-encoder, as Zhao et al. (2022) summarise. For duplicates inside a batch, the Sentence Transformers docs offer the NO_DUPLICATES batch sampler, which keeps duplicates of the anchor or positive out of the in-batch negatives.
Text description of the diagram
- MRR@10 is 32.39 with in-batch negatives and 33.32 with cross-batch negatives.
- Adding hard negatives without filtering dropped it to 26.03, and adding filtered hard negatives raised it to 36.38.
Step 4: The loss compares the positive with every negative
The positive and all its negatives go into one softmax, the InfoNCE loss, whose loss is the negative log of the probability the true item gets. Here is an illustrative toy: cosine similarities of the anchor to its positive and to three in-batch negatives are 0.80, 0.20, 0.10 and 0.30, and the temperature is 0.1. The loss is 0.0101, so the easy negatives leave almost nothing to learn.1
Add one hard negative at similarity 0.60 and the loss rises to 0.1358, a real signal.1 Now suppose that item is an unlabelled positive at 0.75: the loss becomes 0.4804, and the gradient pushes a relevant passage away from the query.1 At scale, DPR's best setting, with a batch of 128 and one BM25 negative per question (Karpukhin et al., 2020, §5.2), gives each question 127 in-batch gold negatives plus the batch's 128 BM25 negatives, 255 in all.1
Step-by-step description
- A labelled pair gives the positive.
- The batch and a retriever give negatives.
- A filter drops suspected false negatives.
- A softmax loss trains on all of them.
- A labelled pair supplies the positive.
- Other items in the batch supply free negatives, and a retriever supplies hard negatives.
- A retriever searches the corpus for hard negatives.
- A filter removes suspected false negatives.
- A softmax loss compares the positive with every negative.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →The Staff-level answer
How do you pick negative examples and positive examples for training a retriever?
I assume a two-tower text retriever. Positives come from labelled pairs, such as a passage known to answer the query. Negatives come in two kinds. Other items in the same batch are free negatives, and a bigger batch gives more of them, but they are mostly easy. So I add about one hard negative per query, mined with a keyword ranker or the current model. The risk is that top-ranked unlabelled passages are often relevant, so I filter them with a cross-encoder and remove duplicates of the positive from the batch. Then I check recall on held-out queries.
What makes it Staff-level: it names the assumption, separates cheap from informative negatives, and states the false-negative failure with its fix. Say it in this order: positives, in-batch negatives, hard negatives, the risk, the check.
The rubric
| Dimension | Weak answer | Strong answer | Staff answer |
|---|---|---|---|
| Correctness | Matching pair is positive, all else negative | Positives from labels; negatives from the batch and a retriever | Says negatives are only presumed irrelevant |
| Depth | Random negatives only | Adds hard negatives and why easy ones teach little | Explains hard-negative mining from an index refreshed during training |
| Trade-offs | None | Notes a bigger batch costs memory | Notes harder negatives raise the false-negative rate |
| Communication | Lists terms | States the case and the order | Opens with the assumption and ends with the check |
| Follow-up readiness | Stuck on "what if it is wrong" | Mentions filtering | Names a cross-encoder filter and duplicate masking |
Three variants of this question
What if you only have query and positive pairs, with no negatives?
Use an in-batch loss: the other positives in the batch become the negatives, so pairs are enough. Later, mine hard negatives with a first-stage retriever to upgrade the pairs to triplets, as the Sentence Transformers docs describe.
What if the in-batch negatives are too easy?
Mine hard negatives from the corpus with BM25 or with your own model through a similarity search index. Xiong et al. (2020) refresh an ANN index while the model trains so the negatives keep matching what the model currently confuses.
What if the retriever you mine from is the one you are training?
The negatives then follow the model's own mistakes, which is the idea behind ANCE. The cost is rebuilding the index as the model changes, and the false-negative risk stays, so keep the filter.
Common mistakes
- "Anything not labelled positive is a safe negative." (wrong) It is only presumed irrelevant. (right) "Unlabelled passages may be relevant, so I filter mined negatives."
- "Harder negatives are always better." (wrong) Added on top of cross-batch negatives, unfiltered hard negatives lowered MRR@10 from 33.32 to 26.03 in the RocketQA ablation. (right) "Hard negatives help once false negatives are filtered."
- "In-batch negatives are enough." (wrong) They are cheap but mostly uninformative. (right) "I start with in-batch negatives and add mined ones."
- "I need hand-labelled negatives." (wrong) In-batch negatives need no labels. (right) "A retriever mines the hard ones."
Follow-up questions
Do you need triplets to train with negatives?
No. A pairs-only dataset works with an in-batch loss because the other items in the batch supply negatives; triplets or n-tuples just let you add mined negatives.
How many hard negatives should you use per query?
There is no sourced rule. DPR's best model used one BM25 negative per question with a batch of 128, which is a reasonable starting point; tune on held-out recall.
How do you stop duplicates from becoming false negatives?
Use a batch sampler that keeps duplicates of the anchor or positive out of the same batch, such as NO_DUPLICATES in Sentence Transformers, and deduplicate the data first.
Why does a bigger batch help contrastive training?
More items per batch means more in-batch negatives per query and a stronger signal, at the cost of memory. See the two-tower page for ways to grow the batch.
How do you choose the temperature?
The temperature divides the similarity scores before the softmax, so a lower value sharpens it and makes hard negatives weigh more in the loss. There is no sourced universal value; tune it on held-out recall, and watch for false negatives, which a low temperature punishes harder.
- Positives come from labelled pairs; every other item is only presumed irrelevant.
- In-batch negatives are free and need no labels, but are mostly easy.
- Hard negatives, mined with BM25 or the model, teach more.
- Top-ranked unlabelled passages are often relevant, so filter them, for example with a cross-encoder.
- Check the result on held-out queries, not on training loss.
Sources
- Dense Passage Retrieval for Open-Domain Question Answering (arXiv 2004.04906) — arXiv (Karpukhin et al.; EMNLP 2020), 10 Apr 2020
- RocketQA: An Optimized Training Approach to Dense Passage Retrieval (arXiv 2010.08191) — arXiv (Qu et al.; NAACL 2021), 16 Oct 2020
- Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval (arXiv 2007.00808) — arXiv (Xiong et al.; ICLR 2021), 1 Jul 2020
- Sentence Transformers documentation: Losses — Sentence Transformers (UKP Lab, Hugging Face), 26 Oct 2026
- Dense Text Retrieval based on Pretrained Language Models: A Survey (arXiv 2211.14876) — arXiv (Zhao et al.), 27 Nov 2022
