What are in-batch negatives and sampled softmax?

In this article

1. Before you start
2. Why interviewers ask this
3. In-batch negatives and sampled softmax from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page belongs to the deep learning learning path.

Why interviewers ask this

A retrieval model trained against the wrong negatives ranks the wrong items, so interviewers use this topic to separate candidates who can name a loss from candidates who know what the loss assumes. The topic fits retrieval, recommendation and dense passage retrieval discussions, where a model must pick a few right items from a huge catalogue.

A weak answer says "the other items in the batch are negatives." A strong answer explains why that approximates a softmax over the catalogue, names the popularity bias it creates and gives the correction. A Staff-level answer adds what the correction still leaves wrong.

In-batch negatives and sampled softmax from first principles

Sampled softmax

Sampled softmax is a training loss that replaces the softmax over every class with a softmax over the true class plus a small sample of wrong classes, so the cost per example depends on the sample size and not on the number of classes. It is a training device, not a different model.

logQ correction

The logQ correction is an adjustment that subtracts the logarithm of an item's sampling probability from that item's logit before the softmax, so items the sampler picks often are not punished more than items it picks rarely. It reduces popularity bias in sampled softmax but does not remove it completely.

Accidental hit

An accidental hit is a sampled negative that is in fact a correct answer for the current example, for instance the same item appearing twice in one batch, so the loss pushes the model away from an item it should rank high.

In plain words

Picture a quiz that teaches a contestant to match requests to books. Showing every book in a huge library for every request is too slow. So each request gets its true book plus a handful of wrong ones. In-batch negatives borrow the wrong ones from the other contestants in the same round: their true books become your wrong books.

A mini-batch is the small group of training examples the model handles in one update. The analogy breaks in one place: contestants ask for popular books more often, so the wrong books are not a fair sample of the library.

Step 1: A full softmax is too costly for a huge catalogue

A full softmax sums over every item in the catalogue. A logit is the raw score for one item before softmax turns the scores into probabilities. The normaliser, the sum of all exponentiated logits, needs one logit per item for every training example.

For translation models, Jean et al. (2015) note that "training complexity as well as decoding complexity increase proportionally to the number of target words." With an illustrative catalogue of 10,000,000 items, each query needs 10,000,000 logits, against 128 for a sampled batch.¹

Step 2: A sampled softmax keeps the positive and a few negatives

A sampled softmax sums over the positive and a small set of sampled negatives. Jean et al. approximate the costly part of the gradient with importance sampling, which is a way to estimate a sum over many items by drawing a few from a proposal distribution Q and reweighting each draw. Their weight is $\exp (E(y)-\log Q(y))$, where $E(y)$ is the item's score. That $\log Q$ term is where the later "logQ" correction comes from: an item the sampler draws often gets a smaller weight.

The TensorFlow documentation states a limit: "This operation is for training only. It is generally an underestimate of the full softmax loss." It also suggests the full softmax for evaluation or inference.

Step 3: The other items in the mini-batch become the negatives

In-batch negatives reuse the other positives in the batch as that set. An in-batch negative is an item borrowed from another example of the same mini-batch, and it costs no extra encoder passes. Karpukhin et al. (2020) describe B questions per batch, with Q the matrix of question embeddings and P the matrix of passage embeddings, so the product of Q and the transpose of P is a B by B similarity matrix.

The two sets of embeddings come from two encoders, one for queries and one for items, called the two towers of a two-tower model. Row i holds the positive on the diagonal and B minus 1 negatives elsewhere, and the batch scores B squared pairs. The loss is a cross-entropy over each row, with column i as the right answer for row i. For an illustrative batch of 128, each query has 127 negatives and the batch scores 16,384 pairs.¹ The matrix product is the same operation as a weight matrix multiplication.

Step 4: The logQ correction undoes most of the popularity bias

Popular items appear in batches more often, so they serve as negatives more often. Khrylchenko et al. (2025) note that in-batch sampling leads "to over-representation and over-penalization of popular items." Yi et al. (2019) found in-batch loss "subject to sampling biases" and built the estimated item frequency into the batch loss, for YouTube retrieval from tens of millions of videos.

The logQ correction subtracts the log of each item's batch probability from its logit: $s^{\prime }=s-\ln p$. For an item sampled with probability 0.05 the logit rises by 3.00, and for one sampled with probability 0.0005 it rises by 7.60 (illustrative probabilities).¹ The rare item's term in the denominator is amplified, which offsets how rarely the batch shows it. The correction lives only in the training loss; at serving time the model ranks by the uncorrected score s. Probability and logarithm rules are in probability and statistics.

1 · Full softmax2 · Sampled softmax3 · Batch as the sample4 · logQ correctionReplay

Step-by-step description

1. Exact, but one logit per catalogue item.
2. Keep the positive and a few negatives.
3. Other positives in the batch are the negatives.
4. Subtract log probability to offset popularity.

• A full softmax sums over every item in the catalogue.
• A sampled softmax sums over the positive and a small set of sampled negatives.
• In-batch negatives reuse the other positives in the batch as that set.
• The logQ correction subtracts the log of each item's batch probability from its logit.

Worked example

One query with four candidate items shows both losses. The scores are illustrative: item 0 is the positive with score 3.0, and the negatives score 1.0, 0.0 and 2.0. The batch probabilities are illustrative too: 0.01 for items 0 and 2, 0.0005 for item 1 and 0.05 for item 3.¹

Without correction the exponentials are 20.09, 2.72, 1.00 and 7.39, so the positive gets probability 0.64 and the loss is 0.44.¹ With the correction each weight is $\exp (s)/p$: 2,008.55, 5,436.56, 100.00 and 147.78. The positive now gets 0.26 and the loss is 1.34.¹ The loss rises because item 1 is rare in batches, so its one appearance stands for many missed appearances. A real batch averages many such rows, so one row is a demonstration, not a measurement.

Each query has B minus 1 in-batch negatives. At a batch of 128 a query has 127 negatives and the batch scores 16,384 pairs, while at 1,024 the pairs number 1,048,576.¹ The 255 comes from adding the 128 keyword-search negatives of the whole batch to the 127 in-batch ones.¹

Text description of the diagram

• Each query has B minus 1 in-batch negatives.
• At a batch of 128 a query has 127 negatives and the batch scores 16,384 pairs.

Trade-offs and edge cases

In-batch negatives trade fidelity for cost. Only the full softmax uses every item, and in-batch negatives need no extra encoder passes. The correction is not exact: Khrylchenko et al. (2025) show it "does not fully eliminate the bias" because the positive item is not sampled and is always present.

An accidental hit is the main edge case. TensorFlow removes accidental hits by default, and in a batch the analogue is a duplicate of the positive, which is then a false negative: a negative that is really a correct answer. Mask duplicates.

Temperature is a number that divides every logit before the softmax; a small one sharpens the competition between close items. Yi et al. report that corrected loss beat uncorrected loss by a large margin at each temperature on a Wikipedia corpus of 5.3 million pages, and that temperature must be tuned carefully.

In-batch negatives are often easy. Karpukhin et al. add one keyword-search hard negative to the in-batch gold passages in their best model. Qu et al. (2020) report, on the MS MARCO passage ranking development set, mean reciprocal rank at 10 (mean reciprocal rank (MRR)@10) of 32.39 for in-batch negatives and 33.32 for cross-batch negatives. Hard negatives without denoising scored 26.03, and with denoising 36.38. Unfiltered hard negatives can hurt through false negatives.

Text description of the diagram

• Only the full softmax uses every item.
• In-batch negatives need no extra encoder passes.

How to answer it in an interview

The spoken structure: say what the negatives are, say why the matrix is cheap, name the popularity bias and its correction, then the limits.

Common mistakes

• "In-batch negatives are random, unbiased negatives." (wrong) They follow the batch distribution, which is skewed toward popular items. (right) "They are cheap but popularity-biased, so I apply the logQ correction."
• "The logQ correction makes the loss unbiased." (wrong) The positive is always present, so some bias remains. (right) "It reduces the bias; it does not fully remove it."
• "I validate with the sampled loss." (wrong) It is generally an underestimate of the full softmax loss. (right) "I train with the sampled loss and evaluate with the full softmax or full retrieval."
• "Every other item in the batch is a true negative." (wrong) A duplicate of the positive is an accidental hit. (right) "I mask duplicates of the positive in each row."
• "I subtract log p when ranking at serving time." (wrong) The correction only fixes the training loss, and applying it at inference distorts the ranking of popular items. (right) "The correction is a training device; serving uses the raw score."

Follow-up questions

What is sampled softmax and why use it?

Sampled softmax approximates the full softmax with the true class plus a few sampled negatives, so cost depends on the sample size and not on the catalogue. It is for training only.

How does the logQ correction work?

It subtracts the log of each item's sampling probability from its logit, so rare items count for more and popular items for less in the denominator. The toy row moved the loss from 0.44 to 1.34.¹

Why not just use a bigger batch to get more negatives?

A larger batch gives more negatives per query, but the score matrix grows with the square of the batch, and gains are modest: Qu et al. report 33.32 for cross-batch negatives, which share negatives across graphics processing units (GPUs), against 32.39 for in-batch negatives, below the denoised hard-negative result.

How do you get the item probability for the correction?

Yi et al. build an estimated item frequency into the loss, which fits catalogues that change over time. Their streaming estimator tracks the gap δ between an item's appearances in the stream and uses p ≈ 1/δ. Any estimate of how often an item appears in a batch serves the same role.

How do hard negatives change the picture?

Hard negatives teach finer distinctions than random batch items, but unfiltered ones can be false negatives and lower quality, as the 26.03 against 36.38 gap shows. Neighbouring topics are in the deep learning interview questions, and the feed-forward network page covers the layers inside the towers.

How do you evaluate a model trained with sampled softmax?

Use full retrieval over the catalogue and a ranking metric such as recall at k on held-out queries. The sampled loss only measures the training objective, and it underestimates the full loss.

Sources

1. Sampling-Bias-Corrected Neural Modeling for Large Corpus Item Recommendations — ACM RecSys 2019 (Yi et al.), 16 Sep 2019
2. Sampling-bias-corrected neural modeling for large corpus item recommendations (author copy) — Academia.edu copy of Yi et al., 16 Sep 2019
3. Correcting the LogQ Correction: Revisiting Sampled Softmax for Large-Scale Retrieval (arXiv 2507.09331) — arXiv / RecSys 2025 (Khrylchenko et al.), 1 Jul 2025
4. On Using Very Large Target Vocabulary for Neural Machine Translation (arXiv 1412.2007) — ACL 2015 (Jean, Cho, Memisevic, Bengio), 26 Jul 2015
5. tf.nn.sampled_softmax_loss — TensorFlow API documentation v2.16.1, 26 Oct 2026
6. Dense Passage Retrieval for Open-Domain Question Answering (arXiv 2004.04906) — EMNLP 2020 (Karpukhin et al.), 10 Apr 2020
7. RocketQA: An Optimized Training Approach to Dense Passage Retrieval (arXiv 2010.08191) — arXiv (Qu et al.), 1 Oct 2020