What is the Bradley-Terry model in RLHF reward models?

In this article

1. Before you start
2. Why interviewers ask this
3. Reward model from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is part of the fine-tuning and alignment learning path, and it assumes you know roughly what a language model does.

Why interviewers ask this

Reward models sit between human opinion and the optimizer that changes the language model, so a mistake in them is copied into every later step. Questions about them are reported by candidates in large language model (LLM) training and alignment interviews: what the Bradley-Terry model is, how a reward model differs from a critic, and how preference data is collected.

A weak answer says "the reward model predicts how good a response is." A strong answer says it is trained on comparisons, writes the loss, and explains why only differences matter. A Staff-level answer adds what breaks: the optimizer finds responses the reward model over-scores, and the true quality falls even as the proxy score rises. This page builds that answer from zero. For the optimizers that consume the reward, see direct preference optimization (DPO) versus RLHF.

Reward model from first principles

Bradley-Terry model

The Bradley-Terry model is a statistical model of paired comparisons, from Bradley and Terry (1952), that gives every item a latent score and says the probability that item A beats item B is the logistic function of the difference of their scores.

Reward overoptimization

Reward overoptimization is the failure in which pushing a policy to raise a proxy reward model's score keeps raising that score while the true quality of the responses (human judgement, or a gold reward model in controlled studies) eventually falls.

In plain words

Imagine a tennis ladder. Nobody measures a player's skill directly. You only see who beat whom, and you give each player a rating so that higher-rated players usually win. A reward model does this for responses: humans say which of two answers they prefer, and the model assigns each answer a rating that explains those choices. The analogy breaks in one place. A response is rated by a neural network that has never seen it before, so the rating has to generalize.

In one precise sentence: the reward model learns a function r(x, y) of prompt x and response y such that the sigmoid of the reward gap between the preferred and rejected response is the probability of the human's choice.

Step 1: Humans compare responses, and each comparison becomes a preference pair

A preference pair is a prompt together with two responses, one marked as preferred (the winner) and one as rejected (the loser). People find it easier to say which answer is better than to give it an absolute score, so the data is comparisons.

In InstructGPT, Ouyang et al. (2022) gave each prompt between 4 and 9 ranked completions, which yields up to K choose 2 pairwise comparisons per prompt, and they dropped ties. They also put all the comparisons from one prompt into a single batch element, because shuffling these correlated pairs across the data led the model to overfit. Input: a prompt and its ranked responses. Output: many winner-loser pairs. The next step turns each pair into a probability.

Step 2: The Bradley-Terry model turns a reward gap into a probability

The logistic function, written σ, is a curve that maps any real number x to 1 / (1 + e to the minus x), so it always lands between 0 and 1. The Bradley-Terry model feeds it the reward gap:

Here r_w and r_l are the rewards of the winner and the loser. Ouyang et al. (2022) describe the same thing in words: the difference in rewards represents the log odds that one response is preferred by a human labeler. Equal rewards give a probability of 0.5, a gap of 1 gives 0.7311 and a gap of 2 gives 0.8808.¹

This is the missing term. The probability depends only on the difference, so adding the same constant to every reward changes nothing: the zero point is not identifiable, and the constant can differ between prompts, so rewards across prompts are not comparable. The scale is identified, since doubling every reward changes every probability. A reward of 3.1 is not "good" by itself. Rafailov et al. (2023) name it as the preference model RLHF relies on.

Step 3: Minimizing a pairwise loss trains the reward model

Training maximizes the probability the model gave to the human's actual choice. That is cross-entropy against a label that is always "the winner won":

In InstructGPT's setup, the network is the supervised fine-tuning (SFT) model with its final unembedding layer removed, so it outputs one scalar per prompt and response (Ouyang et al., 2022). The loss is 0.6931 when the two rewards tie, 0.3133 when the winner leads by 1, 0.1269 when it leads by 2, and 2.1269 when the loser leads by 2.¹ The gradient with respect to the gap is minus (1 minus σ of the gap), so a pair the model ranks wrongly pushes hardest.

Pairs the model already ranks confidently carry a small gradient, so training focuses on the pairs it gets wrong. You now have the full training signal: pairs in, a scalar per response, one cross-entropy loss out.

Step 4: The policy is optimized against the reward model, with a distance check

After training, the reward model scores the outputs of the language model (the policy) during reinforcement learning from human feedback, for example with proximal policy optimization (PPO). The pull of a proxy score is strong, so the policy drifts. The drift is measured by Kullback-Leibler (KL) divergence. KL divergence is the Kullback-Leibler divergence, a number that measures how far one probability distribution has moved from another; here, the policy from its starting version.

Gao et al. (2022, ICML 2023) measure the distance as the square root of that KL and show proxy reward rising while gold reward eventually falls. That is reward overoptimization, and it is why the reward model is a proxy and not the goal.

1 · Compare2 · Score3 · Train4 · OptimizeReplay

Step-by-step description

1. A labeler picks the winner.
2. The reward gap gives a probability.
3. The pairwise loss updates the model.
4. The policy chases the reward.

• A preference pair has a winner and a loser.
• The pairwise loss trains the reward model.
• The policy is then optimized against the reward model.

Worked example

Take one preference pair in which the reward model currently gives the winner 1.2 and the loser 0.4 (illustrative values). The gap is 0.8, so the Bradley-Terry probability of the human's choice is σ(0.8) = 0.6900 and the loss is 0.3711.¹ The gradient weight is 1 minus 0.6900 = 0.3100. If the model had the order wrong (winner 0.4, loser 1.2, gap minus 0.8), the probability would be 0.3100, the loss 1.1711 and the gradient weight 0.6900.¹

The pair counts are the formula K choose 2 from Ouyang et al. (2022), computed in the footnote script; 64 prompts times 36 pairs gives the 2,304-comparison ceiling that paper states for one batch.¹

Text description of the diagram

• The loss is 0.6931 when the two rewards tie.
• The loss is 2.1269 when the loser leads by 2 and 0.1269 when the winner leads by 2.

Trade-offs and edge cases

Three practical choices decide whether a reward model helps or harms.

Training length. Ouyang et al. (2022) report that training was sensitive to the number of epochs and that multiple epochs quickly overfit; they trained for a single epoch at a learning rate of 9e-6.

Model size. The same paper used one 6B reward model for all its PPO models of all sizes. A 175B reward model could reach lower validation loss, but training was less stable, it suited the PPO value function worse and it greatly increased compute. Bigger is not automatically better.

Overoptimization and the KL penalty. In Gao et al.'s (2022, ICML 2023) setup, a KL penalty raised the proxy score reachable at a given KL but did not measurably improve the gold-reward frontier. They also find reinforcement learning far less KL-efficient than best-of-n sampling, where KL grows roughly as the log of n. These findings are for their setup; reported practice elsewhere may differ.

DPO. Direct preference optimization (DPO) keeps the Bradley-Terry assumption but drops the explicit reward model: Rafailov et al. (2023) use a change of variables to write the preference loss as a function of the policy directly, giving a simple binary cross-entropy objective. See DPO vs RLHF for the comparison.

Text description of the diagram

• A reward model scores a finished response from human preference data.
• A critic estimates expected return while reinforcement learning runs.

A reward model scores a finished response from human preference data before reinforcement learning starts. A critic, or value model, estimates the expected return of a state while reinforcement learning runs. In InstructGPT's PPO the value function was initialized from the reward model, but the two are separate roles. Group relative policy optimization (GRPO) drops the critic entirely: it samples a group of responses to the same prompt and uses the group's mean reward (and standard deviation) as the baseline (Shao et al., 2024). The reward signal itself still comes from a reward model or a rule.

The 2025-26 state. Reasoning-model training has moved toward rule-based, verifiable rewards where answers can be checked, such as a correct final answer. DeepSeek-AI (2025) report that they did not use a neural outcome or process reward model for DeepSeek-R1-Zero, because such models can suffer from reward hacking in large-scale reinforcement learning. Neural reward models remain the tool for open-ended qualities that no rule can check.

How to answer it in an interview

The spoken structure: define the model, write the loss, say differences only, then name the failure mode.

Common mistakes

• "The reward model outputs the probability a response is good." (wrong) It outputs an unbounded scalar; the probability appears only after a sigmoid of a difference. (right) "The probability is the sigmoid of the reward gap."
• "DPO has no Bradley-Terry model." (wrong) Rafailov et al. (2023) say DPO relies on a preference model such as Bradley-Terry. (right) "DPO assumes Bradley-Terry but skips the explicit reward model."
• "Train the reward model for several epochs." (wrong) Multiple epochs quickly overfit in InstructGPT. (right) "One epoch, with held-out accuracy checked."
• "A KL penalty stops reward hacking by itself." (wrong) In Gao et al.'s setup it did not improve the gold-reward frontier. (right) "The KL penalty helps, but I still measure a gold or held-out reward."

Follow-up questions

How does DPO implicitly use a Bradley-Terry model?

DPO assumes human preferences follow Bradley-Terry, then rewrites the reward as a function of the policy so the loss is a binary cross-entropy on policy log-ratios. The reward model is implicit in the policy itself (Rafailov et al., 2023).

How is a reward model different from a value or critic model in policy optimization algorithms such as GRPO?

A reward model is trained on preference data and scores whole responses; a critic is trained during reinforcement learning to estimate expected return. In PPO the critic can start from the reward model's weights, but they have separate jobs. GRPO removes the critic and uses the mean and standard deviation of rewards over a group of sampled responses to the same prompt as the baseline, while the rewards still come from a reward model or a rule (Shao et al., 2024).

How do you collect high-quality preference data?

A strong approach writes clear labeling guidelines, shows labelers several completions per prompt so one ranking yields many pairs, and drops ties as InstructGPT did (Ouyang et al., 2022). Checking agreement between labelers on a shared sample is sound practice.

Given a reward model trained on 50,000 comparisons, how do you estimate the KL budget before overoptimization?

No universal number exists. Gao et al. (2022, ICML 2023) report that the fitted curves depend on reward model size and dataset size, so the honest answer is to measure: optimize, track a held-out or gold reward against square-root KL, and stop near its peak.

Why use rule-based rewards instead of a neural reward model?

A rule-based reward, such as checking a final answer, cannot be exploited the way a learned scorer can, and it needs no preference data. DeepSeek-AI (2025) report avoiding neural reward models in R1-Zero for that reason. It only works where correctness can be checked.

Why do larger reward models not always help?

Larger reward models can reach lower validation loss, but in InstructGPT the 175B one trained less stably and cost much more compute in PPO. The paper used a 6B reward model for all policy sizes. For parameter-efficient ways to train big models cheaply, see low-rank adaptation (LoRA) fine-tuning and the overview of fine-tuning an LLM.

Sources

1. Direct Preference Optimization: Your Language Model is Secretly a Reward Model (arXiv 2305.18290, v3) — arXiv (Rafailov et al.), accessed 2026-10-26, 29 Jul 2024
2. Training language models to follow instructions with human feedback (arXiv 2203.02155) — arXiv (Ouyang et al.), accessed 2026-10-26, 4 Mar 2022
3. Scaling Laws for Reward Model Overoptimization (arXiv 2210.10760) — arXiv (Gao, Schulman, Hilton), accessed 2026-10-26, 19 Oct 2022
4. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (arXiv 2402.03300) — arXiv (Shao et al.), accessed 2026-10-26, 5 Feb 2024
5. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (arXiv 2501.12948) — arXiv (DeepSeek-AI), accessed 2026-10-26, 22 Jan 2025
6. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons (Biometrika 39(3/4), doi 10.2307/2334029) — Biometrika (Bradley and Terry), 1 Dec 1952