DPO explained: preference pairs, one loss and no separate reward model

In this article

1. Before you start
2. Why interviewers ask this
3. DPO from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step of the how large language models are trained learning path.

Why interviewers ask this

A candidate who answers "DPO is RLHF without a reward model" is stopped by the first follow-up: where did the reward go? Questions about alignment methods are reported by candidates in interviews at AI labs and large tech companies, and DPO is a common one because it tests whether you can follow a derivation, not only name an acronym.

A weak answer lists the benefits: simpler, more stable, cheaper. A strong answer says that DPO optimizes the same objective as RLHF, shows that the reward can be written as a log-ratio of two language models, and writes the loss. A Staff-level answer adds the failure mode: the loss measures a margin, so the probability of the chosen answer can fall while training looks healthy. Each of those points has its own step below.

DPO from first principles

Preference pair

A preference pair is one training example made of a prompt, a chosen response that people preferred and a rejected response that they liked less. Preference datasets are lists of these triples, usually stored under the column names prompt, chosen and rejected.

Beta

Beta (β) is the coefficient in the DPO loss that sets how strongly the trained model is held near the reference model. The paper uses a value of 0.1 by default.

In plain words

Imagine a writing student who already writes passable essays. A teacher shows two essays on one topic and says which is better. The student needs no scoring rubric. The student makes the better essay a little more likely and the worse one a little less likely, compared with before. The student must not drift into a style the teacher never saw. The analogy breaks in one place: the model shifts probabilities over every possible answer at once, not over one essay.

In precise terms, DPO compares a language model with a frozen reference model. It gives more probability to chosen answers than to rejected ones, relative to that reference. A sigmoid loss on that difference turns the adjustment into plain classification.

Step 1: Preference pairs and the RLHF objective set the target

Standard RLHF has three phases: supervised fine-tuning, then a reward model fitted to human preference pairs, then reinforcement learning against that reward. The reward model uses the Bradley-Terry model. The Bradley-Terry model is a model of paired comparisons in which the chance that answer A beats answer B is the sigmoid of the reward difference. The reinforcement learning phase is usually proximal policy optimization (PPO), a reinforcement learning algorithm that updates a policy in small, clipped steps.

The objective of that last phase is to maximize reward while paying a penalty of β times the Kullback-Leibler (KL) divergence from the reference model. The KL divergence is a measure of how far one probability distribution has moved from another. The penalty stops the model from drifting into outputs that exploit the reward model. The input to this step is a set of preference pairs, which train a reward function; the output is a target: the best policy for reward minus the KL penalty. You now know what DPO has to reproduce.

Step 2: The policy itself becomes the reward

The paper's key move is that the best policy for the RLHF objective already contains a reward, so none needs to be trained separately. Rafailov et al. (2023) write the optimum as the reference model reweighted by the exponential of the reward over β, divided by Z(x). The partition function is a number, written Z(x), that makes the probabilities over all answers to a prompt sum to one.

optimal policy:  pi_r(y|x) = pi_ref(y|x) * exp(r(x,y) / beta) / Z(x)
invert it:       r(x,y) = beta * log( pi_r(y|x) / pi_ref(y|x) ) + beta * log Z(x)

Z(x) is hard to compute, but it depends only on the prompt. Bradley-Terry depends only on the difference of the two rewards for the same prompt, so the Z(x) terms cancel; in the paper's words, "the partition function cancels". What is left is a preference probability written only in terms of the policy and the reference model. As the authors put it, "the policy network represents both the language model and the (implicit) reward". The input is two language models; the output is a reward read from their log-ratio.

Step 3: A sigmoid loss on the margin trains the policy

Maximum likelihood on the cancelled expression gives the DPO loss. For one pair, compute the policy's log-ratio to the reference for the chosen answer and for the rejected answer, subtract, multiply by β and take minus the log-sigmoid.

L = -log sigmoid( beta * ( log(pi(y_w|x)/pi_ref(y_w|x)) - log(pi(y_l|x)/pi_ref(y_l|x)) ) )

Here y_w is the chosen answer and y_l the rejected one. The paper's appendix gives the core as one PyTorch line, which the Hugging Face docs describe as a sigmoid loss that fits a binary classifier under the Bradley-Terry model:

losses = -F.logsigmoid(beta * (pi_logratios - ref_logratios))

The code compares two margins, the policy's chosen-minus-rejected log-probability and the reference's, and is a plain binary cross-entropy. The input is four sequence log-probabilities; the output is one number per pair. The paper uses β = 0.1 by default and 0.5 for its summarization task.

Step 4: Each pair is weighted by how wrongly it is ranked

The gradient of the loss raises the log-likelihood of the chosen answer and lowers that of the rejected one, and it scales each pair by a weight. That weight is the sigmoid of the implicit reward of the rejected answer minus that of the chosen answer, so pairs the model currently ranks wrongly get larger updates. The paper reports that a version without this weighting can make the language model degenerate.

The training pipeline is short: sample two responses per prompt from the reference model, label which is preferred to build a fixed dataset, then minimize the loss. The paper's defaults are a batch size of 64 and a learning rate of 1e-6. The trainer docs advise running SFT first so the data is in-distribution, and show the trainer working with a low-rank adaptation (LoRA) adapter model, so LoRA fine-tuning and other parameter-efficient methods combine with it. The paper's DPO needs no sampling from the policy during training and no value network.

1 · Preference pair2 · Policy and reference3 · Sigmoid loss4 · Weighted updateReplay

Step-by-step description

1. Each example is a prompt, a chosen and a rejected answer.
2. Both models score the pair; the reference is frozen.
3. The margin of two log-ratios feeds one loss.
4. Wrongly ranked pairs get larger updates.

• A preference pair is a prompt with a chosen and a rejected response.
• The frozen reference model supplies the log-ratio baseline.
• A sigmoid loss on the margin of two log-ratios trains the policy.
• Each pair is weighted by how wrongly the implicit reward ranks it.

Worked example

A toy pair shows the loss by hand. Take the paper's default β = 0.1 and illustrative log-ratios. The log-ratio of an answer is its policy log-probability minus its reference log-probability, and the loss depends only on the chosen log-ratio minus the rejected log-ratio, times β.¹

At the start the policy equals the reference, both log-ratios are 0, the margin is 0 and the loss is minus the log of one half, 0.6931, with a gradient weight of 0.5.¹ If training raises the chosen answer by 5 log-units and lowers the rejected one by 5, the margin is 1.0, the loss falls to 0.3133 and the weight to 0.2689.¹ If the pair is ranked wrongly, with the chosen log-ratio at minus 5 and the rejected at plus 5, the margin is minus 1.0, the loss rises to 1.3133 and the weight to 0.7311.¹

The third case is the trap. If both answers fall, the chosen one by 5 and the rejected one by 15, the margin is again 1.0, so the loss is again 0.3133, yet the chosen answer is now only 0.67% as likely as under the reference.¹ A pair where both responses fall but the rejected one falls faster has the same loss as a pair where the chosen response rises.

Text description of the diagram

• The loss is 0.6931 at the start and 1.3133 when the pair is ranked wrongly.
• A pair where both responses fall but the rejected one falls faster has the same loss as a pair where the chosen response rises.

The loss is 0.6931 at the start and 1.3133 when the pair is ranked wrongly. Now vary β on a smaller log-ratio gap, a chosen log-ratio of plus 1 and a rejected one of minus 1. The paper's sentiment experiment swept β over 0.05, 0.1, 1 and 5; on this gap the loss is 0.6444, 0.5981, 0.1269 and 0.000045.¹ A larger β makes the same log-ratio gap count for more, so the loss saturates sooner and the model stops moving away from the reference earlier.

Trade-offs and edge cases

DPO trades flexibility for simplicity. The paper reports a win rate of about 61% for DPO against 57% for PPO on a summarization task, and human raters preferred DPO samples at temperature 0.25 over PPO samples 58% of the time. On a news-article transfer test the paper reports 0.36 for DPO against 0.26 for PPO at temperature 0. The experiments used models of up to 6 billion parameters, with an AI model as the proxy judge, and the authors call scaling to far larger models future work.

The structural differences are what an interviewer wants to hear. RLHF trains a separate reward model while DPO keeps the reward implicit in the policy. RLHF samples from the policy during training while DPO trains on a fixed dataset. PPO needs a value network while DPO needs none; the trainer docs note that PPO expects a model with a value head and DPO a plain causal language model. DPO still needs a frozen reference model. See the full DPO vs RLHF comparison for more.

Text description of the diagram

• RLHF trains a separate reward model while DPO keeps the reward implicit in the policy.
• RLHF samples from the policy during training while DPO trains on a fixed dataset.
• PPO needs a value network while DPO needs none.
• DPO still needs a frozen reference model.

Edge cases follow from the fixed dataset. Vanilla DPO trains on one static set of pairs, whereas online DPO needs only prompts and scores fresh completions with a reward model or a judge. A bigger problem is likelihood displacement, the observation that the probability of preferred responses often decreases during training. In single-token experiments the preferred probability dropped by 0.21 to 0.96, and when the two answers are similar in meaning the mass can move to tokens with opposite meaning. A 2025 analysis reports that both probabilities decrease, with the rejected one dropping more sharply, and notes that in theory the loss can reach its optimum by simply lowering the rejected probability.

Proposed mitigations are a supervised term on the chosen answer, as that analysis discusses, and filtering the dataset, which Razin et al. (2024) propose doing with a similarity score. The extra behavior change DPO buys is related to the alignment tax. Several variants respond to the same weaknesses, such as identity preference optimization (IPO), Kahneman-Tversky optimization (KTO), odds ratio preference optimization (ORPO) and simple preference optimization (SimPO); the trainer docs list alternative loss types. Newer methods such as group relative policy optimization (GRPO), covered in GRPO explained, keep a reward and on-policy sampling but replace the value network with a group-average baseline.

How to answer it in an interview

The spoken structure: say what DPO trains on, say it is the same objective as RLHF, give the log-ratio reward and the loss, then name the fixed-dataset and likelihood-displacement caveats.

Common mistakes

• "DPO has no reward model, so it ignores rewards." (wrong) It optimizes the same KL-regularized reward objective. (right) "The reward is implicit: beta times the log-ratio of policy to reference."
• "DPO is just supervised fine-tuning on the chosen answers." (wrong) The loss is contrastive and measured against a frozen reference model, with a weight per pair. (right) "The paper's plain fine-tuning on preferred answers did not improve significantly over the SFT model on summarization."
• "DPO ensures the chosen answer becomes more likely." (wrong) It raises only the margin. (right) "The chosen likelihood can fall, so I log it."
• "Beta is a learning rate." (wrong) It is the strength of the pull towards the reference. (right) "Beta is the KL-penalty coefficient."
• "DPO needs no reference model or SFT." (wrong) The log-ratios need the reference, normally the SFT model. (right) "I train SFT first and freeze a copy as the reference."

Follow-up questions

What is the difference between DPO and RLHF?

Both optimize reward minus a KL penalty to a reference model. RLHF fits an explicit reward model and runs reinforcement learning with sampling; DPO substitutes the log-ratio reward into the preference loss and trains on a fixed dataset. The DPO vs RLHF page goes further.

What does beta do in DPO?

Beta sets how strongly the policy is anchored to the reference model. In the beta sweep above, larger values saturate the loss on a smaller log-ratio gap, so the policy stays closer to the reference.

Why does the chosen likelihood fall during DPO training?

The loss rewards the gap between chosen and rejected log-ratios, not their absolute level. If the rejected answer falls faster, the gap grows even while the chosen answer falls, which is the pattern called likelihood displacement.

Does DPO need a reference model and an SFT step?

Yes to the reference model, which supplies the log-ratios. The paper initializes it from the SFT model whenever one is available and otherwise fits it by maximum likelihood on the chosen completions.

Is DPO better than PPO?

Not in general. The paper finds DPO as good as or better on tasks up to 6 billion parameters with a proxy judge, and the authors ask for more study of generalization and scale. See what RLHF is for the PPO side.

What is online DPO?

Online DPO samples fresh completions during training and scores them with a reward model or a judge, so it needs only prompts, unlike vanilla DPO, which needs a ready-made preference dataset.

Sources

1. Direct Preference Optimization (arXiv 2305.18290, v3) — arXiv (Rafailov, Sharma, Mitchell, Ermon, Manning, Finn), 29 May 2023
2. DPO Trainer documentation (main branch) — Hugging Face (GitHub), 28 Feb 2026
3. DPO Trainer documentation, v0.8.0 — Hugging Face, 26 Oct 2026
4. Online DPO Trainer documentation — Hugging Face, 26 Oct 2026
5. Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization (arXiv 2410.08847) — arXiv (Razin et al.), 1 Oct 2024
6. Rethinking DPO: The Role of Rejected Responses in Preference Misalignment (arXiv 2506.12725) — arXiv (preprint 2506.12725, Jun 2025), 1 Jun 2025