What does DPO optimize, and what does its loss measure?

In this article

1. Before you start
2. Why interviewers ask this
3. DPO from first principles
4. The Staff-level answer
5. The rubric
6. Three variants of this question
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

DPO is one stage of how large language models are trained, and this page assumes the transformer is already familiar as the model being tuned.

Why interviewers ask this

Alignment questions reach candidates for roles that touch language-model training, and DPO is one of the shortest ways to see whether you understand the field or only its vocabulary. A candidate who has met DPO only as a library function can say "it replaces RLHF". A candidate who understands it can say what the loss is, where the reward went and why a reference model is needed.

A weak answer says "DPO trains on human preferences without a reward model." A strong answer states the objective, names the implicit reward and writes the loss as a logistic function of a margin. A Staff-level answer adds what the loss does not promise: it does not force the chosen answer to become more likely, and it only learns from the pairs it is given. The page below builds that answer from zero, then shows it with numbers.

DPO from first principles

Log probability

A log probability is a score that equals the natural logarithm of the probability a model assigns to a token or a response; it is zero or negative, and summing the per-token values gives a numerically stable score for a whole response.

In plain words

Imagine a chef who has a trusted old recipe. Diners taste two new dishes and say which one they prefer. The chef nudges the recipe toward the preferred dish. The chef does not nudge so far that it stops being the old recipe. DPO works the same way. The language model is the chef. The reference model is the old recipe. The preference pairs are the diners' votes. The analogy breaks in one place: the chef has a tongue, while DPO has no reward judge at all, only the stored votes.

In one precise sentence, DPO adjusts the model so that the chosen response becomes relatively more likely than the rejected one, compared with the reference model. A penalty holds the model near the reference.

Step 1: RLHF asks for reward without drifting

RLHF is the problem DPO solves. It collects human labels of the relative quality of model generations, fits a reward model, then fine-tunes the language model with reinforcement learning to "maximize this estimated reward without drifting too far from the original model" (Rafailov et al., 2023). Kullback–Leibler (KL) divergence measures that drift. KL divergence is a number that says how far one probability distribution is from another; it is not symmetric, so it is not a true distance. The objective is therefore reward minus a KL penalty.

RLHF fits a reward model first and then runs reinforcement learning against it. The paper notes that this pipeline is more complex than supervised learning, involves training multiple models and samples from the policy during training, which costs compute (paper introduction). DPO keeps the objective and removes the machinery.

You now know what is being optimized: reward with a leash. The next step shows where the reward goes.

Text description of the diagram

• RLHF fits a reward model first and then runs reinforcement learning against it.
• DPO trains the policy directly on preference pairs with a classification loss.

Step 2: The reward hides in the ratio of policy to reference

The authors introduce "a new parameterization of the reward model in RLHF that enables extraction of the corresponding optimal policy in closed form" (Rafailov et al., 2023). Closed form means the best policy can be written as a formula instead of searched for with reinforcement learning. Turn that formula around and the reward can be written in terms of the policy. The implicit reward is the scaled log ratio of the policy to the reference model. The inverted formula also has a term, $\beta \log Z(x)$, that depends only on the prompt; because both responses share the prompt, it cancels when the two rewards are subtracted, which is why DPO works on pairs (Rafailov et al., 2023, §4).

In symbols, the implicit reward of a response $y$ to a prompt $x$ is $\beta \log \frac{{\pi }_{\theta }(y|x)}{{\pi }_{\text{ref}}(y|x)}$, where ${\pi }_{\theta }$ is the model being trained, ${\pi }_{\text{ref}}$ is the reference model and $\beta$ is a constant that scales the leash.

This is why the reference model is in the loss. Without it there is no baseline, so the ratio, and with it the reward, would not exist. This is the point of the paper's title: the language model is secretly a reward model.

Step 3: The loss is a logistic loss on a reward gap

Humans only said which response they preferred, so the loss asks one question: does the implicit reward of the chosen response beat that of the rejected response? The paper turns this into "only a simple classification loss" (Rafailov et al., 2023). It models human choices with the Bradley–Terry model, where the probability that the chosen response wins is the sigmoid of the reward gap (Rafailov et al., 2023, §4, Eq. 5–7). The DPO loss is the negative log-likelihood of the observed preferences under that model, and that is what the loss measures. With $y_w$ the chosen and $y_l$ the rejected response, the loss for one pair (the DPO objective, Rafailov et al., 2023, Eq. 7) is:

Read it aloud: take the chosen response's log ratio, subtract the rejected response's log ratio, scale by $\beta$, pass it through the sigmoid and take the negative log. A logistic loss on the chosen-minus-rejected gap trains the policy. When the gap is large and positive the loss is near zero; when the gap is zero the loss is the natural log of two.

Step 4: Training grows the gap, and the numbers show how

Each training step raises the gap, which means the policy moves toward the chosen response and away from the rejected one, relative to the reference. The update is bigger when the model currently ranks the pair wrongly, because the sigmoid of the negative gap is the weight on the step. The paper's gradient analysis shows the same weighting (Rafailov et al., 2023, §4, "What does the DPO update do?").

A pair the model already ranks correctly has a weight near zero, so it gets only a small update and training spends its effort on the pairs the model still gets wrong. Nothing is sampled from the policy in the loop: the pairs are stored, so the loss is computed from log probabilities of fixed text.

1 · RLHF objective2 · Implicit reward3 · Logistic loss4 · Policy updateReplay

Step-by-step description

1. Reward minus a KL penalty.
2. Beta times the log ratio to the reference.
3. The chosen reward should beat the rejected.
4. The gap grows, near the reference.

• The implicit reward is the scaled log ratio of the policy to the reference model.
• A logistic loss on the chosen-minus-rejected gap trains the policy.

A worked example with made-up log probabilities

Take illustrative log probabilities for one pair, with beta set to 0.1. Case A: the chosen response has policy log probability −12.0 and reference −13.0; the rejected one has policy −15.0 and reference −14.0. The log ratios are +1.0 and −1.0, the implicit rewards are 0.1 and −0.1, the gap is 0.2, the sigmoid is 0.5498 and the loss is 0.5981.¹ Before training the policy equals the reference, the gap is zero and the loss is 0.6931.¹

With beta at 0.5 the same log ratios give a gap of 1.0 and a loss of 0.3133.¹ A larger beta is a tighter leash: the loss is satisfied by a smaller shift in log ratios, so training moves the policy less far from the reference. The weight on the update, the sigmoid of the negative gap, is 0.4502 at beta 0.1.¹

Case B: a lower loss with a less likely chosen answer

Case B is illustrative too: the chosen policy log probability falls to −14.0 against a reference of −13.0, and the rejected one falls to −18.0 against −14.0. The gap is 0.3 and the loss is 0.5544, lower than case A, even though the chosen response became less likely.¹

In these illustrative cases, case A has a chosen log ratio of 1.0 and a rejected log ratio of −1.0. Case B has −1.0 and −4.0. Case B has the lower loss, 0.5544 against 0.5981, although its chosen response became less likely.

Text description of the diagram

• Case B has the lower loss, 0.5544 against 0.5981, although its chosen response became less likely.
• Case A has a chosen log ratio of 1.0 and a rejected log ratio of −1.0.

The Staff-level answer

What makes it Staff-level is the last two sentences: it names the hidden reward, the leash and a failure mode the loss does not rule out. Pal et al. (2024) report that standard DPO can reduce the likelihood of the preferred responses, so the failure mode is observed in practice, not only in the toy case B.

The rubric

Three variants of this question

Does DPO use a reward model?

DPO trains no separate reward model, but the reward still exists implicitly: it is the scaled log ratio of the policy to the reference model. The paper's title says the same thing, that the language model is secretly a reward model.

What is the DPO loss and what does beta do?

The loss is the negative log sigmoid of beta times the gap between the chosen and rejected log ratios. In this standard form beta scales that gap and so sets how strongly the policy is held near the reference. A larger beta means the loss is satisfied by a smaller shift in log ratios, so training moves the policy less far from the reference, and a larger beta is a tighter leash. The worked example above gives the illustrative numbers at beta 0.1 and 0.5.

Is DPO a form of RLHF?

DPO solves the same reward-with-a-leash objective as RLHF by a different route, replacing the reward model and the reinforcement learning loop with one classification loss. The DPO vs RLHF comparison covers when each fits.

Common mistakes

• "DPO has no reward model at all." (wrong) The reward is implicit, not absent. (right) "There is no separate reward model; the reward is the policy's log ratio to the reference."
• "DPO is not RLHF." (wrong) Usage differs, but the paper frames DPO as solving the RLHF objective by a different method. (right) "It is a way of solving the RLHF objective directly."
• "The loss pushes the chosen answer's probability up." (wrong) It pushes the gap between ratios up, and both can fall, as in case B; Pal et al. (2024) report that standard DPO can reduce the likelihood of preferred responses. (right) "It raises the chosen response relative to the rejected one."
• "The reference model is optional." (wrong) The ratio, and so the reward, is defined against it. (right) "It is the baseline that anchors the loss."

Follow-up questions

Why does DPO need a reference model?

The implicit reward is a ratio against the reference, and the closeness penalty from the RLHF objective is carried by that ratio, so the loss cannot be written without the reference's log probabilities.

Does DPO make the chosen answer more likely?

Not necessarily. The loss depends on the gap between two log ratios, so the gap can grow while the chosen response's own probability falls, as in the worked example's case B; Pal et al. (2024) report the same effect, a drop in the likelihood of preferred responses under standard DPO. Check the chosen log probability during training rather than assuming it rises.

What are the limits of DPO?

It learns only from the preference pairs it is given, so its quality tracks the quality of that data, and it never samples new responses from the policy during training. Whether those limits matter depends on the task; the alignment tax page discusses the cost side of tuning.

When would you reach for another method?

Reinforcement learning methods that sample from the policy fit tasks where a reward can be checked directly; see group relative policy optimization (GRPO) for one. For adapting weights cheaply, see fine-tuning a large language model (LLM).

Sources

1. Direct Preference Optimization: Your Language Model is Secretly a Reward Model (arXiv 2305.18290, abstract page) — arXiv (Rafailov, Sharma, Mitchell, Ermon, Manning, Finn), 29 Jul 2024
2. Direct Preference Optimization: Your Language Model is Secretly a Reward Model (arXiv 2305.18290, full text) — arXiv (Rafailov et al.), 29 Jul 2024
3. Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive (arXiv 2402.13228) — arXiv (Pal et al.), 3 Jul 2024