What is DPO (direct preference optimization)?

In this article

1. Before you start
2. Why interviewers ask this
3. DPO from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page belongs to the path on how large language models are trained and builds on the transformer architecture.

Why interviewers ask this

Alignment questions separate candidates who can recite a method name from candidates who can say what it replaces. DPO is the usual test case: a strong answer explains why the reward model and the reinforcement learning loop can both be dropped, and a weak answer says "it is RLHF without the reward model" and stops.

A weak answer also claims that DPO has no reward at all, or that it is simply better than the older recipe. A strong answer states the objective DPO still optimizes, names the reference model and beta, and gives one honest limit, such as dependence on a fixed dataset. A Staff-level answer adds the evidence: the original paper reports wins at moderate scale, while an independent study found cases where a well-tuned reinforcement learning method did better. The comparison of DPO and RLHF goes through that contrast in more depth.

DPO from first principles

Reference model

A reference model is a frozen copy of the language model taken before preference training, usually the model after supervised fine-tuning, whose probabilities act as the fixed baseline that the trained model is compared with, so the loss measures how far the policy has moved from where it started.

Implicit reward

An implicit reward is a reward score that DPO never trains as a separate model but reads off the language model itself, as beta times the log of the probability ratio between the trained model and the reference model for a response.

In plain words

Imagine a competent chef, the supervised model. You have a notebook of taste tests where tasters picked the better of two dishes. The older recipe hires a food critic, the reward model, to learn the tasters' taste. Then the chef practices for the critic's scores. DPO skips the critic. It tells the chef to cook more like the winning dish and less like the losing one. Progress is measured against how the chef cooked before. The chef must not drift far from the original style. The analogy breaks in one place: the chef only gets feedback on dishes already in the notebook.

In the technical terms used below, the policy is the language model being trained, and the reference is its frozen copy.

In one precise sentence, DPO minimizes a logistic loss on the difference between two log-ratios, each comparing the model being trained with a frozen reference model, one for the chosen and one for the rejected response. Rafailov et al. introduced it, and NeurIPS published it in 2023 (arXiv 2305.18290).

Step 1: A frozen reference model and preference pairs set the starting point

DPO starts from two ingredients: a frozen reference model and a dataset of preference pairs. A preference pair is a prompt together with one chosen and one rejected response, where a person or an AI judge marked which response is better. In the paper's pipeline, completions are sampled from the reference model for every prompt and labeled with preferences to build the dataset.

The reference model is initialized as the supervised model when one exists. Otherwise the paper fits it by maximizing the likelihood of the preferred completions. The Hugging Face Transformer Reinforcement Learning (TRL) documentation likewise says to train the supervised model first, so the data is in-distribution, and it expects columns named prompt, chosen and rejected.

Input: a prompt with two responses. Output: a frozen baseline and a training set. A frozen reference model and a trainable policy each score the chosen and rejected answers; the next step turns those scores into one number.

Step 2: The implicit reward is a log ratio against the reference model

DPO defines the reward of a response as beta times the log of how much more likely the trained model makes it than the reference model does. This step is the payoff of the whole method, and it comes from the older objective.

The RLHF objective asks for the policy that maximizes expected reward minus beta times a penalty for drifting from the reference. The penalty uses the Kullback-Leibler (KL) divergence. KL divergence is a measure of how far one probability distribution is from another. Beta sets how strongly the policy is pulled back toward the reference, and higher beta means less divergence.

That objective has a closed-form best policy, where Z(x) is a normalizer that depends only on the prompt (the paper calls it the partition function):

Rearranged, the reward is the implicit reward plus a prompt-only term (Section 4 and Appendix A.1 of the paper):

The first part is beta times the log ratio of policy to reference. The second part is the same for every response to a prompt.

Step 3: A logistic loss on the reward gap replaces the reward model

DPO replaces reward-model training with a logistic loss on the gap between the chosen and rejected implicit rewards. The pairwise-choice model called the Bradley-Terry model says that the chance a person prefers response A over response B is the sigmoid of the difference between their rewards:

Substitute the rewritten reward into that model. Both responses answer the same prompt, so their normalizer Z(x), the partition function, is identical and the paper states that the partition function cancels. What remains depends only on the policy and the reference:

Here y_w is the chosen response, y_l the rejected one, and π_θ the policy. The paper's reference code writes it as losses = -F.logsigmoid(beta * (pi_logratios - ref_logratios)). This is a binary cross-entropy on preference pairs, so it needs no reward model and no sampling during training (see the reference code in the paper).

Step 4: The gradient raises the chosen answer relative to the rejected one

The DPO gradient raises the log-probability of the chosen response relative to the rejected one, weighting each pair by how wrongly the implicit reward currently ranks it. In the paper's words, "the DPO update increases the relative log probability of preferred to dispreferred responses" (Rafailov et al., 2023). The loss constrains only that gap, so the chosen answer's own probability is not forced to rise; it can fall if the rejected answer falls further.

The weight is the sigmoid of the rejected implicit reward minus the chosen one. It is one half when the policy still equals the reference, and it grows when the policy ranks a pair the wrong way round. The paper reports that without this weighting a naive version can degenerate.

After training, the policy is the only model you keep. As the paper puts it, "the policy network represents both the language model and the (implicit) reward." You now have the whole method: a reference, a log-ratio reward, a logistic loss and a weighted update.

1 · Pair and reference2 · Implicit reward3 · Logistic loss4 · UpdateReplay

Step-by-step description

1. A frozen reference and a policy score each pair.
2. Reward is beta times a log ratio.
3. The reward gap becomes a loss.
4. Chosen rises relative to rejected.

• A frozen reference model and a trainable policy each score the chosen and rejected answers.
• DPO defines the reward of a response as beta times the log of how much more likely the trained model makes it than the reference model does.
• DPO replaces reward-model training with a logistic loss on the gap between the chosen and rejected implicit rewards.
• The update raises the chosen answer's likelihood relative to the rejected one's.

Worked example

Take one preference pair and a beta of 0.1, the paper's default setting. The log-probabilities are illustrative. For the chosen response the reference model gives -42.0 and the policy gives -40.5; for the rejected response the reference gives -38.0 and the policy gives -39.5.¹ Training has raised the chosen response by 1.5 and lowered the rejected one by 1.5 in log-probability.¹

Multiply each shift by beta. The implicit reward of the chosen response is 0.15 and of the rejected response is -0.15, a gap of 0.3, so the loss is -log σ(0.3) = 0.5544.¹ At initialization the policy equals the reference, both rewards are zero, and the loss is -log σ(0) = ln 2 = 0.6931.¹ The gradient weight is 0.5 at initialization and 0.4256 after this update.¹ If the policy had ranked the pair the wrong way round (chosen -43.5, rejected -36.5, illustrative), the gap would be -0.3, the loss 0.8544 and the weight 0.5744, so wrongly ranked pairs are pushed hardest.¹

Now keep the same shifts and change only beta. At beta 0.05, 0.1, 0.5 and 1 the loss is 0.6209, 0.5544, 0.2014 and 0.0486 (illustrative).¹ A larger beta multiplies the same log-probability shift into a larger reward gap, so the loss saturates sooner and the policy has less reason to move far from the reference.

Text description of the diagram

• At beta 0.05, 0.1, 0.5 and 1 the loss is 0.6209, 0.5544, 0.2014 and 0.0486 (illustrative).
• A larger beta multiplies the same log-probability shift into a larger reward gap, so the loss saturates sooner.

The memory picture is why DPO is called cheaper. Take a 7-billion-parameter model stored in 16-bit numbers at 2 bytes per parameter, so one copy of the weights is 14 GB (illustrative). DPO holds two copies, the policy and the reference, which is 28 GB of weights.

Reinforcement learning with proximal policy optimization (PPO) holds more: besides the policy and the reference it needs a reward model and a value model, and the TRL documentation notes that PPO expects a model with a value head, which DPO does not. A value model is a second network that estimates expected return and is used to stabilize PPO updates. Optimizer states and activations come on top of both, and parameter-efficient fine-tuning shrinks the trainable part further.

Trade-offs and edge cases

DPO trades the flexibility of reinforcement learning for simplicity, and the evidence on whether the trade is worth it is mixed. The authors report that DPO has a win rate of approximately 61% at temperature 0.0 on summarization, exceeding PPO at 57%, and that DPO samples at temperature 0.25 were preferred 58% of the time over PPO samples at temperature 0 in a human study. They also note that they evaluated models only up to 6B parameters.

An independent study finds the picture less one-sided. Xu et al. (2024) show that "DPO may have fundamental limitations", that it can find biased solutions exploiting out-of-distribution responses, that it is sensitive to the shift between the model's outputs and the preference data, and that a tuned PPO surpassed other methods on their benchmarks, including code contests. The honest summary is "simpler and cheaper, not always better."

Text description of the diagram

• DPO needs no separate reward model and no sampling during training.
• PPO-based RLHF also holds a reward model and a value model.

DPO needs no separate reward model and no sampling during training, and the original authors call RLHF a complex and often unstable procedure (see the paper abstract). PPO-based RLHF also holds a reward model and the value model described above. The DPO trainer, by contrast, expects a plain causal language model rather than one with a value head (see the TRL v0.8.0 trainer page).

Edge cases follow from the fixed dataset. If the pairs came from a different model than the one you train, the data is out-of-distribution, and quality can suffer. Online DPO samples fresh answer pairs from the policy during training and labels them with a reward model or a judge, so it needs only prompts as data but brings back a scorer, much as group relative policy optimization samples groups of answers. Noisy labels are handled by variants that soften the target; TRL exposes them through its loss_type option and a label_smoothing setting (see the TRL loss options). Preference training can also cost general skill, the alignment tax.

The default settings quoted here are those documented for the TRL releases v0.8.0 and v0.9.6 (2024): a default beta of 0.1 and sigmoid as the default loss, with hinge, identity preference optimization (IPO) and others as alternatives through loss_type. They are version-specific facts, so check the release you install. As of October 2026, the TRL quickstart shows DPOTrainer being built from a model name and a preference dataset, and the online DPO page documents the prompt-only variant.

Reference-free variants of preference optimization also exist and drop the frozen reference that standard DPO requires. Simple preference optimization (SimPO, Meng et al., 2024) uses the average log-probability of a response as its implicit reward and needs no reference model. Odds ratio preference optimization (ORPO) (Hong et al., 2024) is described by its authors as a "reference model-free monolithic odds ratio preference optimization algorithm".

How to answer it in an interview

The spoken structure: say what it trains on, name the reference model, give the loss in one sentence, then state the trade-off and what beta does.

Common mistakes

• "DPO has no reward model." (wrong) It has no separate reward model, but it fits an implicit reward. (right) "The reward is beta times the log ratio of policy to reference."
• "DPO is reinforcement learning." (wrong) It is a supervised, binary cross-entropy loss on preference pairs with no sampling loop. (right) "It optimizes the same KL-constrained objective without running reinforcement learning."
• "DPO does not need a reference model." (wrong) The loss compares the policy with a reference, usually the supervised model. (right) "The reference is frozen, and beta sets how far the policy may leave it."
• "A higher beta means more aggressive optimization." (wrong) Higher beta means less divergence from the reference. (right) "Beta is the strength of the pull back toward the reference."
• "DPO is proven better than PPO." (wrong) Results differ by benchmark and scale. (right) "It is simpler and cheaper, and one study found a tuned PPO ahead."

Follow-up questions

What is the difference between DPO and RLHF?

RLHF trains a reward model on preferences and then runs reinforcement learning against it with a KL penalty. DPO optimizes the same objective with one classification loss on the preference pairs and no reward model or sampling loop. The DPO versus RLHF page lays out the contrast.

Which paper introduced DPO, and where is it on arXiv?

Rafailov, Sharma, Mitchell, Ermon, Manning and Finn introduced it in arXiv 2305.18290, submitted on 29 May 2023, with the latest revision, v3, on 29 Jul 2024. It appeared at NeurIPS 2023.

Where does the DPO loss come from, in a short derivation?

Start from the KL-regularized reward objective, solve for the best policy in closed form, rearrange that into a reward written as a log ratio plus a prompt-only term, and plug it into the Bradley-Terry model. The prompt-only term cancels, leaving the loss in Step 3.

What does beta do in DPO?

Beta controls how far the policy may deviate from the reference model; higher beta means less divergence. The paper's default is 0.1, with 0.5 for its summarization task (see the full paper), and the TRL default is also 0.1 (TRL v0.9.6).

Is DPO better than PPO?

Not always. The original paper reports DPO matching or beating PPO at up to 6B parameters, while Xu et al. (2024) report a well-tuned PPO ahead on their benchmarks.

Does DPO need a reward model or a reference model?

Standard DPO needs a frozen reference model and no separate reward model; some later variants, such as SimPO, drop the reference. The TRL trainer page says the reference model is ignored as beta approaches zero.

Sources

1. Direct Preference Optimization (arXiv 2305.18290, abstract page) — arXiv (Rafailov et al., Stanford), 29 Jul 2024
2. Direct Preference Optimization (arXiv 2305.18290 v3, full paper) — arXiv / NeurIPS 2023 (Rafailov et al.), 29 Jul 2024
3. Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study (arXiv 2404.10719) — arXiv (Xu et al.), 16 Apr 2024
4. DPO Trainer (TRL v0.9.6 documentation) — Hugging Face, 26 Oct 2026
5. DPO Trainer (TRL v0.8.0 documentation) — Hugging Face, 26 Oct 2026
6. TRL quickstart (Hugging Face documentation, read October 2026) — Hugging Face, 26 Oct 2026
7. Online DPO Trainer (TRL documentation) — Hugging Face, 26 Oct 2026
8. SimPO: Simple Preference Optimization with a Reference-Free Reward (arXiv 2405.14734) — arXiv (Meng et al.), 23 May 2024
9. ORPO: Monolithic Preference Optimization without Reference Model (arXiv 2403.07691) — arXiv (Hong et al.), 12 Mar 2024