DPO vs RLHF: what is the difference?

In this article

1. Before you start
2. Why interviewers ask this
3. What each one is
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

Why interviewers ask this

The wrong choice here wastes a training budget, and a candidate who cannot explain it has usually only copied a recipe. Interviewers raise it as "DPO or proximal policy optimisation (PPO) for alignment?", as "derive the DPO loss" or as "why does DPO need no reward model?". All three test one idea: preference tuning is a constrained optimisation, and the two methods solve the same objective by different routes.

A weak answer says DPO is "RLHF without reinforcement learning" and stops. A strong answer names the preference data, the reference model, the log-ratio that replaces the reward, the role of beta, and what DPO gives up: it never samples from the model it trains. Interviewers use the topic to separate people who have read the paper from people who have only run a trainer.

What each one is

DPO

Direct preference optimisation (DPO) is a training method that aligns a language model to human preferences by minimising a single classification-style loss on pairs of responses, with no separate reward model and no sampling from the model during training.

Value function

A value function is a network that estimates the reward a model can still expect to collect from its current state, which RLHF with PPO uses as a baseline so the policy update reflects how much better an answer was than expected.

RLHF in plain words

RLHF is a three-stage recipe. The model first copies written examples, a reward model then learns to imitate human rankings, and the language model is pushed toward answers the reward model scores highly. Ouyang et al. (2022) name the stages. They are supervised fine-tuning on written demonstrations (SFT), a reward model trained on human rankings, and reinforcement learning (RL) with PPO against that reward model. The policy, meaning the language model being trained, writes answers. The reward model scores them. PPO nudges the policy toward higher scores.

The KL penalty is a per-token cost on the distance from the SFT model. It stops the policy from drifting into answers that fool the reward model. The value function is a network that estimates the reward still to come. The paper initialises it from the reward model. The result was clear. Raters preferred outputs of a 1.3 billion parameter InstructGPT, a language model built on the transformer architecture, to those of the 175 billion parameter generative pre-trained transformer (GPT) model GPT-3.

DPO in plain words

DPO is RLHF's objective solved on paper. Rafailov et al. (2023) observe that the best policy for the reward-with-KL-penalty objective has a closed form. The reward can be rewritten as beta times the log-ratio of the policy to the reference model, plus a term that depends only on the prompt. Plug that into the Bradley-Terry model. The Bradley-Terry model is a model of human preference in which the chance a response wins is the sigmoid of its reward minus the other's.

The prompt-only term cancels, so the reward model disappears. What is left is a log-ratio loss, as in cross-entropy loss. Gradient descent minimises it. RLHF hires a marker and practises against it; DPO studies the marked essays and writes none. The analogy breaks here: DPO still has a marker of sorts, the implicit reward from the log-ratio against the reference model.

Text description of the diagram

• RLHF fits a reward model on the pairs and then runs PPO with a KL penalty.
• DPO trains on the pairs directly with one loss and keeps a reference model.

The diagram shows the difference in one picture. RLHF fits a reward model on the pairs and then runs PPO with a KL penalty; DPO trains on the pairs directly with one loss and keeps a reference model.

Worked example

One preference pair, run through the DPO loss by hand. Every number comes from a script, and the four sequence log-probabilities are illustrative, not measured.¹

The policy gives the chosen response a log-probability of −42.0 and the rejected one −60.0. The reference model gives −45.0 and −58.0. The log-ratio is policy minus reference: 3.0 for the chosen response, −2.0 for the rejected one. With beta at 0.1, the default in the Hugging Face Transformer Reinforcement Learning (TRL) DPO trainer page (read Oct 2026), the implicit rewards are 0.3 and −0.2, a margin of 0.5.

The loss is minus the log of the sigmoid of the margin (the sigmoid squashes the margin into 0 to 1): 0.474.¹ At the very start the policy equals the reference, both log-ratios are zero, the margin is zero, and the loss is minus the log of 0.5, which is the natural log of 2, or 0.693. That is why a standard DPO run, whose policy starts as a copy of the reference, begins there.

The weight is the sigmoid of minus the margin, the factor by which the loss gradient scales this pair relative to the log-ratio gap (the full gradient also carries a factor of beta); pairs the model already orders correctly by a wide margin fade out.¹ The TRL page describes beta as the parameter controlling deviation from the reference model, where a higher value means less deviation. The same gap of 5.0 in log-ratio is a large lead when beta is 1 and almost nothing when beta is 0.01.

Text description of the diagram

• The same gap of 5.0 in log-ratio is a large lead when beta is 1 and almost nothing when beta is 0.01.
• Pairs the model already orders correctly by a wide margin fade out.

Now the memory bill, again illustrative: take four models of 7 billion parameters held in 2-byte bfloat16 (a 16-bit number format) weights. A PPO-style RLHF run needs the policy, the reference, the reward model and the value function, 56 GB of weights alone. DPO needs the policy and the reference, 28 GB. The TRL page notes that the reference log-probabilities can be precomputed over the dataset so the reference model need not stay in memory, leaving 14 GB.¹ These figures exclude optimiser states and activations, which add to the total, so read them as a ratio, not a budget.

The reading: DPO trades the reward model and the sampling loop for a fixed dataset and one sigmoid.

Deriving the DPO loss in four lines

The route of Rafailov et al. (2023), section 4, Eqs. 4 to 7. Here $r$ is the reward, $\pi$ the policy and ${\pi }_{\text{ref}}$ the reference model.

Step 1. The objective RLHF optimises is reward minus a Kullback-Leibler (KL) penalty toward the reference model:

Step 2. Its best solution has a closed form, where $Z(x)$ is the partition function, a normalising sum over all possible responses that makes the probabilities add to 1:

Step 3. Solve that for the reward:

Step 4. Under the Bradley-Terry model the chance that the chosen response $y_w$ beats the rejected one $y_l$ is $\sigma (r(x,y_w)-r(x,y_l))$, a function of the reward difference. In that difference the $\beta \log Z(x)$ terms cancel. Writing ${\mathrm{\Delta }}_w=\log \frac{\pi (y_w|x)}{{\pi }_{\text{ref}}(y_w|x)}$ and ${\mathrm{\Delta }}_l$ likewise for $y_l$, and replacing ${\pi }^{\ast }$ by the trainable $\pi$, the loss is the negative log of that probability:

The partition function $Z(x)$ depends only on the prompt, so it is identical for the chosen and the rejected response and subtracts away: that is where it went, and why no reward model is needed. At the start $\pi ={\pi }_{\text{ref}}$, so ${\mathrm{\Delta }}_w={\mathrm{\Delta }}_l=0$ and the loss is $-\log \sigma (0)=-\log 0.5=0.693$¹.

Trade-offs and edge cases

The two methods differ in what they need, what they hold in memory and what they can learn from, and each difference has a source.

Sources for each row: Ouyang et al. (2022) and its appendix for the left column, Rafailov et al. (2023) and the TRL page for the right.¹

What DPO gives up. It learns only from the pairs it is handed, so it never tries its own answers and never scores them. Xu et al. (2024) report that DPO is not clearly superior to PPO, state that "DPO may have fundamental limitations", and find PPO best in every testbed they ran, from dialogue to code generation.

Reported practice differs: the DPO authors report that it exceeds PPO-based RLHF on sentiment control and matches or improves quality on summarisation and single-turn dialogue. The two papers use different tasks and model sizes, so neither settles the question for your task. Group relative policy optimisation (GRPO), a PPO variant from Shao et al. (2024), drops PPO's value model and estimates the baseline from the scores of a group of sampled answers, which cuts training memory.

Both methods anchor to a reference model. InstructGPT adds the KL penalty "to mitigate overoptimization of the reward model". DPO has no reward model to exploit, but its beta plays the same anchoring role, and TRL's loss-type list includes a variant that normalises by response length to counter the length bias of the original loss, as the TRL page puts it, which matters when training a large language model (LLM) on long answers.

Which one to start with. The decision flow below is our reading of the evidence above, not a quoted rule.

Text description of the diagram

• If you can score fresh samples with a reward model or a checker, consider PPO, or run DPO first if compute is tight.
• If you cannot score fresh samples, you hold only fixed pairs, so use DPO on them.

The flow starts at the SFT-only baseline, then asks one question: can you score fresh samples from the model, with a reward model or a checker such as unit tests? If not, you hold only fixed pairs, so use DPO on them. If yes, consider PPO, as Xu et al. found for code, or run DPO first if compute is tight. Both branches leave the question, so the rule is a fork, not a chain.

How to answer it in an interview

The spoken structure is: same objective, the two routes, the pick, then the caveat.

Common mistakes

"DPO is supervised fine-tuning on the chosen answers." (wrong) The loss compares chosen and rejected responses against a reference model, and TRL notes it typically works by suppressing the rejected completions. (right) "DPO is a classification loss on log-ratios, not plain likelihood training."

"DPO has no reward." (wrong) The reward is there, but implicit: beta times the log-ratio. (right) "DPO has no separate reward model."

"Drop the reference model, it is just a regulariser." (wrong) It defines the ratio itself; without it the loss changes meaning. (right) "The reference anchors both the ratio and the KL constraint."

"DPO is always better, it is newer." (wrong) Xu et al. found PPO ahead on their testbeds. (right) "It is simpler; whether it is better depends on the task."

Follow-up questions

DPO vs PPO: what is the difference?

PPO is the reinforcement-learning algorithm inside RLHF: it samples answers from the policy, scores them with a reward model and updates with a clipped objective, which removes the incentive to move the policy far in one update. DPO has no sampling loop and no reward model; it trains on stored pairs. Schulman et al. (2017) describe PPO as alternating between collecting data by interaction and optimising a surrogate objective.

SFT vs DPO vs PPO: where does each sit in the pipeline?

SFT comes first and gives the reference model. DPO or PPO then comes after it: DPO consumes preference pairs directly, while PPO consumes samples scored by a reward model fitted to those pairs. The fine-tuning hub covers the whole lifecycle.

Does DPO use the Bradley-Terry model?

Yes. TRL's documentation describes the default sigmoid loss as fitting a binary classifier according to the Bradley-Terry model. Other loss types, such as the hinge loss, change that link.

When would you still pick PPO?

Pick PPO when you can score fresh samples, for example with unit tests for code, because Xu et al. found it strongest in code competitions. It also lets the policy explore answers that no stored pair contains.

What does beta do in the DPO loss?

Beta scales the log-ratio inside the sigmoid. A higher beta keeps the model closer to the reference, as the TRL page describes, and in the worked example a larger beta drives the loss to zero faster for the same pair.

Sources

1. Direct Preference Optimization: Your Language Model is Secretly a Reward Model — arXiv (Rafailov et al.), v3 revised 2024-07-29, 29 May 2023
2. Training language models to follow instructions with human feedback — arXiv (Ouyang et al.), 4 Mar 2022
3. Proximal Policy Optimization Algorithms — arXiv (Schulman et al.), v2 revised 2017-08-28, 20 Jul 2017
4. Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study — arXiv (Xu et al.), ICML 2024, v3 revised 2024-10-10, 16 Apr 2024
5. DPO Trainer — Hugging Face TRL documentation (read 2026-10-24), 24 Oct 2026
6. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models — arXiv (Shao et al.), v3 revised 2024-04-27, 5 Feb 2024