Why has DPO been replacing PPO?

In this article

1. Before you start
2. Why interviewers ask this
3. Preference training from first principles
4. The Staff-level answer
5. The rubric
6. Three variants of this question
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This question sits in the large language model training path, after pretraining and before evaluation.

Why interviewers ask this

Interviewers ask why DPO has been replacing PPO to see whether a candidate understands the mechanism of preference training, not only the acronyms. The question is reported by candidates as a follow-up to "walk through RLHF," and it separates people who can name the stages from people who can say what each stage costs.

A weak answer says "DPO is simpler and better." A strong answer says what is removed (the separate reward model, the value network and the sampling loop), what is kept (preference pairs, a reference model and a KL-style control), and what is lost. A Staff-level answer also knows that "replacing" is a statement about convenience and adoption, not a proof of quality: a comparative study found that a well-tuned PPO can beat DPO (Xu et al., 2024).

Preference training from first principles

Value network

A value network is a model trained alongside the policy in PPO that estimates the expected reward of a state, which lets the update judge whether an answer was better or worse than usual, at the cost of an extra model to train.

In plain words

Imagine teaching a writer by showing two drafts and saying which you prefer. The PPO route hires a critic to learn your taste. The writer then produces new drafts again and again, and the critic scores them. The DPO route skips the critic and the new drafts. It reads the pile of "this draft beat that one" judgments and nudges the writer toward the winners. The analogy breaks in one place. DPO's writer is still measured against an earlier version of itself. A hidden critic exists in the math, though nobody trains it.

In one sentence, DPO and PPO-based RLHF both raise the probability of preferred answers while keeping the model near a starting model. They differ in how.

Step 1: RLHF with PPO trains a reward model, then runs a sampling loop

RLHF with PPO has three stages: supervised fine-tuning (SFT) on example answers, a reward model trained on human preferences (the learned scorer defined above), and a reinforcement-learning stage (Ouyang et al., 2022).

In the last stage, the model samples answers, the reward model scores them, and PPO updates the model. PPO limits how far one update can push the probability ratio between new and old policy (Schulman et al., 2017). A KL penalty against the starting model discourages drift into answers that fool the reward model.

In a standard setup four roles are live at once: the policy being trained, a frozen reference copy, the reward model and a value network that estimates expected reward. Some implementations share weights between roles, so the count can be lower. Input: preference pairs and prompts. Output: a tuned policy. You now know the cost DPO attacks: two extra models and a loop that must generate text during training.

Step 2: DPO hides the reward inside a log-ratio and drops the sampling loop

DPO, introduced by Rafailov et al. (2023), rests on one observation: the KL-regularized reward objective of RLHF has a closed-form optimum, so the reward can be written in terms of the policy itself, with no sampling from the model during training.

That rewritten reward is called the implicit reward: beta times the log-ratio of the policy's probability to the reference model's probability for an answer. The full expression also holds a prompt-only normalising term (beta times the log of a partition function) that cancels when two answers to the same prompt are compared. Beta is the number that sets how strongly the policy is held near the reference model, the frozen starting copy defined above.

So DPO does have a reward; it is computed from two log probabilities rather than learned by a third network. Input: a prompt with a chosen and a rejected answer. Output: a loss. The next step shows that loss on one pair, with numbers.

Step 3: The DPO loss is a classification step on one preference pair

For each pair, DPO measures how much the policy has moved the log probability of each answer relative to the reference, then asks whether the chosen answer moved up more than the rejected one.

Here d_w is the policy's log probability of the chosen answer minus the reference's, d_l is the same for the rejected answer, and σ is the sigmoid function, which squeezes any number into the range zero to one.

Take illustrative log probabilities: the policy gives the chosen answer -12.0 and the rejected answer -15.0, while the reference gives -13.0 and -14.0. Then d_w is 1.0 and d_l is -1.0.¹ With beta of 0.1 the margin is 0.2 and the loss is 0.5981; with beta of 0.5 the margin is 1.0 and the loss is 0.3133.¹ With beta 0.5 the loss is already 0.3133 after the same small shift, so the optimiser has less reason to push further from the reference. When the policy equals the reference, the margin is zero and the loss is 0.6931, the natural log of two.¹

A loss near 0.69 means the pair is not yet separated. Training lowers it by raising the chosen answer's relative probability.

Step 4: DPO saves memory, sampling and tuning effort

DPO saves memory because only the policy and a frozen reference sit in memory (some setups precompute the reference log probabilities, so the live count can be fewer), and it saves time because no answers are generated during training. The Hugging Face Transformer Reinforcement Learning (TRL) documentation describes a trainer that takes a dataset with prompt, chosen and rejected columns and uses a reference model.

For a count of weights alone, take an illustrative 7-billion-parameter model stored in bfloat16 (brain floating point, 16-bit), which uses 2 bytes per parameter. One copy is 14 GB (gigabytes). DPO holds two copies, 28 GB; a four-role PPO setup with same-size reward and value models holds four, 56 GB.¹ Optimizer state and generated text add more to PPO, and that is not counted here.

The memory saving is only half of the practical story: a loss on stored pairs is also easier to debug, to rerun and to combine with parameter-efficient methods such as low-rank adaptation (LoRA) than a loop that depends on a learned reward and a sampler.

Text description of the diagram

• DPO holds two copies, 28 GB.
• A four-role PPO setup with same-size reward and value models holds four, 56 GB.

Step 5: Cheaper is not the same as better

DPO is cheaper to run than PPO, but the evidence does not show that it is stronger. Xu et al. (2024) note that applications such as ChatGPT and Claude use reward-based methods with PPO, while state-of-the-art academic benchmark results are often reached by reward-free methods such as DPO. So "replacing" describes open and academic post-training, and adoption rather than a proven win: their own study argues that a well-tuned PPO can outperform DPO, including on code-generation tasks.

Two reasons explain the gap between convenience and quality. DPO learns only from the fixed pairs it is given, so it cannot explore answers the data never contained, while PPO samples fresh answers from the current model and scores them. And DPO is not free of control knobs: beta and the reference model still decide how far the policy may move, and a poor choice shows up as a model that barely changes or one that drifts.

The 2025 state: for reasoning, work swung back to online reinforcement learning with verifiable rewards, meaning scores a program can check, such as whether a final answer matches or code passes tests. Group relative policy optimization (GRPO), introduced in DeepSeekMath (Shao et al., 2024), was used in DeepSeek-R1 (DeepSeek-AI, 2025). Reach for DPO with a good pair dataset and limited compute; consider PPO or GRPO when you can afford online sampling and the task has a checkable signal.

1 · RLHF with PPO2 · Implicit reward3 · Pair loss4 · Two copies5 · Not always betterReplay

Step-by-step description

1. A reward model, then a sampling loop.
2. DPO writes the reward as a log-ratio.
3. One classification-style loss per pair.
4. Policy and reference only.
5. A tuned PPO can still win.

• RLHF with PPO trains a reward model, then runs a sampling loop against it.
• DPO writes the reward as a log-ratio of policy to reference.
• DPO computes one classification-style loss on each chosen and rejected pair.
• DPO keeps two model copies in memory, the policy and the reference.
• A well-tuned PPO can still beat DPO.

The five steps in one pass: RLHF with PPO trains a reward model, then runs a sampling loop against it. DPO writes the reward as a log-ratio of policy to reference. DPO computes one classification-style loss on each chosen and rejected pair. DPO keeps two model copies in memory, the policy and the reference. A well-tuned PPO can still beat DPO.

The Staff-level answer

What makes this Staff-level is that it names what DPO removes, what it keeps, and the evidence against "DPO is simply better." It also lands in about a minute.

The rubric

A strong answer shows the mechanism and the trade-off, not only the headline. The table below is advice on what to show, not a scoring sheet from any company.

Three variants of this question

Walk through RLHF, then explain DPO and why it has largely displaced PPO-based RLHF

Give the three RLHF stages first, then the closed-form rewrite, then the cost saving and the caveat. This is the full answer above, and the "largely displaced" wording is worth softening to "widely adopted," since the evidence on quality is mixed.

Is DPO better than PPO?

Say it depends. DPO is simpler and cheaper, while a comparative study found tuned PPO can come out ahead (Xu et al., 2024). Name the conditions: pair quality, compute budget and whether online sampling helps the task.

What does DPO remove from the RLHF pipeline?

It removes the separately trained reward model, the value network and the generation loop during training. It keeps the preference data, the reference model and a control strength, beta. For the broader comparison, see DPO versus RLHF.

Common mistakes

• "DPO has no reward model, so it is not reinforcement learning from preferences at all." (wrong) Why: the reward exists implicitly as a log-ratio. "DPO has an implicit reward, beta times the log-ratio of policy to reference, and no separately trained reward model." (right)
• "DPO needs no preference data." (wrong) Why: it still needs chosen and rejected pairs. "DPO removes the reward model, not the preference data." (right)
• "DPO has no KL control." (wrong) Why: control comes through beta and the reference model. "Beta sets how far the policy may move from the reference." (right)
• "DPO is strictly better than PPO." (wrong) Why: a comparative study argues tuned PPO can win. "DPO is cheaper and simpler; quality depends on the data and the tuning." (right)

Follow-up questions

What does beta do in DPO?

Beta is the strength of the pull toward the reference model. Because beta multiplies the log-ratio gap, a larger beta makes the same gap count more in the loss: the margin is 0.2 at 0.1 and 1.0 at 0.5.¹ The sigmoid therefore saturates after a smaller change from the reference, gradients fade earlier and the policy moves less, which is why a larger beta means a stronger constraint, not more freedom to drift. Tuning it is part of making DPO work.

Does DPO still need preference data?

Yes. DPO trains on prompts with a chosen and a rejected answer, so the cost of collecting or generating good pairs remains. What it removes is the step of training a reward model on those pairs.

When would you still choose PPO?

Choose PPO when you can afford online sampling, when the model must explore answers the stored pairs never contained, or when you already have a trusted reward signal. Xu et al. (2024) report cases where tuned PPO beats DPO. For the algorithm itself, see PPO in reinforcement learning.

Where does GRPO fit in?

GRPO keeps online sampling but drops the value network by scoring a group of sampled answers against each other (Shao et al., 2024). It is a different trade-off from DPO; the GRPO page covers it, and the RLHF page covers the pipeline both are compared to.

How does this fit with full fine-tuning?

Preference training usually follows supervised fine-tuning; see fine-tuning a large language model (LLM) for the memory picture that both methods share.

Sources

1. Direct Preference Optimization: Your Language Model is Secretly a Reward Model (arXiv 2305.18290) — arXiv (Rafailov et al.), 29 May 2023
2. Training language models to follow instructions with human feedback (arXiv 2203.02155) — arXiv (Ouyang et al., OpenAI), 4 Mar 2022
3. Proximal Policy Optimization Algorithms (arXiv 1707.06347) — arXiv (Schulman et al.), 20 Jul 2017
4. Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study (arXiv 2404.10719) — arXiv (Xu et al.), 16 Apr 2024
5. Hugging Face TRL documentation: DPO Trainer — Hugging Face, 26 Oct 2026
6. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (arXiv 2402.03300) — arXiv (Shao et al.), 5 Feb 2024
7. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (arXiv 2501.12948) — arXiv (DeepSeek-AI), 22 Jan 2025