Fine-Tuning & LLM Training

How does GRPO compare to DPO?

By Interview Clozo · Published 27 October 2026 · Last verified 26 October 2026 · 13 min read

How does GRPO compare to DPO? GRPO (group relative policy optimization) learns from fresh groups of samples scored by a reward function; standard DPO (direct preference optimization) learns from fixed chosen-and-rejected pairs, with no sampling or separate reward model during training. The one case where a prompt gives GRPO no reward signal is a group whose rewards are all equal, and Step 2 shows the arithmetic.¹

In this article

Before you start
Why interviewers ask this
DPO and GRPO from first principles
The Staff-level answer
The rubric
Three variants of this question
Common mistakes
Follow-up questions
Sources

Before you start

GRPO and DPO are both steps in how large language models (LLMs) are trained, and both assume the transformer basics.

Why interviewers ask this

The question appears in interviews about how modern reasoning and chat models are tuned after pretraining. It separates candidates who memorized two acronyms from those who know where each method gets its data and its learning signal.

A weak answer says "both align the model with preferences." A strong answer says DPO reads stored pairs of a better and a worse answer, while GRPO samples several answers from the current model and compares them with one another. A Staff-level answer adds the consequences: what happens when the whole group scores the same, what the extra samples cost, and why neither method removes the need for a good reward or good pairs. The related DPO vs RLHF comparison covers the older contrast.

DPO and GRPO from first principles

Fixed-data training: Fixed-data training is a way of tuning a model in which every training example was collected before training began and stays unchanged while the model improves. The model never produces new examples during the run. Standard DPO works this way.
Live-sampling training: Live-sampling training is a way of tuning a model in which the current model generates new answers at each update, and those answers are scored and used for the next update. The data shifts as the model changes. GRPO works this way.
Group-relative advantage: A group-relative advantage is a number for one sampled answer that says how much better or worse its reward is than the average reward of all answers drawn for the same prompt, scaled by how spread out those rewards are.

In plain words

Picture a cooking class. In the first class, the teacher shows two finished dishes and says, "this one is better." Students learn from a fixed stack of such pairs. That is DPO.

In the second class, each student cooks eight versions of one dish today. A judge scores all eight. Each dish is then compared with the average of the eight. Dishes above average are copied. Dishes below average are avoided. That is GRPO.

The analogy breaks in one place. Real GRPO changes the model's probabilities of whole answers. It does not remember dishes.

In one precise sentence: DPO optimizes a loss on stored preference pairs, and GRPO optimizes a policy-gradient objective on its own samples, using the group's average reward as the baseline.

Step 1: GRPO samples a group and scores it

For each prompt, GRPO asks the current model for a group of G answers. G is the group size. In the DeepSeekMath setup, Shao et al. (2024) sample 64 outputs for each question. Each answer receives a reward, which can come from a reward model or, in later work, from a rule such as checking a final number. A reward model is a separate network that scores an output with a number.

Input: one prompt. Output: G answers, each with one reward. Nothing here needs a person to rank anything. The prompts need a way to be scored, and that is all.

The paper's table that compares training data groups DPO with methods that train on outputs from a fixed supervised model, and groups GRPO with methods that sample from the current policy. You now know the first difference: stored pairs versus fresh samples. The next step turns the rewards into a learning signal.

Step 2: The group z-score gives every sample an advantage

GRPO subtracts the group's average reward from each answer's reward and divides by the group's standard deviation. In the paper's words, rewards "are normalized by subtracting the group average and dividing by the group standard deviation" (Shao et al., 2024). The result is the group-relative advantage, and under outcome supervision every token of one answer shares it. For answer i with reward r_i, the formula is:

{\hat{A}}_{i} = \frac{r_{i} - mean (r)}{std (r)}

Take four answers with illustrative rewards 1, 0, 0 and 1. The mean is 0.5. With the population standard deviation, 0.5, the advantages are +1, −1, −1 and +1. With the sample standard deviation, 0.5774, they are about ±0.866.¹ The paper's text does not say which convention it uses, so state yours. Whether to divide by the standard deviation at all is contested; the second follow-up below gives the boundary.

Now the error candidates make. If every answer scores the same, the standard deviation is 0 and each numerator is 0. The advantage is then 0, or undefined without a small constant in the denominator, so the policy-gradient term for that prompt is zero and only the KL term still acts.¹

Step 3: A KL term and the update keep the policy near its reference

The update raises the probability of above-average answers and lowers it for below-average ones. GRPO keeps the clipped probability ratio of proximal policy optimization (PPO), which limits how far one update can move an answer's probability, and swaps only the baseline. A KL term holds the model near a starting copy. The reference policy is that frozen copy of the model, usually the supervised starting point. The KL divergence, defined above, is added to the loss.

PPO (proximal policy optimization) handles it differently. It puts a per-token KL penalty into the reward. GRPO "regularizes by directly adding the KL divergence between the trained policy and the reference policy to the loss" (Shao et al., 2024).

The biggest saving is the critic. A critic is a learned value model that estimates the baseline for each state, and in PPO it is "typically another model of comparable size as the policy model." GRPO "foregoes the critic model, instead estimating the baseline from group scores" (Shao et al., 2024). The DeepSeekMath run used a KL coefficient of 0.04, a learning rate of 1e-6 and a batch size of 1024 (Shao et al., 2024).

Step 4: DPO reads one fixed pair and needs no sampling

DPO takes a prompt, a chosen answer and a rejected answer. Using a frozen reference model, it computes how much more the tuned model favors the chosen answer than the reference does, minus the same quantity for the rejected one. The loss, as written by Rafailov et al. (2023) and conditioned on the prompt x, with σ the sigmoid function and β a scale factor, is:

- \log σ (β [(\log π_{θ} (y_{w} | x) - \log π_{ref} (y_{w} | x)) - (\log π_{θ} (y_{l} | x) - \log π_{ref} (y_{l} | x))])

Here x is the prompt, y_w the chosen answer and y_l the rejected one. Rafailov et al. show that the tuned model itself plays the role of a reward model, so no separate reward model is trained. With illustrative log-probabilities and β of 0.1, the margin is 0.2 and the loss is 0.5981, against 0.6931 (the natural log of two) before any training.¹

Standard DPO needs two answers per prompt and no live generation. That makes it cheap to run, and it limits it to the quality and coverage of the stored pairs. Online or iterative DPO variants, such as the one in Guo et al. (2024), instead resample pairs from the current model and label them again, which brings DPO closer to live sampling at a higher cost.

GRPO builds its learning signal from a fresh group of its own samples; DPO reads a stored pair.CC BY 4.0 — source: How does GRPO compare to DPO?

Step-by-step description

GRPO samples G answers per prompt.
Rewards become advantages.
A KL term anchors the policy.
DPO reads one fixed pair.

GRPO samples a group of answers for each prompt from the current model.
Each answer's advantage is its reward minus the group mean, divided by the group standard deviation.
A KL term in the loss keeps the policy close to the reference policy.
DPO reads a fixed pair of chosen and rejected answers instead.

GRPO samples a group of answers for each prompt from the current model. Each answer's advantage is its reward minus the group mean, divided by the group standard deviation. A KL term in the loss keeps the policy close to the reference policy. DPO reads a fixed pair of chosen and rejected answers instead.

GRPO and DPO differ in where the data comes from, what scores it and which extra models they need.
Dimension	GRPO	DPO
Training data	Fresh group per prompt from the current model	Stored chosen and rejected pairs
Learning signal	Reward minus group mean, scaled by group spread	Margin between tuned and reference log-ratios
Needs a reward function	Yes (model or rule)	Not during training
Needs a critic	No	No
Needs generation during training	Yes, G answers per prompt	No in standard DPO; online variants sample

A lone success shows how the group scale behaves. With one correct answer among G, its advantage under the population standard deviation is the square root of G minus 1. At G of 64, the paper's group size, that is 7.9373.¹

A lone success earns a larger advantage as the group grows, so rare wins stand out strongly.CC BY 4.0 — source: How does GRPO compare to DPO?

Text description of the diagram

With one success in a group, its advantage is the square root of G minus 1 under the population standard deviation.
At G of 64 that advantage is 7.9373.

Knowing it is half the job. Saying it is the other half.

Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.