In this article
  1. Before you start
  2. Why interviewers ask this
  3. GRPO from first principles
  4. Worked example
  5. Trade-offs and edge cases
  6. How to answer it in an interview
  7. Common mistakes
  8. Follow-up questions
  9. Sources

Before you start

Why interviewers ask this

GRPO is the question that shows whether a candidate understands what reinforcement learning actually changes in a language model, not only which acronym a lab used. A weak answer says "it is proximal policy optimization (PPO) without the critic" and stops. A strong answer explains what replaces the critic, why that replacement works, what the group's standard deviation does to the update, and where the method is known to bias training.

The question appears in large language model (LLM) training and alignment rounds, usually as "explain PPO and GRPO and why one drops the value model". The fine-tuning hub places it after supervised fine-tuning and preference methods such as direct preference optimization (DPO) versus reinforcement learning from human feedback (RLHF).

GRPO from first principles

GRPO
Group relative policy optimization (GRPO) is a reinforcement-learning algorithm that updates a language model using, for each prompt, a group of sampled answers whose rewards are compared with the group's own mean instead of with a learned value model.
Group
A group is the set of G answers the current model samples for one prompt, which serve together as the baseline for scoring each of them.

In plain words

Picture a teacher who sets one question and asks eight students to answer. Instead of grading against a fixed answer key of expected scores, she compares each student with the class that day. A student who is right when most are wrong stands out. A student who is right when everyone is right earns nothing extra. The analogy breaks in one place. The eight answers all come from one model, and it is that model, not a class, that learns after every update.

Stated precisely, GRPO is a variant of proximal policy optimization (PPO). PPO updates a model's policy, meaning its rule for choosing the next token, with a clipped objective. The DeepSeekMath authors describe GRPO as a PPO variant that improves mathematical reasoning "while concurrently optimizing the memory usage of PPO" (Shao et al., 2024). The change is where the baseline comes from.

Step 1: Sample a group of answers

The first step draws G answers to the same prompt from the current model. The input is one prompt; the output is a group of G complete answers. In DeepSeekMath's reinforcement-learning run the authors sampled 64 outputs for each question (Shao et al., 2024, section 4.2). The sampling uses the policy as it stood before this round of updates, called the old policy, because the objective later compares the new policy with it. The group has to be large enough that its average is a usable baseline, because a mean over two answers is mostly noise. A larger group costs more generation time. What you now know: one prompt becomes a group of G answers.

Step 2: Score every answer

The second step gives each answer a reward. The input is the group; the output is G numbers. A reward model is a network trained to score how good an answer is. The reward can come from such a model, as in DeepSeekMath, or from a rule that checks the answer, such as an exact match on a math result.

The DeepSeek-R1 authors argue that pure reinforcement learning on verifiable tasks can encourage reasoning without human-labelled reasoning traces (DeepSeek-AI, 2025). In this article a reward of one means correct and zero means wrong, which is illustrative. With outcome supervision the reward is a single number for the whole answer, not one per token. What you now know: every answer in the group has one score.

Step 3: Compare each answer to its group

The third step turns raw rewards into advantages by comparing each answer with its group. The input is the G rewards; the output is one advantage per answer. DeepSeekMath normalises "by subtracting the group average and dividing by the group standard deviation", and in outcome supervision every token of an answer receives that same advantage (Shao et al., 2024, section 4.1.2). The formula is Aˆi=(ri−mean(r))/std(r). Answers above the mean get a positive advantage and answers below it a negative one. What you now know: an answer is judged against its siblings, not against a learned prediction.

One edge case follows from the formula. If all eight answers are correct, every reward equals the mean, so the numerator is zero for every answer, and the standard deviation is zero too, which makes the printed formula 0/0. In practice the advantage is simply zero, so the group gives no gradient.

Yu et al. describe exactly this: if all outputs for a prompt are correct and receive the same reward, "the resulting advantage for this group is zero", and their Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO) method over-samples and filters out prompts with accuracy 1 or 0 so every prompt in the batch carries a gradient (Yu et al., 2025, section 3.2). With pass or fail rewards, a group is all-pass or all-fail whenever the prompt is too easy or too hard.

Step 4: Clip the update

The fourth step limits how far one update can move the policy. The input is the probability ratio, the new policy's probability of a token divided by the old policy's; the output is a bounded objective term. As in PPO, the term is the smaller of ratio times advantage and the ratio clipped to the range 1 minus epsilon to 1 plus epsilon, times the advantage (Schulman et al., 2017). Clipping removes the incentive to push a token's probability far from the old policy in one step.

The objective is also averaged per answer: in DeepSeekMath's equation 3 the sum over an answer's tokens is divided by that answer's length, and then averaged over the G answers (Shao et al., 2024). A short answer therefore gives each of its tokens a larger share than a long answer does. A long wrong answer spreads its negative advantage thinner. What you now know: the update is a bounded step, not a jump, and each answer is weighted by one over its length.

Step 5: Add the KL anchor and update

The fifth step keeps the model close to a frozen copy of the starting model. The reference model is that frozen copy. GRPO adds a KL term directly to the loss, rather than folding it into each token's reward as PPO usually does. DeepSeekMath estimates it with the unbiased form πref/πθ−log(πref/πθ)−1, which is never negative and is zero only when the policy matches the reference (Shao et al., 2024).

Their run used a KL coefficient of 0.04 and a learning rate of 1e-6 (Shao et al., 2024, section 4.2). The optimiser then takes a gradient step on the whole objective, usually a single update per batch of samples in that run. What you now know: the loss pushes up answers that beat their group, pushes down the rest, and penalises drift from the reference model.

GRPO scores each answer against its own groupFive steps of GRPO: sample a group of answers, score each one, subtract the group mean and divide by the standard deviation, clip the probability ratio, and add a KL penalty to a reference model.SampleG answersScoreG rewardsComparegroupadvantageClipratio bandAnchorKL, updateDiagram: Interview Clozo, CC BY 4.0 ·interview.clozo.ai/blog/grpo.html

GRPO replaces the learned value model with the average reward of a group of sampled answers.CC BY 4.0 — source: GRPO explained: how group relative policy optimization trains LLMs
Step-by-step description
  1. Draw G answers for one prompt.
  2. Give every answer a reward.
  3. Subtract the group mean, divide by the deviation.
  4. Bound the probability ratio.
  5. Add KL to the reference model, then update.
  • Sample a group of answers for each prompt.
  • The advantage is reward minus the group mean, divided by the group standard deviation.
  • Clip the probability ratio and add a KL penalty to a reference model.

Knowing it is half the job. Saying it is the other half.

Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.

Start Free Trial →

Worked example

A hand-made group of eight answers to one prompt shows the arithmetic. Three answers are correct and five are wrong, so the rewards are 1, 0, 0, 1, 0, 0, 0, 1 (illustrative: not measurements of a real model). The mean is 0.375 and the population standard deviation is 0.4841. A correct answer therefore has an advantage of (1 minus 0.375) divided by 0.4841, which is 1.291. A wrong answer has minus 0.7746. The eight advantages sum to zero by construction.1

A rare correct answer earns the largest advantageBar chart of the advantage a correct answer receives in an illustrative group of eight: 1.291 when three are correct, 2.6458 when one is correct, and 0 when all eight are correct.Advantage of a correct answer01233 of 8 correct1 of 8 correct8 of 8 correctIllustrative group of eight1.2912.64580Diagram: Interview Clozo, CC BY 4.0 ·interview.clozo.ai/blog/grpo.html
A correct answer earns more advantage the fewer answers in its group are correct, and none when all are.CC BY 4.0 — source: GRPO explained: how group relative policy optimization trains LLMs
Text description of the diagram
  • A correct answer in a group with three correct of eight has an advantage of 1.291.
  • If only one of the eight is correct, that answer gets an advantage of 2.6458.
  • If all eight are correct, every advantage is 0.

The group's difficulty changes the scale. If only one of the eight is correct, that answer gets an advantage of 2.6458 and each wrong one gets minus 0.378. If all eight are correct, every reward equals the mean, the numerator is zero, and every advantage is zero, so that prompt teaches nothing. Clipping works per token with epsilon of 0.2: for the correct answer at a ratio of 1.3 the term is capped at 1.5492 instead of 1.6783, and for a wrong answer at a ratio of 0.7 it is capped at minus 0.6197 instead of minus 0.5422, so no gradient flows in either case.1

A rare correct answer earns the largest advantage, a group where everything is correct earns none, and clipping stops the gradient once the ratio leaves the 0.8 to 1.2 band in the direction the advantage favours.
FactValue
Group rewards (illustrative)1, 0, 0, 1, 0, 0, 0, 1
Mean and population standard deviation0.375 and 0.4841
Advantage of a correct answer1.291
Advantage of a wrong answerminus 0.7746
Advantage of the one correct answer in eight2.6458
Advantages when all eight are correctall 0
Clipped term, correct answer at ratio 1.31.5492 (unclipped 1.6783)
KL estimator at reference-to-policy ratio 0.80.02314

The KL estimator value in the last row is computed from the formula in step five.1 Sample standard deviation, dividing by seven instead of eight, would give 0.5175 and slightly smaller advantages; the paper does not say which is used.

Trade-offs and edge cases

GRPO saves memory and costs samples. PPO trains a value model "of comparable size as the policy model", which the DeepSeekMath authors call "a substantial memory and computational burden" (Shao et al., 2024). GRPO removes it, but it must sample G answers for every prompt, here 64, so generation time grows. The value model also gave a learned estimate of progress at every token, which GRPO does not have: with outcome supervision, every token of an answer shares one advantage.

GRPO has known biases. Liu et al. report an optimisation bias that "artificially increases response length (especially for incorrect outputs)" and propose a corrected variant they call Dr. GRPO (Liu et al., 2025). Yu et al. present a further modified algorithm, DAPO, that reaches 50 points on the American Invitational Mathematics Examination (AIME) 2024 maths benchmark with a Qwen2.5-32B base model (Yu et al., 2025). Reported practice differs on which fixes matter, so treat the exact recipe as moving.

The length bias comes from the per-answer average above: Liu et al. say dividing by the response length means "longer responses are penalized less" among incorrect answers, and correct answers get larger updates when short (Liu et al., 2025, section 3.1).

They also describe a question-level difficulty bias from the division by the group standard deviation: questions with low standard deviation, those that are too easy or too hard with rewards almost all 1 or 0, get higher weight in the update. Dr. GRPO removes both the length average and the standard deviation division (Liu et al., 2025, section 3.2). Process supervision, scoring each reasoning step and summing later step rewards into each token's advantage, is the other option in the paper.

GRPO trades a trained value model for more samplesComparison of PPO and GRPO by where the baseline comes from, which models are trained, how many answers are sampled per prompt and the main weakness of each.DimensionPPOGRPOBaselineLearned valuemodelGroup meanrewardModels trainedPolicy and valuePolicy onlySamples per promptFewA group of GKL penaltyIn the per-tokenrewardIn the lossMain weaknessCritic memoryNo signal ifrewards tieDiagram: Interview Clozo, CC BY 4.0 ·interview.clozo.ai/blog/grpo.html
GRPO trades a trained value model for more samples per prompt.CC BY 4.0 — source: GRPO explained: how group relative policy optimization trains LLMs
Text description of the diagram
  • PPO trains a policy and a value model; GRPO trains only the policy.
  • GRPO samples a group of answers per prompt.

The ratio is built from the same token log-probabilities as cross-entropy loss; see also gradient descent for the update step.

How to answer it in an interview

Interviewer asks

What is GRPO, and how does it differ from PPO?

A strong answer

GRPO is a PPO variant that removes the value model. For each prompt I sample a group of answers, score them, and use each reward minus the group mean, divided by the group standard deviation, as the advantage for every token of that answer. The clipped ratio objective stays; the KL penalty to a reference model moves out of the reward and into the loss. That saves the memory of a critic as large as the policy, at the cost of sampling many answers per prompt. It gives no signal when every answer in a group scores the same, and it has a known length bias.

The spoken structure is: say what is removed, say what replaces it, then name the cost and one weakness.

Common mistakes

  1. Saying GRPO has no baseline. (wrong) "GRPO drops the baseline." (right) "GRPO uses the group mean as the baseline instead of a learned value function."
  2. Forgetting the standard deviation. (wrong) "The advantage is reward minus the mean." (right) "It is reward minus the group mean, divided by the group standard deviation."
  3. Thinking clipping bounds the reward. (wrong) "Clipping limits the reward." (right) "Clipping limits the probability ratio between the new and old policy."
  4. Claiming GRPO needs no reference model. (wrong) "No extra models at all." (right) "It drops the value model; the reference model for the KL term and the reward source remain."
  5. Calling it free. (wrong) "GRPO is strictly cheaper." (right) "It trades a critic for many samples per prompt, so generation cost rises."

Follow-up questions

What is the difference between GRPO and PPO?

PPO estimates advantages with a learned value function and a per-token KL penalty inside the reward. GRPO estimates them from the rewards of a sampled group and puts the KL term in the loss. Both use the same clipped ratio objective (Shao et al., 2024).

Why did GRPO drop the value model?

The value model is a second network about the size of the policy, and the reward model usually scores only the last token, which makes a per-token value hard to learn. The group mean is a cheap baseline that matches how reward models are trained on comparisons between answers to one prompt (Shao et al., 2024).

When does the ratio actually get clipped?

The ratio is clipped when the new policy has already moved more than epsilon from the old one in the direction the advantage rewards: above 1 plus epsilon for a positive advantage, below 1 minus epsilon for a negative one. With epsilon of 0.2, a token with a positive advantage stops contributing a gradient once its ratio is above 1.2, and a token with a negative advantage stops once its ratio is below 0.8. A ratio on the other side keeps its gradient, because the minimum picks the unclipped term: a positive-advantage token at ratio 0.7 still gets pushed up, as the toy computation shows.1

Can GRPO use outcome supervision, process supervision or both?

Both are defined in the DeepSeekMath paper. Outcome supervision gives every token the same normalised final reward; process supervision scores each step and sums the normalised rewards of the steps that follow a token (Shao et al., 2024). The wider evaluation picture is in LLM evaluation.

What are the known weaknesses of GRPO?

The length bias reported by Liu et al. and the zero signal from groups where every answer scores the same are the two usual answers. The transformer basics page covers the model being trained, and deep learning the optimisation background.

Key takeaways
  • GRPO replaces PPO's value model with the mean reward of a group of sampled answers.
  • The advantage is reward minus group mean, divided by group standard deviation, shared by every token of the answer.
  • The clipped ratio objective stays from PPO; the KL penalty to a reference model moves out of the per-token reward and into the loss.
  • It saves critic memory but needs many samples per prompt, and a group with equal rewards gives no signal.
  • Known biases such as length growth are active research, so quote the source for any fix.

Sources

  1. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models — arXiv (Shao et al.), v3 revised 2024-04-27, 5 Feb 2024
  2. Proximal Policy Optimization Algorithms — arXiv (Schulman et al.), v2 revised 2017-08-28, 20 Jul 2017
  3. Understanding R1-Zero-Like Training: A Critical Perspective — arXiv (Liu et al.), v2 revised 2025-10-06, 26 Mar 2025
  4. DAPO: An Open-Source LLM Reinforcement Learning System at Scale — arXiv (Yu et al.), v2 revised 2025-05-20, 18 Mar 2025
  5. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning — arXiv (DeepSeek-AI), v2 revised 2026-01-04, 22 Jan 2025