Fine-Tuning & LLM Training

Is GRPO a policy gradient algorithm?

By Interview Clozo · Published 27 October 2026 · Last verified 26 October 2026 · 11 min read

Is Group Relative Policy Optimization (GRPO) a policy gradient algorithm? Yes: GRPO is a critic-free variant of Proximal Policy Optimization (PPO) whose gradient has the policy-gradient form, with the average reward of a group of sampled answers as its baseline. The follow-up trap is how a group average can stand in for a critic, and Step 2 works the arithmetic.¹

In this article

Before you start
Why interviewers ask this
Policy gradient from first principles
The Staff-level answer
The rubric
Three variants of this question
Common mistakes
Follow-up questions
Sources

Before you start

This page belongs to the large language model training path and assumes the basics of the transformer architecture.

Why interviewers ask this

A candidate who answers "no, GRPO has no critic, so it is not policy gradient" has confused a feature of PPO with the definition of the whole family. The question separates people who memorised "GRPO drops the critic" from people who know what makes a method policy gradient and can say what the group average does instead.

A weak answer repeats the name, "group relative", without saying relative to what. A strong answer says yes, gives the advantage formula and names PPO as the parent. A Staff-level answer also states what the shortcut costs: a baseline estimated from few samples, and normalising terms that bias the update. The question is reported by candidates in interviews on large language model training; no company is named here because no public source names one.

Policy gradient from first principles

Policy gradient algorithm: A policy gradient algorithm is a reinforcement learning (RL) method that improves a policy by moving its parameters along an estimate of the gradient of expected reward, written as a reward-like weight times the gradient of the log-probability of the sampled actions, without first learning a value for every state.

In plain words

Picture a class where each student writes an answer to the same question. A teacher gives every answer one score. A student learns by comparing their score with the class average. Above average, do more of that; below, do less. GRPO works like that, with the model as the student. The analogy breaks in one place: the score covers the whole answer, yet the update is applied to every token of it.

In one precise sentence, GRPO nudges the policy so that answers scoring above the average of their group become more probable and answers below it become less probable.

Step 1: The gradient has the policy-gradient form

GRPO's gradient is a coefficient times the gradient of the log-probability of each generated token, which is the policy-gradient (score-function) form. Shao et al. (2024), Appendix A write the PPO gradient with the coefficient A and the GRPO gradient with the coefficient Â plus a KL term, both multiplying the log-probability gradient. The same paper writes supervised fine-tuning, rejection sampling, Direct Preference Optimization (DPO), PPO and GRPO in one template and concludes they are "either direct or simplified RL techniques."

That template shows GRPO has policy-gradient form. It does not make the methods identical. The textbook chapter on policy gradient methods lists a Monte Carlo method with no critic (13.3), a baseline version (13.4) and actor-critic methods (13.5). A critic is optional in the family. GRPO is the Monte Carlo policy gradient method called REINFORCE (REward Increment = Nonnegative Factor times Offset Reinforcement times Characteristic Eligibility) with a baseline (13.4) plus the PPO clip and a KL loss term. You now know GRPO qualifies; Step 2 shows what takes the critic's place.

GRPO keeps the policy-gradient form and the PPO clip, and swaps the learned critic for a group average.CC BY 4.0 — source: Is GRPO a policy gradient algorithm?

Step-by-step description

Coefficient times log-probability gradient.
The group mean replaces the critic.
Both come from PPO.
Length and std terms add bias.

The GRPO gradient is a coefficient times the gradient of the log-probability.
A group average replaces the learned value model as the baseline.
The clipped ratio and the KL term come from PPO.
The normalising terms add biases that Dr. GRPO removes.

Step 2: A group average replaces the critic as the baseline

A group average replaces the learned value model as the baseline in GRPO. A critic is a second network, often as large as the policy, that predicts how good each state is. An advantage is how much better an output is than the baseline expects. GRPO instead samples a group of answers to one prompt and normalises each reward by the group mean and standard deviation (Shao et al., 2024):

{\hat{A}}_{i} = \frac{r_{i} - mean (r)}{std (r)}

Every token of output i receives that same value. Take a group of four with rewards 1, 0, 0, 1 (illustrative): the mean is 0.5, the standard deviation is 0.5, and the advantages are +1 and −1.¹ With one correct answer in four, the correct one gets +1.7321 and each wrong one −0.5774.¹ With three correct, the correct ones get only +0.5774. A rare success earns a larger push. The group mean includes r_i itself, so the baseline is not strictly independent of the action; this only shrinks the expected gradient by a factor of (G−1)/G, and leave-one-out variants avoid it.

A correct answer is worth most when it is rare in its group, which is the group baseline at work.CC BY 4.0 — source: Is GRPO a policy gradient algorithm?

Text description of the diagram

With one correct answer in four, the correct one gets +1.7321.
With three correct, the correct ones get only +0.5774.

Step 3: The clipped ratio and the KL term come from PPO

GRPO keeps PPO's clipped probability ratio and moves the KL penalty from the reward into the loss. Schulman et al. (2017) describe PPO as "a new family of policy gradient methods" that optimise a surrogate objective, with a clipped ratio that "forms a pessimistic estimate (i.e., lower bound)" (stated in the introduction of the PPO paper) and allows several epochs of minibatch updates. PPO for language models adds a per-token KL penalty to the reward; GRPO instead "regularizes by directly adding the KL divergence between the trained policy and the reference policy to the loss" (Shao et al., 2024).

In the gradient, the coefficient becomes:

{\hat{A}}_{i, t} + β (\frac{π_{ref}}{π_{θ}} - 1)

With the paper's β of 0.04 and an advantage of +1, values of πref/πθ of 0.5, 1 and 2 (illustrative) give coefficients 0.98, 1.0 and 1.04.¹ The KL term is a small correction beside the advantage. The setting for DeepSeekMath was 64 sampled outputs per question (Shao et al., 2024).

Step 4: The normalising terms add bias

GRPO's two normalising terms, division by output length and by the standard deviation, add biases that the Dr. GRPO authors report and remove. Liu et al. (2025) "identify the bias in optimization of GRPO," which "may lead to progressively longer incorrect responses," and propose "GRPO Done Right (Dr. GRPO)."

With the 1/length factor, two outputs with an advantage of +1 and lengths 100 and 400 tokens (illustrative) get per-token weights 0.01 and 0.0025, so the short one is pushed harder per token.¹ The reported effect on negative advantages is the mirror image: long wrong answers are penalised less. Dividing by the standard deviation reweights questions: at illustrative success rates of 0.5, 0.1 and 0.01, the weight is 2, 3.3333 and 10.0504, so very easy or very hard questions count more.¹ These remain policy-gradient updates, only biased ones.

Knowing it is half the job. Saying it is the other half.

Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.