In this article
Before you start
This page belongs to the fine-tuning and alignment learning path; the full method is in the GRPO explainer, and the model itself is described in transformer architecture.
Why interviewers ask this
The question separates candidates who know how GRPO builds its learning signal from candidates who remember only its name. It is reported by candidates in interviews at AI labs and large tech companies that train reasoning models, usually as a follow-up after a question on reinforcement learning from human feedback (RLHF).
A weak answer says "GRPO needs a reward model, so a pass/fail grade is too coarse," or "yes, it works" with no reason. A strong answer explains that the signal is the contrast between answers inside one group, so two reward values are enough. A Staff-level answer adds the failure case: a group where every answer gets the same grade has no contrast. It then says how to keep groups mixed, and notes that dividing by the standard deviation reweights prompts by difficulty.
Binary rewards in GRPO from first principles
- Binary reward
- A binary reward is a score that takes only two values, such as one for a passing answer and zero for a failing one, produced by a rule such as an exact answer match or a set of test cases, so the grader never says how close a failing answer came.
- Silent group
- A silent group is a set of sampled answers to one prompt that all received the same reward, so every answer sits exactly at the group average, every advantage is zero, and the prompt adds nothing to the policy gradient, the direction in which the weights are updated.
- Dynamic sampling
- Dynamic sampling is a training-data filter that draws more prompts than a batch needs and keeps only those whose sampled group contains both passing and failing answers, so no batch slot is spent on a silent group.
In plain words
Picture a class of eight students who each answer the same question. The teacher marks only pass or fail. She does not need a score out of one hundred: she compares each student with the class average. Passers sit above it and failers below it, and she praises the methods of the passers.
The analogy breaks in one place. If all eight pass, or all eight fail, nobody stands out, and she learns nothing about which method was better.
In one precise sentence, GRPO samples several answers to one prompt, grades each, and pushes up the probability of answers that beat the group average while pushing down the rest. It drops the learned critic, a second network that Proximal Policy Optimization (PPO) uses to estimate a baseline, and takes the baseline from the group scores instead (Shao et al., 2024).
Step-by-step description
- Grade the group, subtract the mean.
- Same grades mean zero advantage.
- Dividing by the spread reweights prompts.
- Keep groups mixed.
- A group where every answer passes or every answer fails has zero advantage.
- Dynamic sampling, larger groups and difficulty filtering keep groups mixed.
Step 1: A pass/fail grade becomes an advantage
A pass/fail grade becomes a training signal by comparison with the group average. An advantage is a number attached to one sampled answer that says how much better or worse its reward was than the rest of its group. Liu et al. (2025) give it as the reward minus the group mean, divided by the group standard deviation, and say GRPO sets the advantage of all tokens of that answer to this one value.
The grader can be a plain rule. DeepSeek-AI (2025) used rule-based accuracy rewards that check whether the response is correct, and for code a compiler runs predefined test cases. The Nature version describes an exact match with the ground-truth solution (DeepSeek-AI, 2025).
Take a group of eight answers with two passes (illustrative). The mean reward is 0.25 and the population standard deviation is 0.4330. Each pass gets an advantage of +1.7321, each fail gets -0.5774, and the advantages sum to zero.1 Without the standard-deviation division the values are +0.75 and -0.25.1 You now know a zero-or-one grade is a valid input, because the contrast lives inside the group.
Step 2: A group where every answer has the same outcome gives zero signal
A group where every answer passes or every answer fails has zero advantage for every response. This page calls it a silent group, its own label; Yu et al. (2025) note that when all outputs for a prompt are correct and get the same reward, the advantage is zero. They add that a zero advantage gives zero policy gradient, which shrinks the batch gradient and makes it noisier.
With a per-sample pass rate p and group size G, the chance that a group is silent follows from counting. All G samples pass with probability p to the power G, and all fail with probability one minus p to the power G.
For G of 8, a prompt at pass rate 0.5 is silent with probability 0.0078, but a prompt at pass rate 0.05 or 0.95 is silent with probability 0.6634.1 In an illustrative batch of 100 prompts that the model nearly always passes, about 66.3 groups carry no signal.1 That is the payoff of the follow-up trap: a binary reward works, but only on prompts of useful difficulty.
Step 3: Dividing by the standard deviation reweights prompts by difficulty
With a binary reward, the group standard deviation (population form) is the square root of the observed pass rate times one minus that rate. It peaks at a pass rate of one half and falls toward the extremes, so dividing by it boosts prompts that are very easy or very hard. Liu et al. (2025) say the standard-deviation and length terms can bias optimization by giving different weights to different questions and responses.
At pass rates of 0.5, 0.1 and 0.01 the standard deviation is 0.5, 0.3 and 0.0995, so the division multiplies the update by 2, 3.3333 and 10.05.1 Their fix, the Dr. GRPO variant, removes the standard-deviation and the length normalization. Training libraries typically expose the same choice as a setting.
| Passes in the group of 8 | Pass advantage, scaled | Fail advantage, scaled | Pass, unscaled | Fail, unscaled |
|---|---|---|---|---|
| 1 | 2.6458 | -0.378 | 0.875 | -0.125 |
| 2 | 1.7321 | -0.5774 | 0.75 | -0.25 |
| 4 | 1.0 | -1.0 | 0.5 | -0.5 |
| 7 | 0.378 | -2.6458 | 0.125 | -0.875 |
Step 4: Three fixes keep groups mixed
Dynamic sampling, larger groups and difficulty filtering keep groups mixed. Yu et al. address the silent case with dynamic sampling: oversample prompts and keep those whose group has both correct and incorrect answers. A larger group lowers the chance of a tie. At a pass rate of 0.2, a group of 4 is silent with probability 0.4112, a group of 8 with 0.1678 and a group of 16 with probability 0.0281.1 A group of 64 is silent with a probability near zero, about six in ten million.1 Each extra sample costs generation time, so group size is a budget choice.
Text description of the diagram
- At a pass rate of 0.2, a group of 4 is silent with probability 0.4112 and a group of 16 with probability 0.0281.
- A group of 64 is silent with a probability near zero.
Difficulty filtering picks prompts whose pass rate is neither near zero nor near one for the current model. The reward must also stay reliable. DeepSeek-AI (2025) warn that "If the reward signal is assigned by a model instead of predefined rules, it becomes more susceptible to exploitation as training progresses."
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →The Staff-level answer
Can GRPO work with binary rewards, pass or fail?
Yes. GRPO samples a group of answers per prompt, grades each one, and gives each answer an advantage equal to its reward minus the group mean, usually divided by the group's standard deviation. A pass/fail grade works because the signal is the contrast inside the group. The failure mode is a prompt where every sample passes or every sample fails: all advantages are zero, so it adds no gradient. I handle it with a larger group, by over-sampling and filtering so groups are mixed, and by choosing prompts at a useful difficulty. I would also check the standard-deviation scaling, since it upweights very easy and very hard prompts.
What makes it Staff-level: it names the zero-signal case, gives three remedies and their cost, and flags the scaling bias instead of stopping at yes.
The rubric
| Dimension | Weak answer | Strong answer | Staff answer |
|---|---|---|---|
| Correctness | Says it needs a reward model or a graded score | Says a rule-based pass/fail grade is enough | Adds that the signal is the within-group contrast |
| Depth | Names GRPO only | Gives the advantage as reward minus group mean over spread | Derives when the advantage is zero and how common that is |
| Trade-offs | None | Mentions that identical grades give no signal | Weighs group size, filtering and difficulty bias against cost |
| Communication | Rambles through PPO history | States the answer first, then the reason | Answer, failure mode, fix, in one pass |
| Follow-up readiness | Stuck on the all-correct case | Handles the all-correct case | Also covers scaling, length bias and reward reliability |
Three variants of this question
What if the model already passes nearly every prompt?
Most groups turn silent, because the silent probability for a group of eight at a pass rate of 0.95 is 0.6634.1 Move to harder prompts or filter out prompts the model always solves.
What if only one answer in a thousand passes?
At a pass rate of 0.001 with a group of eight, a group is silent with probability 0.992.1 Training then needs easier prompts, a model that already passes sometimes, or far more samples per prompt.
Should you add partial credit to a pass/fail reward?
Partial credit gives mixed groups more contrast, but it is a design choice with a cost: each shaped term is something the model can exploit. The reward in DeepSeek-AI (2025) adds only a format reward to the accuracy reward, so a shaped reward is justified by testing, not assumed.
Common mistakes
- "GRPO needs a learned reward model, so pass/fail will not work." (wrong) Rule-based accuracy checks, such as answer matching and test cases, trained DeepSeek-R1-Zero. (right) "A rule-based pass/fail grade is a valid reward."
- "If all eight answers pass, the prompt reinforces good behavior." (wrong) Every advantage is zero, so it gives no policy gradient. (right) "An all-pass group is silent; I filter or re-sample it."
- "Standard-deviation normalization is neutral." (wrong) It reweights prompts by difficulty. (right) "I check the scaling, and Dr. GRPO removes it."
- "The reward in the original reasoning runs is just pass or fail." (wrong) The accuracy reward and a format reward are summed with equal weight. (right) "Each component is rule-checked; the total is their sum."
Follow-up questions
Does GRPO with pass/fail rewards need a reward model?
No. GRPO takes its baseline from the group, so it needs neither a critic nor a learned reward model. The grade can come from a rule such as an answer match or test cases.
Does a format reward make the reward non-binary?
Not by itself. DeepSeek-AI (2025) sum an accuracy reward and a format reward with the same weight. Each part is a rule check, so the total takes a few discrete values rather than a graded score.
How many samples per prompt should you use with a binary reward?
Enough that most groups contain both outcomes, which depends on the prompt's pass rate. A bigger group lowers the silent probability, as the chart above shows, but multiplies generation cost.
Is a rule-based pass/fail reward safer than a learned one?
Generally it is harder to exploit. The Nature paper reports that model-assigned rewards become more susceptible to exploitation as training progresses, which is why rule-based checks are used where answers can be verified.
How does GRPO compare to DPO when you only have pass/fail labels?
GRPO consumes the grade directly during sampling, while direct preference optimization (DPO) learns from pairs of preferred and rejected answers. The GRPO and DPO comparison and the DPO versus RLHF page cover the choice.
- GRPO needs only a contrast inside the group, so a pass/fail grade is enough.
- A group where every answer gets the same grade has zero advantage and no gradient.
- Larger groups, dynamic sampling and difficulty filtering keep groups mixed.
- Dividing by the standard deviation reweights prompts by difficulty; Dr. GRPO removes it.
- The grader should be a reliable rule, because model-assigned rewards are easier to exploit.
Sources
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (arXiv 2402.03300) — arXiv (Shao et al.), 5 Feb 2024
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (arXiv 2501.12948) — arXiv (DeepSeek-AI), 22 Jan 2025
- DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning — Nature (DeepSeek-AI), 17 Sep 2025
- An Open-Source LLM Reinforcement Learning System at Scale (arXiv 2503.14476) — arXiv (Yu et al.), 18 Mar 2025
- Understanding R1-Zero-Like Training: A Critical Perspective (arXiv 2503.20783) — arXiv (Liu et al.), 26 Mar 2025
