PPO reinforcement learning: what it is and how it works

In this article

1. Before you start
2. Why interviewers ask this
3. PPO from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step of the how large language models are trained path. The transformer is the network whose outputs PPO adjusts when it is used on language models.

Why interviewers ask this

A candidate who can only say "PPO is what ChatGPT uses" stops being useful after one follow-up. PPO is a common topic in questions about aligning language models, because it joins three things an interviewer can probe separately: the policy-gradient idea, the clipping trick and the way a reward model plugs in.

A weak answer names the algorithm and says it is "stable." A strong answer writes the clipped objective, says what the ratio and the advantage are, and states that clipping removes an incentive rather than enforcing a limit. A Staff-level answer also names the memory cost of the extra models and explains why methods such as group relative policy optimization (GRPO) drop the critic; the GRPO page covers them. The vocabulary below carries over to questions about RLHF.

PPO from first principles

Proximal policy optimization

Proximal policy optimization is a family of policy-gradient methods that alternate between sampling data from an environment and optimizing a surrogate objective with stochastic gradient ascent, reusing each batch for several epochs of minibatch updates while keeping the new policy close to the old one.

Advantage

Advantage is a number that says how much better the return after an action turned out to be than the critic expected in that situation. A positive advantage says the action beat expectations, a negative advantage says it fell short, and the size says by how much.

In plain words

Imagine learning to throw darts by changing your stance. A big change after one lucky throw can wreck what already works. PPO is the habit of making only small changes and re-checking. The analogy breaks in one place: PPO does not forbid a big change, it only stops rewarding it.

In one precise sentence, PPO collects a batch with the old policy, scores each action with an advantage, and maximizes an objective in which the gain from changing an action's probability is cut off outside a small interval around its old value. The method is from Schulman et al. (2017).

Step 1: The old policy samples a batch

PPO first freezes a copy of the current policy, called the old policy, and lets it act. In a game or robot task it plays episodes. In a language model the state is the prompt plus the tokens written so far, the action is the next token, and the policy is the model itself.

Input: the old policy and an environment. Output: a batch of states, actions, their old probabilities and rewards. The Spinning Up page lists 4,000 steps per epoch as its default batch. Rewards are then summed forward into rewards-to-go, the total reward collected from each step onward. You now have the raw material. Nothing has been learned yet, and the next step turns rewards into a signal about each action.

Step 2: A critic turns rewards into advantages

A critic is a second network that estimates the expected reward-to-go from a given state. Subtracting that estimate from the actual reward-to-go gives the advantage: positive when the action did better than expected, negative when worse.

Input: rewards-to-go and the critic's estimates. Output: one advantage per action. Spinning Up fits the critic by mean squared error regression onto the rewards-to-go and smooths the advantages with generalized advantage estimation (GAE), using a discount of 0.99 and a smoothing value of 0.97 by default (Spinning Up defaults). The critic is trained alongside the policy, so it is a second model to keep in memory. An action with advantage zero gives the policy nothing to learn from.

Step 3: Clipping removes the incentive, not the freedom

The policy loss compares new and old: $r_t(\theta )={\pi }_{\theta }(a_t|s_t)/{\pi }_{{\theta }_{\text{old}}}(a_t|s_t)$. The PPO paper maximizes the clipped objective:

Here ${\hat{A}}_t$ is the advantage. Spinning Up sets ε to 0.2, so the interval is 0.8 to 1.2 (Spinning Up). The min takes the lower of the clipped and unclipped terms. The gradient goes flat only when the ratio has already moved past the interval in the direction the advantage favors: a good action made more likely than 1.2, or a bad action made less likely than 0.8.

In the other two cases the gradient stays on, even far outside the interval: a bad action made more likely, and a good action made less likely. These are the cases the lead hinted at, and they are why the clip is a removed incentive, not a wall. The Spinning Up page says PPO-Clip "doesn't have a constraint at all."

Step 4: The batch is reused for several epochs

Plain policy gradient takes one update per sample and throws the data away. PPO reuses the same batch for several epochs of minibatch updates, which is cheaper than gathering new data, and clipping is what makes that reuse safe. Spinning Up takes up to 80 policy iterations per epoch by default, each a full-batch gradient step rather than a minibatch one.

Clipping alone can still let the policy drift. Spinning Up adds an early stop: when the mean KL divergence of the new policy from the old passes a threshold, a target KL of 0.01 by default (the code stops at 1.5 times that), it stops the gradient steps (Spinning Up). That check is an add-on, not part of the original algorithm; one reimplementation study includes it but turns it off by default. A common diagnostic is clipfrac, the fraction of training data that triggered the clipped objective.

Step 5: In RLHF a reward model and a KL penalty are added

Reinforcement learning from human feedback (RLHF) applies PPO to language models. InstructGPT used three steps: supervised fine-tuning (SFT), training a reward model on human comparisons, then PPO against that reward model. PPO never reads human preferences itself; it maximizes the one number the reward model returns.

The paper treats the setting as a bandit: a random prompt appears, the model writes a response, the reward model scores it, and the episode ends. A KL penalty from the SFT model is added during PPO, per token, to limit over-optimization of the reward model, and a variant mixes in pretraining gradients to fix regressions on public benchmarks. The result: human labelers preferred the 1.3-billion-parameter InstructGPT to the 175-billion-parameter generative pre-trained transformer (GPT)-3 (Generative Pre-trained Transformer 3), despite 100 times fewer parameters (Ouyang et al., 2022).

1 · Sample2 · Advantage3 · Clip4 · Reuse5 · RLHFReplay

Step-by-step description

1. The old policy samples a batch.
2. A critic turns rewards into advantages.
3. Clipping removes the incentive to move far.
4. The batch is reused for several epochs.
5. A reward model and a KL penalty are added.

• The old policy samples a batch and the critic estimates each advantage.
• The clipped ratio removes the incentive to move far from the old policy.
• The batch is reused for several epochs, and RLHF adds a reward model and a KL penalty.

Text description of the diagram

• InstructGPT used three steps: supervised fine-tuning, reward model training, then PPO against the reward model.
• A KL penalty from the SFT model is added during PPO.

Worked example

Take one action with a reward-to-go of 3.0 and a critic estimate of 1.0 (both illustrative). The advantage is 3.0 minus 1.0, which is +2.0; a second action with reward-to-go 0.5 and estimate 2.5 has advantage -2.0.¹ Now the ratio. Say the old policy gave an action a log-probability of -2.30 and the new one gives -1.90 (illustrative). The ratio is exp(-1.90 - (-2.30)), which is 1.4918, so the action became about 49% more likely.¹

With ε = 0.2 the interval is [0.8, 1.2]. The four cases below show the objective for each sign and ratio (illustrative values).¹

Row one says: this action already got more likely by enough, stop pushing. Row two says: this good action got less likely, so the unclipped term is used and the gradient lifts it back up. Row four says: this bad action got more likely, so keep pushing it back down.

Plotted over ratios from 0.4 to 1.6, the same objective shows the shape.¹ For advantage +2 it rises to 2.4 at a ratio of 1.2 and stays flat from there upward. For advantage -2 it stays flat at -1.6 below a ratio of 0.8 and keeps falling above 1.2, reaching -3.2 at 1.6.

Text description of the diagram

• For advantage +2 the objective rises to 2.4 at a ratio of 1.2 and stays flat from there upward.
• For advantage -2 it stays flat at -1.6 below a ratio of 0.8 and keeps falling above 1.2.

Trade-offs and edge cases

PPO trades simplicity and reuse of data for a soft safeguard. The paper describes two objectives: the clipped surrogate objective and an adaptive KL penalty between the new and old policy (Spinning Up calls them PPO-Clip and PPO-Penalty). In the authors' experiments the KL penalty performed worse than the clipped objective, which is why "PPO" usually means the clip version.

The larger cost in language-model work is memory. PPO trains an actor and a critic and also needs a reward model and a frozen reference model for the KL term. For a 7-billion-parameter model stored at 2 bytes per parameter (illustrative), each copy is 14 GB of weights; four same-size copies are 56 GB, and without a critic three copies are 42 GB, before optimizer state and activations.¹ The DeepSeekMath paper introduces group relative policy optimization (GRPO), which forgoes the critic and estimates the baseline from group scores to reduce training resources; the GRPO page covers it.

Text description of the diagram

• Illustrative: four 7-billion-parameter copies at 2 bytes per parameter take 56 GB of weights.
• Three copies without a critic take 42 GB.

Edge cases: two different KL terms exist and are easy to confuse (see the mistakes below); a clip that is too tight slows learning; and practice differs on early stopping, as above. Whether PPO is still the default for aligning language models is a moving target; DeepSeekMath calls it widely used in the reinforcement learning (RL) fine-tuning stage, and direct preference optimization (DPO) versus RLHF covers the alternative route.

How to answer it in an interview

The spoken structure: say what PPO optimizes, say what clipping does and does not do, then say how RLHF plugs in.

Common mistakes

• "Clipping keeps the policy inside a trust region." (wrong) PPO-Clip has no constraint; it only removes an incentive, and the policy can still drift. (right) "Clipping flattens the gradient past the interval, and I add a KL check if drift matters."
• "PPO learns from human preferences." (wrong) PPO maximizes a number. (right) "A reward model trained on human comparisons supplies that number."
• "The KL penalty in RLHF is the PPO-Penalty KL." (wrong) These are two terms. (right) "PPO-Penalty compares new and old policy to set step size; the RLHF penalty compares the policy to the SFT model to limit reward over-optimization."
• "RLHF with PPO is a long multi-step episode." (wrong) InstructGPT treats it as a bandit. (right) "One prompt, one response, one reward, then the episode ends."

Follow-up questions

How does PPO work in reinforcement learning?

PPO samples a batch with the old policy, computes advantages with a critic, and maximizes a clipped objective for several epochs on that batch. Step 3 above shows why the clip makes the reuse safe.

Why does PPO clip the probability ratio?

Without a limit, repeated updates on one batch can move the policy far from the policy that collected the data, and the data stops being representative. Clipping removes the incentive for large moves. It does not forbid them.

What is the difference between PPO and DPO in reinforcement learning?

Direct preference optimization (DPO) drops the separate reward model and the sampling loop. It trains the policy with a classification-style loss on a fixed set of preference pairs, a preferred and a rejected answer each, measured against a frozen reference model (Rafailov et al., 2023). PPO instead samples new answers from the current policy and scores them with a reward model, which costs more memory. See why DPO has been replacing PPO and DPO versus RLHF.

What is the difference between PPO and GRPO?

Group relative policy optimization (GRPO) is a PPO variant that drops the learned critic and estimates the baseline from a group of sampled answers, which cuts the number of trainable models. See GRPO explained and, for how such training is used, how reasoning models work.

What does the KL penalty do in RLHF with PPO?

It is a per-token penalty for drifting from the SFT model, added to limit over-optimization of the reward model. Fine-tuning basics are in fine-tuning a large language model (LLM).

Sources

1. Proximal Policy Optimization Algorithms (arXiv 1707.06347) — arXiv (Schulman et al., OpenAI), 20 Jul 2017
2. Proximal Policy Optimization - Spinning Up documentation — OpenAI (Achiam); undated page, read 2026-10-26, 26 Oct 2026
3. Training language models to follow instructions with human feedback (arXiv 2203.02155) — arXiv (Ouyang et al., OpenAI), 4 Mar 2022
4. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (arXiv 2402.03300) — arXiv (Shao et al., DeepSeek), 5 Feb 2024
5. The 37 Implementation Details of Proximal Policy Optimization — ICLR Blog Track (Huang et al.), 25 Mar 2022
6. Direct Preference Optimization: Your Language Model is Secretly a Reward Model (arXiv 2305.18290) — arXiv (Rafailov et al.), 29 May 2023