What is RLHF, and how is it used?

In this article

1. Before you start
2. Why interviewers ask this
3. RLHF from first principles
4. The Staff-level answer
5. The rubric
6. Three variants of this question
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

RLHF is one stage of how large language models are trained. This page stays on the interview angle; the RLHF overview covers the basics.

Why interviewers ask this

A candidate who cannot explain RLHF cannot explain why a chat model behaves differently from the raw model it was built on. The question is reported by candidates in interviews at AI labs and large tech companies, often as a walk-through followed by "where does it fail?".

A weak answer lists three words: "reward model, proximal policy optimization (PPO), humans." A strong answer says what data each stage uses, writes the objective as reward minus a penalty, and names why the penalty exists. A Staff-level answer adds the failure mode (the reward model is only a proxy), the cost of the loop, and where preference-optimization shortcuts fit. Ouyang et al. (2022) open with the problem RLHF addresses: making a language model bigger does not by itself make it better at following a user's intent.

RLHF from first principles

SFT

SFT is a training stage that continues next-token training of a language model on human-written examples of the desired behaviour, which gives a starting policy that roughly follows instructions before any preference learning begins.

In plain words

Think of training a chef. First the chef copies your dishes. Then diners compare pairs of dishes, and a taster-in-a-box learns their tastes. The chef then practises against the box, with a rule: do not drift far from your trained style. That rule is what keeps the chef honest. The analogy breaks in one place: the box is imperfect, and a clever chef can find odd dishes that fool the box but would not fool real diners.

In one precise sentence, RLHF fits a scalar reward function to human comparisons and then optimises a language-model policy to maximise that reward minus a penalty for moving away from a reference model.

Step 1: Supervised fine-tuning gives a starting policy

Supervised fine-tuning (SFT) is ordinary next-token training on examples of the behaviour you want. Ouyang et al. (2022) start from labeler-written prompts and prompts submitted through the OpenAI application programming interface, collect labeler demonstrations of the desired behaviour, and tune generative pre-trained transformer (GPT)-3 (Generative Pre-trained Transformer 3) on them with supervised learning.

Input: prompts with human-written good replies. Output: an SFT model that already follows instructions roughly. This model matters twice. It is the starting point for the later stages, and a frozen copy of it usually becomes the reference model. You now have a policy that imitates demonstrations; imitation is limited by how many good demonstrations people can write, which is why the next step collects cheaper judgments instead.

Step 2: A reward model learns from human comparisons

A reward model is a network that reads a prompt and a reply and outputs one number, higher meaning more preferred. It is trained on preference pairs, because people compare two replies more reliably than they score one. The same paper collects a dataset of rankings of model outputs for this stage.

The usual loss comes from the Bradley-Terry model: the chance that the chosen reply beats the rejected one is the sigmoid of the reward difference, and the loss is the negative log of that chance. With illustrative rewards of 1.5 for the chosen and 0.5 for the rejected reply, the difference is 1.0, the probability is 0.7311 and the loss is 0.3133.¹ Training pushes the gap up on every pair. Input: preference pairs. Output: a scorer you can call millions of times without a human.

Step 3: Reinforcement learning against a reference model

Reinforcement learning (RL) trains a policy to maximise a reward by trying outputs and adjusting toward the ones that score higher. In RLHF the policy is the language model, and proximal policy optimization (PPO, Schulman et al., 2017) is the usual algorithm: a policy-gradient method that limits how far each update can move the policy. Ouyang et al. also mix pretraining gradients into this objective, a variant they call PPO-ptx, to limit regressions on public datasets.

The reward it maximises is not the raw score. It is the reward model's score minus a Kullback-Leibler (KL) penalty: a charge, scaled by a coefficient, on the summed per-token log-ratio of the policy's probability to the reference model's probability for its reply. On one sampled reply that ratio can be negative. The frozen reference model feeds the penalty. Rafailov et al. (2023) describe the goal as maximising the estimated reward without drifting too far from the original model.

A worked number: the penalty on two replies

Take an illustrative coefficient of 0.05. An ordinary reply scores 2.0 and has a summed log-ratio of 8.0, so the penalty is 0.4 and the net is 1.6. An exploit reply scores 3.0 but has a summed log-ratio of 40.0, so the penalty is 2.0 and the net is only 1.0.¹ The log-ratio measures how much more likely the policy makes its reply than the reference model would, so a large value means the reply sits far from the reference's usual text. The coefficient sets how much each unit of drift costs. The penalty makes the exploit unprofitable: without it the exploit wins, 3.0 against 2.0.

1 · SFT model2 · Reward model3 · Policy and penaltyReplay

Step-by-step description

1. Demonstrations give a starting policy.
2. Human comparisons train a scorer.
3. Reward minus a drift penalty is maximized.

• Supervised fine-tuning gives a starting policy.
• The reward model scores replies and is trained on human comparisons.
• The policy maximizes the reward minus a penalty for drifting from the reference model.
• The frozen reference model feeds the penalty.

Text description of the diagram

• An ordinary reply has a raw reward of 2.0 and a net reward of 1.6.
• An exploit reply has a raw reward of 3.0 and a net reward of only 1.0.

The Staff-level answer

What makes it Staff-level is the third stage: it explains why the penalty exists, names the failure it prevents, and places the shortcut (DPO) without overclaiming.

The rubric

Three variants of this question

Walk me through the RLHF pipeline. Where does it usually fail?

Collect demonstrations and run SFT, collect comparisons and train a reward model, then run PPO with a KL penalty. It fails mainly at the proxy: the reward model is imperfect, so the policy can exploit it, and the labels it learns from are noisy and costly to gather.

What is the reference model in RLHF and what role does it play?

The reference model is a frozen copy of the starting policy. Its log-probabilities are the yardstick in the KL penalty, so the policy is rewarded for improving but charged for leaving the region where the reward model has seen data.

What is reward-model overoptimization, and how do you detect and bound it?

Gao et al. (2022) study how a proxy reward model and a gold reward diverge as the policy moves away from its starting point. Detect it by watching an independent signal, such as held-out human or stronger-judge comparisons, diverge from the training reward. A strong answer bounds it with the KL penalty, stops early when the held-out signal turns, and refreshes the reward model with new comparisons from the current policy.

Common mistakes

• "RLHF teaches the model new knowledge." (wrong) Broad knowledge comes from pretraining; the DPO paper describes it as learned by unsupervised models, with preference tuning for control of behaviour, and the Less Is More for Alignment study (LIMA) reports that almost all knowledge is learned during pretraining and that instruction tuning mainly teaches style, a reported tendency rather than a law. (right) "RLHF mostly steers behaviour the model already can produce."
• "A higher reward means a better model." (wrong) The reward is a learned proxy and can be exploited. (right) "I track held-out preference alongside the training reward."
• "DPO has no reference model because it is not RL." (wrong) DPO's loss compares policy and reference log-probabilities. (right) "DPO drops the reward model and sampling loop, not the reference."
• "The KL penalty makes the model safer." (wrong) It limits drift; it does not make the reward correct. (right) "It bounds how far the policy can exploit the proxy."

Follow-up questions

What is RLHF and what are the practical issues with it?

RLHF uses human comparisons to tune a model toward preferred replies. The practical issues are label cost and noise, reward hacking, the complexity and instability of the RL loop, and a possible loss of benchmark performance; the alignment tax page covers that last one, and Ouyang et al. (2022) report minimal regressions on public datasets with their setup.

What is the difference between RLHF and DPO?

Direct preference optimization (DPO) is a method that trains the policy directly on preference pairs with a classification loss and no separate reward model. In an illustrative pair with a coefficient of 0.1, the loss starts at 0.6931 when policy equals reference and falls to 0.5981 once the policy favours the chosen reply a little more than the reference does.¹ See DPO vs RLHF.

Has DPO displaced PPO-based RLHF?

Not universally. DPO is widely used in post-training because DPO itself needs no reward model and no sampling loop (a larger pipeline may still train a reward model for other stages); one public example is the Llama 3 report, §4 (Grattafiori et al., 2024), which describes post-training rounds of supervised fine-tuning, rejection sampling and DPO. RL against an explicit reward is still used, for example group relative policy optimization (GRPO) on verifiable reasoning rewards in the DeepSeek-R1 paper, §2. So "largely displaced" holds for many open-model pipelines but is not a universal fact.

What happens if you drop the KL penalty?

The policy is free to chase the reward model's mistakes, so the proxy score rises while real preference can stall or fall, and replies can drift into odd text. In the illustrative example the exploit reply would then beat the ordinary one, 3.0 against 2.0.¹ Gao et al. (2022) tie that divergence to distance from the initial policy.

How is the KL coefficient chosen?

The coefficient is a hyperparameter set by trial. A larger value keeps the policy closer to the reference and limits gains; a smaller one allows more reward and more exploitation. A sensible approach is to compare held-out preference against the training reward at several values; the 0.05 in the worked example is illustrative. See large language model (LLM) fine-tuning for the neighbouring choices.

Sources

1. Training language models to follow instructions with human feedback (arXiv 2203.02155) — arXiv (Ouyang et al., OpenAI), 4 Mar 2022
2. Direct Preference Optimization: Your Language Model is Secretly a Reward Model (arXiv 2305.18290) — arXiv (Rafailov et al.), 29 May 2023
3. Proximal Policy Optimization Algorithms (arXiv 1707.06347) — arXiv (Schulman et al.), 20 Jul 2017
4. Scaling Laws for Reward Model Overoptimization (arXiv 2210.10760) — arXiv (Gao, Schulman, Hilton), 19 Oct 2022
5. The Llama 3 Herd of Models (arXiv 2407.21783) — arXiv (Grattafiori et al., Meta), 31 Jul 2024
6. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (arXiv 2501.12948) — arXiv (DeepSeek-AI), 22 Jan 2025
7. LIMA: Less Is More for Alignment (arXiv 2305.11206) — arXiv (Zhou et al.), 18 May 2023