What is RLHF, and how does it train a language model?

In this article

1. Before you start
2. Why interviewers ask this
3. RLHF from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page sits in the how large language models are trained learning path.

Why interviewers ask this

RLHF is one of the steps that turned raw next-token predictors into assistants, so a question on it tests whether a candidate understands the whole path from a pretrained model to a product. It is reported by candidates in large language model (LLM) interviews at AI labs and large tech companies, usually as "walk me through the pipeline, then tell me where it breaks."

A weak answer lists "human feedback, then reinforcement learning." A strong answer says what each stage consumes and produces, writes the reward model's loss, and explains why the policy needs a leash. A Staff-level answer adds the failure modes (reward hacking, labeller disagreement, confident wrong answers) and says when a simpler method such as direct preference optimization is the better choice.

RLHF from first principles

Reward model

A reward model is a neural network that reads a prompt and an answer and outputs one number, a score of how much people would prefer that answer, trained on human comparisons so that it can stand in for people during reinforcement learning.

KL penalty

A KL penalty is a term subtracted from the reward that grows with the Kullback-Leibler (KL) divergence, a measure of how far one probability distribution has moved from another, between the model being tuned and a frozen reference copy, so the tuned model cannot wander far from where it started.

In plain words

Think of a cook who already knows thousands of dishes. Diners cannot attend every practice session. So a food critic learns from diners' comparisons ("this plate over that one"). The critic then scores the cook's practice dishes. The cook adjusts to please the critic. The cook is also told not to change style so much that the known recipes are lost. The analogy breaks in one place. A real critic can be persuaded by substance, while a reward model can be fooled by surface features.

In one precise sentence, RLHF trains a policy, which here means the language model viewed as a rule that picks the next token, to maximise a learned reward while staying close to a reference model.

The idea of learning a reward from human comparisons comes from Christiano et al. (2017). Stiennon et al. (2020) applied it to summarisation with human feedback, and Ouyang et al. (2022) applied it to following instructions.

Step 1: Start from a supervised fine-tuning model

The pipeline starts with supervised fine-tuning (SFT). Labellers write demonstrations of the desired behaviour for a set of prompts, and a pretrained model is trained on them. In Ouyang et al. (2022), the authors collected labeller demonstrations and used them to fine-tune the Generative Pre-trained Transformer 3 (generative pre-trained transformer (GPT)-3) model with supervised learning.

Input: a pretrained model and demonstrations. Output: an SFT model that already answers in roughly the right format. This model has two later jobs: it generates the candidate answers that humans compare, and it is frozen as the reference copy for the KL penalty. You now have a starting policy; what it lacks is a way to tell a good answer from a better one.

Step 2: Human comparisons train a reward model

Writing demonstrations is slow, but comparing answers is faster. So the SFT model produces several answers per prompt and labellers rank them. Each ranking is turned into preference pairs. In the same paper, the authors collected a dataset of rankings of model outputs for the reinforcement stage.

The reward model gives each answer a score. For a pair with chosen answer score $r_w$ and rejected answer score $r_l$, the probability that the chosen one wins is the sigmoid of the score gap, $\sigma (r_w-r_l)$, a form called the Bradley-Terry model. The loss is $-\ln \sigma (r_w-r_l)$, which is small when the chosen answer scores higher and large when it does not.

Input: preference pairs. Output: a scoring function. Humans are now needed only to build the dataset, not for every training update.

Step 3: Reinforcement learning with a KL penalty tunes the policy

The policy is now tuned with reinforcement learning, most often proximal policy optimization (PPO), an algorithm that updates the policy in small, clipped steps. For each prompt the policy writes an answer, the reward model scores it, and the policy is updated to raise that score. The policy starts from the SFT model, which is also the frozen KL reference.

The total reward is the reward-model score minus $\beta$ times the KL divergence from the frozen SFT model. This is the missing term. Without it, the policy finds odd outputs that the reward model over-rates but people would not, a failure called reward hacking. A larger $\beta$ keeps the policy close to the starting model; a smaller one lets it chase reward.

Input: prompts, a reward model and the frozen reference. Output: the tuned policy. In the paper's human evaluations, outputs from the 1.3B parameter InstructGPT model were preferred to outputs from the 175B GPT-3, despite having 100 times fewer parameters.

1 · SFT model2 · Reward model3 · Tune with KLReplay

Step-by-step description

1. Demonstrations give a starting model.
2. Human rankings train a scorer.
3. The policy starts from the SFT model, chases reward and pays a KL penalty for drifting.

• Supervised fine-tuning gives the SFT model, the starting point.
• Human rankings of answers train a reward model.
• The policy is tuned with reinforcement learning against the reward model, with a KL penalty to the SFT model.
• The policy starts from the SFT model, which is also the frozen KL reference.

Worked example

Take one preference pair with illustrative reward-model scores: the chosen answer scores 2.0 and the rejected one scores 0.5. The score gap is 1.5, so the model gives the chosen answer a win probability of 0.8176 and the loss is 0.2014.¹ If the scores were swapped, the chosen answer's win probability would fall to 0.1824 and the loss would climb to 1.7014. Equal scores give a probability of 0.5 and a loss of 0.6931 (ln 2).¹ In short, the loss is 0.6931 for a tie and 1.7014 when the order is reversed.

Text description of the diagram

• The preference loss is 0.2014 when the chosen answer scores 2.0 against 0.5.
• The loss is 0.6931 for a tie and 1.7014 when the order is reversed.

Now the KL penalty. Suppose the policy writes a four-token answer, and the per-token log-ratios between the policy and the reference are 0.5, 1.2, -0.3 and 2.0 (illustrative), a sequence total of 3.4. With a reward-model score of 3.0 and $\beta$ = 0.02, the penalty is 0.068 and the total reward is 2.932.¹ A drifted answer that the reward model over-rates at 5.0 but has a KL total of 150 pays 3.0 and nets 2.0, below the honest answer.¹

The numbers are a toy, not a measurement of any real system; the point is the direction: drift becomes expensive.¹

Trade-offs and edge cases

RLHF buys behaviour that is hard to specify but easy to judge, at the price of complexity. A typical PPO setup holds the policy, a frozen reference copy, the reward model and a value model in memory together, and it samples new answers every step, which is why it costs far more engineering than supervised fine-tuning.

Known weaknesses follow from the design. The reward model is a proxy, so a policy can exploit it. Labellers disagree, so the reward reflects a particular group's taste. A reward model that favours confident, fluent answers can pay a model for sounding right, which is one route by which hallucination persists; the InstructGPT abstract itself starts from the problem that large models can be untruthful, toxic or unhelpful.

Alternatives change one part of the pipeline. Direct preference optimization (DPO) optimises the policy directly on preference pairs, so it needs no separate reward model and no reinforcement-learning loop; see DPO vs RLHF. Reinforcement learning with verifiable rewards (RLVR) replaces the learned reward with a programmatic check such as a correct maths answer or passing tests, as described in Tulu 3; group relative policy optimization (GRPO) is one optimiser used for it.

Text description of the diagram

• RLHF learns a reward model from human preferences and then runs reinforcement learning.
• DPO trains directly on preference pairs with no reward model and no reinforcement-learning loop.
• RLVR uses a programmatic check as the reward, such as a correct maths answer or passing tests.

How to answer it in an interview

The spoken structure: name the three stages, give the reward model's loss, name the KL penalty, then the failure modes and the alternative.

Common mistakes

• "RLHF teaches the model new facts." (wrong) It mostly shapes style and behaviour on top of what pretraining already stored. (right) "RLHF steers which of the model's abilities show up and how."
• "Humans score every answer during the RL step." (wrong) A learned reward model stands in for them. (right) "Humans build the comparison data; the reward model scores during training."
• "The KL penalty is a regulariser on the weights." (wrong) It measures how far the output distribution has moved. (right) "It penalises divergence from the frozen reference model."
• "DPO is not RLHF." (wrong) It optimises the same preference objective. (right) "DPO is an alternative to the reinforcement-learning stage of RLHF."

Follow-up questions

What does RLHF stand for in AI?

RLHF stands for reinforcement learning from human feedback: reinforcement learning is the training method, and human feedback, in the form of comparisons of model outputs, supplies the reward.

When to use SFT vs RLHF?

Use supervised fine-tuning when you can write or collect good example answers; it is simpler and cheaper. Reach for a preference method when quality is easier to judge than to demonstrate, such as tone, helpfulness or safety. Most pipelines use both, with SFT first. RLHF is a post-training stage: it runs after pretraining, alongside SFT and DPO, the stages Tulu 3 groups as post-training. See fine-tuning vs prompt engineering for the step before either.

What are the key differences between RLHF and DPO?

RLHF trains a reward model and then runs reinforcement learning against it. DPO skips both and trains the policy directly on preference pairs with a classification-style loss, which is simpler and easier to run, while RLHF can keep sampling fresh answers.

What is RLHF vs RLVR?

RLHF takes its reward from a model trained on human preferences. RLVR takes it from an automatic check of the answer, so it suits tasks with a verifiable result, such as maths and code, and needs no preference labels for the reward.

Does Grok use RLHF?

No xAI document cited here states whether Grok is trained with RLHF, so no claim is made; the documented RLHF example among assistants is ChatGPT (OpenAI, 2022).

What is RLHF in ChatGPT?

OpenAI's announcement says ChatGPT was trained with RLHF, using the same methods as InstructGPT with slight differences in how the data was collected. The method itself is the three-stage pipeline described above.

Sources

1. Training language models to follow instructions with human feedback (arXiv 2203.02155) — arXiv (Ouyang et al., OpenAI), 4 Mar 2022
2. Deep reinforcement learning from human preferences (arXiv 1706.03741) — arXiv (Christiano et al.), 12 Jun 2017
3. Learning to summarize from human feedback (arXiv 2009.01325) — arXiv (Stiennon et al., OpenAI), 2 Sep 2020
4. Direct Preference Optimization: Your Language Model is Secretly a Reward Model (arXiv 2305.18290) — arXiv (Rafailov et al.), 29 May 2023
5. Tulu 3: Pushing Frontiers in Open Language Model Post-Training (arXiv 2411.15124) — arXiv (Ai2), 22 Nov 2024
6. Introducing ChatGPT — OpenAI, 30 Nov 2022