What is a reward function in reinforcement learning and RLHF?

In this article

1. Before you start
2. Why interviewers ask this
3. Reward function from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page sits in the how large language models are trained learning path.

Why interviewers ask this

The reward function is where "what we want" becomes a number a computer can optimize, so every mistake in it is trained into the model. Interviewers ask about it to see whether you understand that a score is a stand-in for a goal, and what happens when the model chases the stand-in.

A weak answer says "the reward is the score the model gets." A strong answer separates the reward from the loss, explains the return over a sequence of steps, and says that in RLHF the reward comes from a learned model that can be fooled. A Staff-level answer adds the fix: a penalty that keeps the model close to where it started, and a check against real human judgement. Gao, Schulman and Hilton (2022) study exactly that failure, and RLHF is the setting where it shows up.

Reward function from first principles

Reward function

A reward function is a rule that maps what a system just did, or what it just wrote, to a single number, so that a reinforcement learning algorithm has something to maximize. A higher number means the outcome is better by the designer's definition.

Return

The return is the total reward collected over a run, usually with later rewards counted for less, so that one number summarizes how well the whole sequence of actions went.

In plain words

Think of training a dog. You cannot explain the goal, so you hand out a treat when it does something right. The treat is the reward, and the rule for when to give it is the reward function. The analogy breaks in one place. A dog wants treats, but it also has its own judgement. A model does exactly what raises the number, including things you never intended.

In one precise sentence: a reward function takes a situation and an action and returns a number. The system learns a policy that makes the expected total of those numbers as large as possible.

Step 1: A policy takes an action

The policy looks at the current situation and picks an action. In a game, the action is a move. For a language model, the situation is the prompt plus the words written so far, and the action is the next token, or the whole answer when you score at the end.

Input: a situation. Output: an action. Nothing has been scored yet. You now know where the number will attach: to what the policy did. The Spinning Up introduction lays out this loop of situation, action and reward in its reward and return topic.

Step 2: The reward function scores the result

After the action, the reward function returns one number. It can be a hand-written rule, such as plus one for reaching a goal and zero otherwise, or a check on the output, such as whether a math answer matches the key. Where the output is open text with no rule, the number comes from a reward model, covered in Step 4.

Input: the situation and the action. Output: a single number. Many steps may score zero, and that is normal: a reward that arrives only at the end is called sparse. The designer does not tell the system how to reach the goal, only how the result is graded, so a vague grade produces vague behaviour.

Step 3: The return sums the rewards over time

One reward does not describe a whole run, so the return adds them up. The discount factor is a number between 0 and 1, written γ. It multiplies each later reward by one more power of itself, so sooner rewards count more than later ones.

Here $r_t$ is the reward at step $t$ and $G$ is the return. A discount factor near 1 makes the system patient. A lower one makes it favour quick wins. Input: the list of rewards. Output: one number. The return sums the rewards over a run, and the policy is updated to raise expected return, the average of $G$ over many runs. The numbers are worked out below.

Step 4: In RLHF the reward is a learned score minus a penalty

In RLHF the reward function is a reward model. Ouyang et al. (2022) collect rankings of model outputs by human labellers and use them in a reinforcement learning stage that fine-tunes the model. A common way to train the reward model, described on the Bradley-Terry reward models page, is to push the score of the preferred answer above the score of the rejected one.

The reward model is only a proxy. Gao, Schulman and Hilton (2022) report that optimizing a proxy reward model too much can hurt true performance, in line with Goodhart's law, and that the setup adds a KL penalty coefficient to the reward. KL stands for Kullback-Leibler divergence; the KL penalty is a cost that grows as the tuned model's outputs drift away from the starting model's. The reward the policy sees is the model's score minus β times that cost:

In RLHF the penalty is subtracted from the reward model score before the return is summed. A high score with a large KL cost can total less than a modest score close to the start. Exploiting a flaw in a reward is called reward hacking, and the penalty is one defence against it. The RLHF page walks through the full loop.

1 · Act2 · Score3 · Sum and update4 · PenalizeReplay

Step-by-step description

1. The policy takes an action.
2. The reward function returns one number.
3. The return feeds back to the policy.
4. In RLHF a KL penalty lowers the score.

• A policy takes an action.
• The reward function scores the result with one number.
• The return sums the rewards over a run, and the policy is updated to raise expected return.
• In RLHF the penalty is subtracted from the reward model score before the return is summed.

Worked example

Take a three-step episode with rewards 0, 0 and 1, all illustrative. With a discount factor of 0.9 the return is 0.81, because the reward at the last step is multiplied by 0.9 twice; with a factor of 0.99 it is 0.9801.¹ The same reward of 1 arriving at step 10 instead of step 3 is worth 0.3874 at 0.9 and 0.9135 at 0.99.¹ A low discount factor therefore makes the system strongly prefer early reward.

Text description of the diagram

• A reward of 1 at step 3 is worth 0.81 at a discount factor of 0.9.
• The same reward at step 10 is worth 0.3874 at 0.9 and 0.9135 at 0.99.

Now the penalty. Two candidate answers get illustrative reward-model scores and KL costs. Answer A is short, with score 1.5 and KL cost 0.5. Answer B is long and padded, with score 2.5 and KL cost 6.0, because it sits far from the starting model. With β = 0.1 the totals are 1.45 for A and 1.9 for B, so B wins; with β = 0.5 they are 1.25 and −0.5, so A wins.¹

The reward model itself is trained on pairs. For a preferred answer scored $r_w$ and a rejected one scored $r_l$, the pairwise loss is:

Here σ is the logistic function, $\sigma (z)=1/(1+e^{-z})$, which turns any number into a value between 0 and 1. If the model scores the preferred answer 1.2 and the rejected one 0.4 (illustrative), the gap $r_w-r_l$ is 0.8, σ(0.8) is about 0.690 and the loss is −ln 0.690, about 0.3711. With the scores swapped the gap is −0.8, σ(−0.8) is about 0.310 and the loss is about 1.1711.¹ The loss falls as the preferred answer pulls ahead, which is how the proxy learns human taste.

Trade-offs and edge cases

A reward function trades ease of writing for fidelity to the goal. A hand-written rule is cheap and cannot be fooled the way a learned model can, but a checker that differs from the goal can still be gamed, for example code that passes the tests without solving the task. The group relative policy optimization (GRPO) page shows a method that trains reasoning with checkable scores. DeepSeek-AI's DeepSeek-R1 paper (January 2025) trains with rule-based accuracy and format rewards and avoids a neural reward model, citing reward hacking.

Lightman et al. (May 2023) compare outcome and process supervision and report that process supervision, which scores each reasoning step, outperforms outcome supervision on a benchmark of competition math problems. A learned reward model covers open-ended quality but can be gamed: Gao et al. (2022) find the overoptimization curve follows a different form for RL and for best-of-n sampling, and that its coefficients scale smoothly with reward model size.

Edge cases follow from this. A sparse reward gives little signal, so designers add extra intermediate rewards, called shaping, and a careless shaping term can teach the system to farm the bonus instead of reaching the goal. A reward model trained on one kind of prompt can mis-score another. And you can skip an explicit reward model: direct preference optimization (DPO) versus RLHF explains a method that trains directly on preference pairs.

A reward and a loss are different objects. Training maximizes the expected return of the rewards. Gradient descent minimizes a differentiable loss; in RL the loss is built from the rewards.

Text description of the diagram

• Training maximizes the expected return of the rewards.
• Gradient descent minimizes a differentiable loss; in RL the loss is built from the rewards.

How to answer it in an interview

The spoken structure: define the reward, say what the return adds, say where the reward comes from in RLHF, then name the proxy risk and the penalty.

Common mistakes

• "The reward is the loss." (wrong) A reward is a per-outcome signal whose expected return training maximizes; a loss is a differentiable function of the parameters that gradient descent minimizes. (right) "In RL the loss is built from the rewards, so they are linked but not the same thing."
• "A higher reward score always means a better answer." (wrong) The score is a proxy, and over-optimizing it can lower true quality. (right) "Higher reward means better only up to the point where the proxy stays reliable."
• "The reward model is the true objective." (wrong) It is trained to predict human preferences, so it inherits their gaps. (right) "It is a learned stand-in for the objective."
• "The KL penalty improves answers." (wrong) It limits drift. (right) "It trades some reward for staying close to the starting model."

Follow-up questions

What is the difference between a reward function and a loss function?

A reward function scores outcomes, and training maximizes their expected return; a loss function is a differentiable function of the model's parameters that the optimizer minimizes. In RL the loss is built from the rewards, and in RLHF the reward model itself is trained with its own loss.

What is reward hacking?

Reward hacking is when a system finds a way to raise the score without doing what the designer meant, because the reward function is an imperfect description of the goal. Overoptimizing a learned reward model is the language-model version, studied by Gao et al. (2022).

What is a reward model in RLHF?

A reward model is a network trained on human rankings of answers to predict which one a person prefers, and its score becomes the reward. It is described in the training recipe of Ouyang et al. (2022). RLHF is a fine-tuning stage that uses a reward instead of target text; the page on fine-tuning a large language model (LLM) covers the supervised stage that usually comes first.

How do you design a reward function?

State the outcome you want in a way that can be checked, prefer rules where a rule exists, and test the score against human judgement on fresh examples. Add a penalty or constraint for behaviour you do not want, and watch for the system gaming any term.

Can a reward come from a rule instead of a learned model?

Yes. When the answer is checkable, a program can return the reward, for example one for a correct final answer and zero otherwise. That avoids a learned proxy, though a checker that differs from the goal can still be gamed, and it is the idea behind training in how reasoning models work.

What is the difference between an outcome and a process reward model?

An outcome reward model scores only the final answer; a process reward model scores each reasoning step. Lightman et al. (2023) report that process supervision outperforms outcome supervision on a benchmark of competition math problems.

Sources

1. Scaling Laws for Reward Model Overoptimization (arXiv 2210.10760) — arXiv (Gao, Schulman, Hilton), 19 Oct 2022
2. Training language models to follow instructions with human feedback (arXiv 2203.02155) — arXiv (Ouyang et al.), 4 Mar 2022
3. Key Concepts in RL (Spinning Up) — OpenAI Spinning Up, 26 Oct 2026
4. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (arXiv 2501.12948) — arXiv (DeepSeek-AI), 22 Jan 2025
5. Let's Verify Step by Step (arXiv 2305.20050) — arXiv (Lightman et al.), 31 May 2023