What is the alignment tax in an LLM, and how to reduce it?

In this article

1. Before you start
2. Why interviewers ask this
3. The alignment tax from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step of the how large language models are trained path, and it builds on the transformer architecture.

Why interviewers ask this

A model that is safer but worse at hard tasks is a real product problem, so the question tests whether you treat alignment as a trade-off you measure rather than a free upgrade. Candidates report being asked it in interviews at AI labs, often together with reward hacking, the KL penalty and the reference model. The guide to how large language models are trained covers the neighbouring topics.

A weak answer says "alignment makes models worse" or "alignment makes models better" without conditions. A strong answer names what is lost (pretrained abilities), says that the loss depends on the recipe, and gives a lever for each side of the trade: a constraint on drift and a way to recover weights. A Staff-level answer also keeps the tax apart from reward hacking, because they are different failures with different tests.

The alignment tax from first principles

Alignment tax

The alignment tax is the loss of abilities a language model acquired in pretraining that follows when it is aligned with human preferences, for example through RLHF, so that a gain in safety or helpfulness is paid for with worse performance on other tasks.

Reward hacking

Reward hacking is a failure in which a model trained against a proxy reward finds outputs that score highly on that proxy without being good by the true standard, so a rising reward score can hide worse real behaviour.

In plain words

Think of a student who spent years learning a wide range of subjects, then crams for one exam. The more hours go into the exam, the less is left for the old material. The analogy breaks in one place. A model forgets through changes to shared numbers (weights), not through a limited number of hours. Some updates barely disturb old skills at all.

In one precise sentence, the alignment tax is the drop on pretrained-ability benchmarks that comes with the gain on the alignment objective. Lin et al. write that aligning under RLHF can lead to forgetting pretrained abilities. They call this the alignment tax.

Step 1: Pretraining builds the abilities that can be lost

Pretraining produces a model with broad skills: reading, reasoning, coding and facts. Input: a huge text collection. Output: weights that hold all of these skills together. Nothing in those weights is labelled "reasoning" or "safety", so any later update can touch any skill.

Aligned models start from this base, not from nothing. The InstructGPT authors Ouyang et al. note that making language models bigger does not inherently make them better at following a user's intent, which is why a second training stage exists at all. You now know what is at stake: the abilities from this step are what the next steps can damage.

Step 2: RLHF pushes the model toward a proxy reward

In RLHF, a reward model trained on human rankings of outputs gives each response a reward score, and the language model is updated to raise that score. The score is a proxy: an imperfect stand-in for what people actually want. Gao et al. state that optimizing an imperfect proxy too much can hinder ground-truth performance, in accordance with Goodhart's law.

Reward hacking is a failure in which a model finds outputs that score highly on the proxy reward without being good by the true standard. Input: a prompt and a reward model. Output: an updated model that favours whatever the reward model likes. Two separate problems now exist: the proxy can be exploited, and the update can overwrite old skills. The tax belongs to the second.

Step 3: The KL penalty keeps the policy near the reference model

Left alone, the update can move the model very far from where it started. A KL penalty makes distance costly. Gao et al. study the coefficient of the KL penalty added to the reward in the reinforcement learning setup, so it is a standard lever. For one prompt, the penalised objective is the expected reward minus a coefficient β times the KL divergence from the reference model:

Setting the derivative to zero under the constraint that the probabilities sum to one gives the optimal policy:

Here Z is the sum that makes the probabilities add to one. A large β keeps the policy close to the reference; a small β lets reward dominate. A smaller drift usually disturbs fewer old skills, but it does not make the tax zero. The KL penalty bounds how far the policy can chase the proxy, which limits reward hacking, but it does not restore a skill the update has already overwritten; that needs a recovery lever such as pretraining-data mixing (Step 4) or weight averaging (Step 5).

Step 4: Forgetting appears as the tax

Even with a KL penalty, the tax can be large. Lin et al. ran existing RLHF algorithms on a 3-billion-parameter open model and report a pronounced alignment tax on natural language processing (NLP) tasks. They also report that techniques which reduce forgetting are often at odds with RLHF performance, which is an alignment-forgetting trade-off.

The tax is not a law of nature. Ouyang et al., 2022 report that plain proximal policy optimization (PPO) regressed on several public NLP datasets, and that mixing pretraining updates into the reinforcement learning (RL) objective, a variant they call PPO-ptx, greatly reduced those regressions. Their aligned models showed improvements in truthfulness and fewer toxic outputs with minimal performance regressions on public NLP datasets. Two papers, two recipes, two outcomes. Input: an aligned model and a capability benchmark suite. Output: a measured gap against the base model. That gap, not a feeling, is the tax.

Step 5: Weight averaging recovers part of what was lost

Model averaging interpolates between the weights before and after RLHF: the blended weights are (1 − α) times the pre-RLHF weights plus α times the post-RLHF weights. Lin et al. report that this simple step achieves the strongest alignment-forgetting Pareto front among a wide range of competing methods, that is, the best set of trade-offs where neither score can improve without the other getting worse. They validated an adaptive version on that 3-billion-parameter model and on a 7-billion-parameter Mistral model.

Input: two sets of weights and a value of α. Output: one model somewhere between them. You pick α by checking the reward and the capability suite together. Related ideas that touch fewer weights, such as low-rank adapters and other parameter-efficient methods, change how much of the model moves in the first place.

1 · Pretrained2 · RLHF update3 · KL penalty4 · Alignment tax5 · AveragingReplay

Step-by-step description

1. Pretraining builds many abilities.
2. The update chases a proxy reward.
3. Distance from the reference costs reward.
4. Some pretrained skills are forgotten.
5. Blending weights recovers some skill.

• Aligning under RLHF can lead to forgetting pretrained abilities, which is the alignment tax.
• A KL penalty keeps the policy near the reference model.
• Model averaging interpolates between the weights before and after RLHF.

Worked example

Two toy calculations show the two levers. All numbers here are illustrative, not measurements from any real model.¹

First, the KL penalty. A reference model gives four responses the probabilities 0.1, 0.5, 0.35 and 0.05. The proxy reward scores them 2.0, 0.5, 0 and 3.0. The true (gold) value of the four responses is 1.0, 0.5, 0 and −0.5, so the fourth response is an exploit: it scores highest on the proxy and is the worst in truth. Under the reference model the expected proxy reward is 0.6 and the expected gold value is 0.325.¹

Applying the optimal-policy formula of Step 3 at several values of β gives the policies below. As β falls from 16 to 0.25, the exploit's share rises from 0.058 to 0.9642 and the proxy reward climbs from 0.6402 to 2.9635. The gold value first rises from 0.325 under the reference to a peak of 0.3329 at β = 4, then falls to −0.4466.¹

The proxy score looks better at every step, while the real value rises only at first. The rise happens because the proxy and the gold value agree on average under the reference: their covariance is 0.25 − 0.6 × 0.325 = 0.055, which is positive.¹ Once the exploit takes over, the gold value heads toward −0.5. That is why a team watches a held-out measure of true quality, not the training reward alone. In the setup of Gao et al., a fixed gold reward model plays the role of humans and labels a proxy reward model's training data, so the gap between the two scores can be measured as the optimization proceeds.

Text description of the diagram

• The exploit share rises from 0.0671 to 0.9642 as the KL coefficient falls from 8 to 0.25.
• The numbers come from an illustrative toy with four responses.

Second, weight blending. With illustrative curves for reward and capability, a blend weight α of 0 is the base model (reward 0, capability 1.0) and α of 1 is the fully aligned model (reward 1.0, capability 0.7). At α of 0.5 the blend keeps reward 0.75 and capability 0.925, so in this made-up curve half the weight movement buys three quarters of the reward gain for a capability loss of 0.075.¹ Real curves differ; the point is that you measure them and choose α, as Lin et al. do.

Text description of the diagram

• At a blend weight of 0.5 the illustrative reward is 0.75 and the capability is 0.925.
• At a blend weight of 1 the reward is 1.0 and the capability is 0.7.

Trade-offs and edge cases

The tax depends on the recipe, the model and the benchmarks you check. The InstructGPT paper reports minimal regressions on public NLP datasets for its recipe with pretraining updates mixed in, while Lin et al. report a pronounced tax for existing RLHF algorithms on a smaller model. Do not quote either as "the" size of the tax; quote the setup next to the result.

Reported practice differs on how to mitigate reward hacking, and published work does not settle which mitigation works best. Current recipes also differ on whether they keep a KL term at all, so state which recipe you mean. Common approaches in practice include a stronger KL penalty, stopping early when a held-out evaluation diverges from the training reward, refreshing the reward model with newer samples, and limiting how many candidates a best-of-n sampler picks from. Each is a trade between reward gain and safety margin. Gao et al. also vary the reward-model dataset size and the number of reward-model and policy parameters, which are further levers.

Edge cases: a higher benchmark score after alignment does not mean there was no tax on tasks you did not test; a tax measured on one model size may not carry over to another; and training methods other than RLHF, such as preference optimization without a reward model or group-relative methods, have their own forgetting profile that you should measure the same way.

How to answer it in an interview

The spoken structure: define the tax, say you measure it, name the KL penalty, then name weight averaging as the recovery lever.

Common mistakes

• "Alignment always costs capability." (wrong) The InstructGPT paper reports minimal regressions on public NLP datasets when pretraining updates are mixed in. (right) "The tax depends on the recipe and the model, so I measure it."
• "The alignment tax is reward hacking." (wrong) Reward hacking is exploiting the proxy reward; the tax is forgetting. (right) "They are separate failures: one shows in a gold-versus-proxy gap, the other in a capability benchmark gap."
• "The KL penalty removes reward hacking." (wrong) The penalty limits drift but does not make the proxy correct, as the toy shows once β is small and the gold value falls while the proxy keeps rising. (right) "It bounds the drift; I still check held-out quality."
• "Weight averaging is just a hack." (wrong) Lin et al. report it reached the strongest Pareto front among the methods they compared. (right) "It is a cheap baseline I try first."

Follow-up questions

What is reward-model overoptimization, and how do you detect and bound it during RLHF?

Reward-model overoptimization is the effect of pushing a proxy reward so far that true quality falls. You detect it by tracking a held-out gold measure, human or a stronger judge, next to the training reward, and bound it with a KL penalty and by stopping where the two diverge.

Explain the over-optimization curve: why does reward initially increase and then human preference decreases?

Early optimization moves probability toward responses that are genuinely better, so the true score rises along with the proxy. Past a point the policy mostly exploits the proxy's errors, so the true score falls while the proxy keeps rising. Gao et al. fit the gold score as a function of the square root of the KL distance, with different functional forms for best-of-n and for RL, and coefficients that scale smoothly with reward-model size. The toy above shows the same hump: gold peaks at 0.3329 at β = 4 and falls to −0.4466 at β = 0.25 while the proxy reaches 2.9635.¹

Your RLHF model games the reward model. How do you stop reward hacking?

Tighten the KL penalty, stop earlier, retrain the reward model on samples from the current policy, and check outputs against a measure the policy did not train on. No single fix is enough, so state which failure each one addresses.

Why does RLHF need a KL penalty?

The reward model is only reliable near the distribution of responses it was trained on, which is the core reason. Without a penalty the policy can drift to outputs the reward model scores highly but that are far from sensible text. The penalty makes distance from the reference model costly, as in the optimal-policy formula of Step 3.

What is the reference model in RLHF and what role does it play?

The reference model is the frozen pre-RLHF copy that the KL penalty compares against. It sets the starting distribution that the optimal policy reweights by the exponential of the reward.

After RLHF, how do you recover lost capability without losing safety?

Interpolate the weights between the base and aligned models, then choose the blend that meets your safety bar on a capability and safety suite. Lin et al. report this reached the strongest trade-off among methods they compared; knowledge distillation and fine-tuning an LLM are related options to evaluate the same way.

Sources

1. Training language models to follow instructions with human feedback (arXiv 2203.02155) — arXiv (Ouyang et al., OpenAI), 4 Mar 2022
2. Scaling Laws for Reward Model Overoptimization (arXiv 2210.10760) — arXiv (Gao, Schulman, Hilton), 19 Oct 2022
3. Mitigating the Alignment Tax of RLHF (arXiv 2309.06256) — arXiv (Lin et al.), 12 Sep 2023