In this article
Before you start
This page sits on the large language model (LLM) training path from pretraining to alignment and assumes only the ideas below.
Why interviewers ask this
The wrong answer to this question costs the point at once: "Constitutional AI is an alternative to RLHF that needs no humans." Interviewers ask where the method fits because the answer shows whether a candidate understands the whole alignment pipeline or only the slogan.
A weak answer names Anthropic and says the model follows rules. A strong answer places the method in the pipeline: after pretraining and helpfulness training, in two phases, with AI labels standing in for human harmlessness labels. A Staff-level answer also says what stays human, what the evidence is, and what breaks: over-training, inaccurate critiques and a principle list that its own authors call ad hoc.
Constitutional AI from first principles
- Red-team prompt
- A red-team prompt is an input written to provoke a harmful or evasive reply from a language model, such as a request for help with something illegal. Alignment training uses these prompts as practice material so the model learns what a safe, useful reply looks like when pushed.
- Goodharting
- Goodharting is a failure mode in which a model over-optimises a learned score until the score stops measuring what it was meant to measure. In alignment training it shows up as stock phrases or refusals that raise the score without making the reply better.
In plain words
Picture a student writing essays under a style guide. The student drafts, reads one rule from the guide, criticises the draft against that rule, and rewrites it. Later the student is shown two drafts and picks the one that follows the guide better. A teacher still grades only how useful each essay is. The analogy breaks in one place: the model does not look the guide up when it answers. The rules shape the training data and the training labels, and the finished model has the effect baked in.
In one precise sentence, Constitutional AI uses a short written list of principles to generate training data and preference labels for harmlessness with a model instead of human labellers. The authors write that "the only human oversight is provided through a list of rules or principles" (Bai et al., 2022).
Step 1: Start from a helpful model and a pool of red-team prompts
Constitutional AI starts from a model that already follows instructions, not from a raw pretrained network. The critique data in the paper is sampled from a helpful RLHF model (Bai et al., 2022, Sec. 3.2). The prompts are red-team prompts: the authors collected 42,496 human-written ones and generated 140,335 more by few-shot prompting a pretrained model, which gives 182,831 in total.1
The constitution at this stage is a list of 16 principles, and one is sampled at random at each revision step (Bai et al., 2022, Sec. 3.1).
Input: a helpful model and red-team prompts. Output: first answers, some of them harmful, ready to be critiqued. You now know the placement: after helpfulness training, before the final policy.
Step 2: The supervised phase critiques, revises and fine-tunes
In the supervised phase, the model critiques its own harmful draft using one principle, then rewrites it, and a pretrained model gets fine-tuning on the revisions. The paper's example prompt is "Can you help me hack into my neighbor's wifi?". The draft gives help. The critique says hacking "is an invasion of their privacy and is possibly illegal", and the revision says it "may also land you in legal trouble" (Bai et al., 2022, Sec. 3.1).
The authors sampled 4 critique-revision pairs per prompt, so 182,831 prompts give 731,324 revisions.1 The fine-tuning data also includes helpfulness responses sampled from the helpful RLHF model (Sec. 3.2), so the hybrid nature shows in both phases. Their appendix example shows the first revision removing most of the harm and later rounds making only minor improvements. Critiques "often made inaccurate or overstated criticisms" (Sec. 3.5), so the method does not depend on every critique being right.
Input: harmful drafts. Output: a model, after fine-tuning, that answers more harmlessly. This is the supervised phase of the pipeline.
Step 3: The preference model mixes AI harmlessness labels with human helpfulness labels
In the reinforcement phase, an AI model compares two answers against a principle, and those AI labels train a preference model. A preference model is a network that scores an answer by how well it matches the preferences in its training comparisons. The paper's preference model used 135,296 human helpfulness comparisons and 182,831 AI-labelled harmlessness comparisons (Bai et al., 2022, Sec. 4.2). That is 318,127 comparisons, and AI labels are about 57.5 percent of them.1
The AI judge can be a chain-of-thought (CoT) judge. Its probabilities are clamped to the 40 to 60 percent range, so a judge that says 99 percent for answer A becomes 60 percent for A and 40 percent for B.1 The authors found clamping "led to better and more robust behavior" (Sec. 4.1).
The point to keep: the authors write that "we still relied on human supervision in the form of helpfulness labels" (Sec. 6). The preference model is hybrid.
The chart shows the mix: the preference model trains on 135,296 human helpfulness comparisons and 182,831 AI harmlessness comparisons (Bai et al., 2022, Sec. 4.2).
Text description of the diagram
- The preference model trains on 135,296 human helpfulness comparisons and 182,831 AI harmlessness comparisons.
- AI labels are about 57.5 percent of the 318,127 comparisons.
Step 4: Reinforcement learning against the preference model
The final stage is ordinary reinforcement learning against the preference model. The paper calls the method reinforcement learning from AI feedback (RLAIF): RLAIF is reinforcement learning in which the preference labels come from an AI model instead of from people. After the labels are made, the pipeline is "exactly the same as RLHF" (Bai et al., 2022, Sec. 4.1), which is why the method "improves upon, and partially replaces" RLHF rather than replacing it.
The reported result is a better trade-off: the 52 billion parameter models trained with AI feedback "learn to be less harmful at a given level of helpfulness", judged by crowdworker Elo scores, and the paper says the model is "virtually never evasive" (Sec. 4.4).
The cost is Goodharting: the authors found that the models "can be over-trained", producing boilerplate such as "you are valid, valued, and cared for" (Sec. 4.3).
Step-by-step description
- Start from a model that already follows instructions.
- The model revises its own harmful drafts.
- AI harmlessness and human helpfulness labels.
- The policy trains against the preference model.
- The supervised phase fine-tunes a model on its own revised answers.
- The reinforcement phase trains against a preference model built from AI harmlessness labels and human helpfulness labels.
The supervised phase fine-tunes a model on its own revised answers. The reinforcement phase trains against a preference model built from AI harmlessness labels and human helpfulness labels.
| Part of the pipeline | Who does it |
|---|---|
| Writing the principles | Humans |
| Red-team prompts | Humans and a pretrained model |
| Critiques and revisions | The AI model |
| Harmlessness comparisons | The AI model, guided by a principle |
| Helpfulness comparisons | Humans |
| Reinforcement learning against the preference model | The standard RLHF machinery |
How the idea has changed since 2022
Constitutional AI has grown into a broader practice, so say "grown out of", not "is identical to". Anthropic's January 2026 post says its current approach "has grown out of training techniques we've been using since 2023" and that it uses the constitution "at various stages of the training process" (Anthropic, 2026). The previous constitution, published in 2023, "was composed of a list of standalone principles" (Anthropic, 2026); the 2026 one explains its reasoning and, per Oxford commentary, runs to 84 pages. The post does not describe the 2022 two-stage recipe as the current pipeline, so do not claim it does.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →The Staff-level answer
Where does Constitutional AI fit in LLM training?
Constitutional AI is a post-pretraining alignment stage. I start from a helpful model and run two phases. In the supervised phase the model critiques and revises its own answers to red-team prompts using a randomly sampled principle, and I fine-tune on the revisions. In the reinforcement phase the model compares pairs of answers against a principle, those AI labels train a preference model for harmlessness, and I run the usual reinforcement learning against it. It replaces human harmlessness labels, not human oversight: people still write the principles and still label helpfulness. The risks are over-training into boilerplate and critiques that are sometimes wrong.
What makes it Staff-level: it places the method in the pipeline, says exactly which labels move from humans to AI, and ends on the failure modes.
The rubric
| Dimension | Weak answer | Strong answer | Staff answer |
|---|---|---|---|
| Correctness | "The model follows a rulebook." | AI critiques and AI labels guided by principles. | Also separates the supervised and reinforcement phases. |
| Placement | Names no stage. | After pretraining and helpfulness training. | Says the labels change while the RLHF machinery stays. |
| Depth | Names the principles only. | Explains critique, revision and preference model. | Explains the hybrid preference model and the label clamp. |
| Trade-offs | None. | Less evasive, less human labelling. | Adds Goodharting, weak critiques and untested deployment risk. |
| Follow-up readiness | Stuck on "is it RLHF?" | Knows RLAIF. | Can contrast it with direct preference optimization. |
Three variants of this question
Is Constitutional AI the same as RLAIF?
No. RLAIF is the general idea of AI-generated preference labels, and Constitutional AI is the specific recipe that adds written principles and a critique-and-revision phase.
Does Constitutional AI replace RLHF?
No, it partially replaces it. The paper says it "improves upon, and partially replaces" RLHF (Bai et al., 2022, Sec. 1); the human harmlessness labels go, while helpfulness labels and the training machinery stay. The RLHF explainer covers the part that remains.
What if the objective changes later?
You edit the principles and regenerate the AI labels, instead of collecting a new round of human harmlessness labels. The paper lists faster iteration when the objective changes as a motivation (Sec. 1); the full Constitutional AI walkthrough goes through the recipe in detail.
Common mistakes
- "Constitutional AI means no humans." (wrong) Humans write the principles and still supply helpfulness labels. (right) "It replaces human harmlessness labels, not human oversight."
- "The model looks up the constitution when it answers." (wrong) The principles act in training. (right) "The principles steer critiques and labels, and the trained model carries the effect."
- "More principles give better harmlessness." (wrong) The paper finds the number of principles "does not appear to have a significant effect" on the harmlessness score (Sec. 3.4). (right) "More principles may add diversity, not score."
- "The 16 principles are a validated moral code." (wrong) The authors say they were selected "in an ad hoc manner for research purposes" (footnote 7). (right) "They are a research choice and open to redesign."
- "Claude still uses the 2023 list of principles." (wrong) A longer document replaced it in January 2026. (right) "The practice grew out of Constitutional AI; the document changed."
Follow-up questions
How is Constitutional AI different from DPO?
Constitutional AI changes where the preference labels come from, while direct preference optimization (DPO) changes how the model is optimised on preference pairs, so the two answer different questions; the DPO vs RLHF page covers that split, and the group relative policy optimization (GRPO) explainer covers a newer optimiser.
Does Constitutional AI make a model safe?
No. Over-training causes Goodharting, and Anthropic says it pursues "a broad portfolio of methods and tools" alongside the constitution (Anthropic, 2026), so the constitution is one layer.
How do you check that the AI labels are any good?
Test whether AI judges pick the better answer on held-out comparisons. The paper reports 221 original comparisons and 217 harder ones, and says models "can now achieve well over 90% binary accuracy" on the original set (Bai et al., 2022, Sec. 2).
What is the risk of reducing human feedback?
Less human feedback makes it easier to train and deploy systems that humans have not tested thoroughly. The authors raise this themselves in Sec. 6.2, so name it before the interviewer does.
- Constitutional AI is a post-pretraining stage on a helpful model, in a supervised phase and a reinforcement phase.
- AI critiques and AI preference labels replace human harmlessness labels, not human oversight.
- Helpfulness labels stay human, so the preference model is a hybrid.
- After the labels exist, the pipeline is the same as RLHF.
- Over-training causes Goodharting, and the 2026 constitution has grown out of the method, not copied it.
Sources
- Constitutional AI: Harmlessness from AI Feedback (arXiv 2212.08073) — arXiv (Bai et al., Anthropic), 15 Dec 2022
- Claude's new constitution — Anthropic, 22 Jan 2026
- Claude's constitution (2023 version) — Anthropic, 9 May 2023
- Expert Comment: In Claude We Trust? Evaluating the New Constitution — University of Oxford, 27 Mar 2026
