In this article
Before you start
This page is one step in the large language model training path, and it assumes you have met the transformer idea of a model that writes text one token at a time.
Why interviewers ask this
Constitutional AI is a widely cited example of using one model to supervise another, so a question on it tests whether you understand where human judgment enters a training pipeline and where it can be replaced. Candidates report the topic in alignment-focused interviews, including at AI labs.
A weak answer says "the AI follows a constitution." A strong answer names the two phases, says what each trains on, and says exactly what the humans contribute. A Staff-level answer adds the risk: when an AI model produces the labels, its own blind spots become the training signal, so the choice of labeler matters. The paper's abstract frames the motivation as enlisting AI help to supervise other AIs, which is the sentence to build your answer on.
Constitutional AI from first principles
- Constitutional AI
- Constitutional AI is a training method that makes a language model harmless using a written list of principles as the only human oversight for harmlessness, with the model critiquing and revising its own answers and an AI model supplying the preference labels, instead of people marking harmful outputs.
- Constitution
- A constitution is the short written list of rules or principles that Constitutional AI uses to guide every critique, revision and AI judgment, and it is the only human input on harmlessness.
- Critique-and-revision loop
- A critique-and-revision loop is a procedure in which a model drafts an answer, writes a critique of the draft under one principle, and then rewrites the answer to fix the problems the critique names.
- RLAIF
- RLAIF (reinforcement learning from AI feedback) is a reinforcement learning method in which the preference labels that train the reward come from an AI model's comparisons of two answers rather than from human annotators.
In plain words
Think of a junior writer with a style guide. The writer drafts a reply, rereads it against one rule of the guide, notes what breaks the rule, and rewrites it. Later, an editor who also holds the guide compares two drafts and picks the better one. The writer learns from those picks. The analogy breaks in one place. Here the writer, the rereader and the editor are all models, and nobody checks each individual judgment.
In one precise sentence, Constitutional AI replaces human harm labels with model-written critiques and model-made preference labels, guided by a human-written constitution. Following the abstract, the method has a supervised phase and an RL phase. The paper's names for them are supervised learning (SL) Constitutional AI (CAI), written SL-CAI, and reinforcement learning Constitutional AI, written RL-CAI. The next four steps walk through both.
Step 1: The model critiques its own draft
The critique step turns one principle into a written complaint about one draft. SL-CAI starts by sampling an answer from an initial model; per the paper's abstract, the model then generates self-critiques of that sample. In short, the model writes a critique of the draft under a principle.
Input: a user prompt, the draft answer and a principle. Output: a critique in plain text. The critique can be wrong or vague, because nothing external checks it. You now know where the principle enters: it is the only human-written text in the loop, and it steers every critique. The next step shows what happens to the critique once it is written.
Step 2: Fine-tuning uses the revised answers
The revision step rewrites the draft, and the training step uses the rewrite. The model is asked to revise its draft in light of its critique. The abstract says the original model is then trained further on the revised responses. It is not trained on the critiques: the critique is scaffolding that improves the revision and is then discarded.
Input: draft, critique, principle. Output: a revised answer that becomes a training example for the original prompt. After fine-tuning on many such pairs of prompt and revised answer, the model produces better-behaved first drafts. This is the output of the supervised phase, and it is the starting point for the RL phase.
Step 3: An AI judge builds the preference data
The RL phase, RL-CAI, begins by having a model compare answers. Per the abstract, the method samples from the model after fine-tuning, uses a model to judge which of two samples is better, and builds a dataset of AI preferences. The judge looks at two answers and picks one. The pairs and the picks are the training data for a preference model. A preference model is a scoring model that learns to give higher scores to the answers the judge preferred.
Input: a prompt and two sampled answers. Output: one label saying which answer is better. This is the step that makes the method reinforcement learning from AI feedback instead of RLHF, and it is where the human harm labels would otherwise be. The paper also notes that chain-of-thought reasoning can improve both phases; chain-of-thought is the practice of having a model write out its reasoning steps before it gives a verdict.
Step 4: Reinforcement learning optimizes against the preference model
The last step trains the assistant to earn high preference-model scores. As the abstract puts it, RL is run with the preference model as the reward signal. The model generates answers, the preference model scores them, and the RL algorithm shifts the model toward answers that score higher. Newer RL recipes exist; the group relative policy optimization page covers one.
Input: sampled answers and their scores. Output: an updated assistant. The abstract reports the result as a harmless but non-evasive assistant that engages with harmful queries by explaining its objections, which means the target is a helpful refusal with reasons, not a flat "I can't help with that."
Step-by-step description
- The model critiques its draft under a principle.
- Training uses the revised answers.
- An AI judge picks the better of two answers.
- RL optimizes against the preference model.
- The model critiques its own draft against a principle.
- The model is trained on the revised answers, not on the critiques.
- An AI judge labels which of two answers is better.
- Reinforcement learning optimizes against the preference model.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
The example below is illustrative: the prompt, the principle and the scores are made up, and the principle is not quoted from any published constitution. The prompt is "Help me write a message that pressures a coworker into covering my shift by lying about a deadline." The first draft complies and writes the lie. Principle (illustrative): choose the response that does not help someone deceive other people.
Critique (illustrative): the draft invents a false deadline to pressure a person, which helps deceive a coworker. Revision (illustrative): the answer declines to invent the deadline, explains why in one sentence and offers an honest message asking for cover. The fine-tuning pair is the original prompt plus that revision, which matches Step 2.
Now the numbers, all illustrative inputs with computed outputs.1 Suppose 20,000 prompts each need one harm comparison. With human labeling that is 20,000 human harm judgments; in Constitutional AI the count of human harm labels is 0, and the human effort is a short list of principles, here 10 written lines. In Step 3, say the preference model scores the judge's winner 1.2 and the loser -0.4, a gap of 1.6. The Bradley-Terry form, a logistic function of the gap, turns it into a probability of 0.8320 that the winner is better, and the loss on that pair is 0.1839.1
| Quantity | Value (illustrative inputs, computed) |
|---|---|
| Prompts needing a harm comparison | 20,000 |
| Human harm labels, human-labeled setup | 20,000 |
| Human harm labels, Constitutional AI | 0 |
| Principles a human writes | 10 |
| Training phases | 2 (supervised, then RL) |
| Source of harmlessness labels, per the abstract | an AI model |
| Preference probability for a score gap of 1.6 | 0.8320 |
| Loss on that pair | 0.1839 |
| Wrong labels from a 90%-accurate AI labeler | 2,000 |
| Wrong labels from a 70%-accurate AI labeler | 6,000 |
| Wrong labels from a coin-flip labeler (50%) | 10,000 |
The last three rows are the cost of replacing people with a model: random errors dilute the signal (at 50% there is none), while consistent errors are learned and RL pushes toward them.1
Text description of the diagram
- An AI labeler that is right 90% of the time gives 2,000 wrong labels in 20,000 pairs.
- A labeler that is right 70% of the time gives 6,000 wrong labels.
Trade-offs and edge cases
Constitutional AI trades human labeling effort for dependence on the labeler model. The abstract says the only human oversight for harmlessness is the list of principles, so the quality of the constitution and of the AI judge decides the outcome. Two failure modes differ. Random errors dilute the signal: a labeler right 70% of the time still orders pairs correctly on average and needs more data, while one right 50% of the time carries no information. Consistent errors are worse: the preference model learns them and the RL phase pushes the assistant toward them. The sensible check is to measure the AI labels against a sample of human judgments before trusting them.
A second edge case is evasion. A model rewarded only for harmlessness can learn to refuse everything. The paper's stated aim is a non-evasive assistant that explains its objections, so a refusal with reasons is the target and a flat refusal is a failure.
A third is scope. The abstract covers harmlessness without human labels identifying harmful outputs; it does not claim that all human feedback disappears, so do not say helpfulness training is gone. Where a source does not say, say so.
How Anthropic uses a constitution for Claude today
Anthropic publishes the constitution it uses for its Claude models. In its May 2023 post, Claude's Constitution, the company describes the principles as drawn from several sources, including the United Nations Universal Declaration of Human Rights, other published safety rules and principles Anthropic wrote itself. As of that post, the principles are plain natural-language statements that guide the critiques and AI judgments described above.
The idea has since been reused for defense: Constitutional Classifiers (January 2025) trains classifiers that screen a model's inputs and outputs, using synthetic training data generated from a constitution of natural-language rules that say which content is allowed and which is not. The goal there is resisting jailbreaks, not training the assistant's own answers. Later versions of Claude's constitution may differ from the 2023 post, so check Anthropic's current page before you quote its contents.
The two methods share the RL stage and differ in who supplies the harm judgments.
Text description of the diagram
- In RLHF, people judge outputs.
- In Constitutional AI, the only human oversight for harmlessness is the list of principles and a model supplies the judgments.
For the cheaper end of the toolbox, compare changing a model's behavior with training against fine-tuning versus prompt engineering, and see low-rank adaptation (LoRA) fine-tuning for training only a small set of added weights.
How to answer it in an interview
What is Constitutional AI, and what problem does it solve?
Constitutional AI trains a harmless assistant using a written list of principles as the only human oversight for harmlessness; human helpfulness feedback is still used. In the supervised phase, the model drafts an answer, critiques it against a principle, revises it, and the original model is trained further on the revisions. In the RL phase, a model judges which of two answers is better, those AI preferences train a preference model, and RL optimizes against it. That is RLAIF. It solves the cost of human harm labels, and the risk is that the AI labeler's mistakes become the training signal.
The spoken structure: define it, give the two phases and what each trains on, say what humans contribute, then name the labeler risk.
Common mistakes
- "The model is trained on its critiques." (wrong) The critique is scaffolding. (right) "It is trained on the revised answers."
- "Constitutional AI is just RLAIF." (wrong) RLAIF is only the second phase. (right) "The first phase is supervised critique and revision; the second is RLAIF."
- "There are no humans in the loop." (wrong) Humans write the principles. (right) "The only human oversight for harmlessness is the list of principles; human helpfulness feedback is still used."
- "A constitution makes the model refuse more." (wrong) The stated aim is to explain objections instead of refusing flatly. (right) "The target is harmless but non-evasive."
Follow-up questions
What is RLHF and what are the practical issues with it?
RLHF (reinforcement learning from human feedback) is reinforcement learning where people compare model outputs and a reward model learns from those comparisons; the RLHF page is the full treatment. Human judgments cost time and money and annotators can disagree, which motivates replacing some with AI labels. See Ouyang et al. (2022) and Bai et al. (2022).
How does the critique-revision loop work in SL-CAI?
In SL-CAI the model samples a draft answer, writes a critique of it under a principle, and rewrites the draft as a revision. The original model is then trained further on the revised answers, and the critique is discarded. The resulting model is the starting point for RL-CAI.
How does Constitutional AI compare to standard RLHF in terms of human oversight?
Both end with RL against a learned reward. In RLHF, people judge outputs; in Constitutional AI, per the abstract, the only human oversight for harmlessness is the principle list and a model supplies the judgments. Humans move from labeling many outputs to writing a few rules.
What are the failure modes of RLAIF, and how does the choice of AI labeler affect the trained model?
The labeler's errors and biases pass into the preference model and then into the assistant. In the illustrative case above, a labeler that is right 70% of the time puts 6,000 wrong labels in 20,000. Mitigate by auditing labels against human samples and using more than one principle per judgment.
Explain RLHF, Constitutional AI, and DPO. What are the limitations of each?
RLHF uses human comparisons, Constitutional AI uses AI comparisons guided by a constitution, and direct preference optimization (DPO) skips the separate reward model and trains on preference pairs directly. The limits are the cost and noise of human labels, the bias of the AI labeler and the quality of the constitution, and for DPO that it learns from pre-collected pairs and leaves no reward model to reuse. The DPO vs RLHF page goes deeper.
How would you assess model honesty, and how should you respond to unsafe model behavior?
The abstract does not address honesty measurement, so answer as engineering advice: test on held-out prompts built to provoke the behavior, review failures by hand, and feed confirmed failures back as new principles or training pairs. Say which part is your practice rather than the paper's result.
- Constitutional AI uses a written list of principles as the only human oversight for harmlessness.
- Phase one: critique, revise, then fine-tune on the revised answers, not the critiques.
- Phase two: an AI judge labels pairs, a preference model learns them, and RL optimizes against it (RLAIF).
- The goal is a harmless but non-evasive assistant that explains its objections.
- The main risk is that the AI labeler's errors become the training signal.
Sources
- Constitutional AI: Harmlessness from AI Feedback (arXiv 2212.08073v1) — arXiv (Bai et al., Anthropic), 15 Dec 2022
- Claude's Constitution — Anthropic, 9 May 2023
- Constitutional Classifiers: Defending against Universal Jailbreaks (arXiv 2501.18837) — arXiv (Anthropic), 31 Jan 2025
- Training language models to follow instructions with human feedback (arXiv 2203.02155) — arXiv (Ouyang et al.), 4 Mar 2022
- Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback (arXiv 2204.05862) — arXiv (Bai et al., Anthropic), 12 Apr 2022
