Read

Constitutional AI: Harmlessness from AI Feedback

Bai et al.

Artificial Intelligence Computer Science

This Anthropic paper by Bai et al. introduces Constitutional AI (CAI), a method for training a harmless and non-evasive AI assistant without any human feedback labels for harmful outputs. The approach uses a short list of natural language principles, a constitution, to guide an AI through a supervised critique-and-revision phase followed by reinforcement learning from AI-generated preference labels (RLAIF), producing a model that crowdworkers rate as both more helpful and more harmless than models trained with human harmlessness feedback.

Read the paper

Key takeaways

Constitutional AI operates in two stages: a supervised learning phase in which the model critiques and revises its own harmful responses according to constitutional principles, followed by a reinforcement learning phase using AI-generated preference labels (RLAIF) rather than human harmlessness labels.
The resulting RL-CAI model is preferred by crowdworkers over models trained with human harmlessness feedback, achieving a Pareto improvement on the helpfulness-harmlessness trade-off frontier compared to standard RLHF baselines.
Iterative critique and revision progressively reduces harmfulness, with each additional revision improving harmlessness preference model scores, and the critiquing step is shown to be more effective than generating revisions directly without critique.
Chain-of-thought reasoning during AI evaluations significantly improves the accuracy of harm identification, with ensembled chain-of-thought approaching the performance of preference models trained on human feedback labels for larger model sizes.
The constitutional approach produces non-evasive responses: rather than refusing harmful queries outright, the trained model engages with them by explaining its objections, reducing the tension between helpfulness and harmlessness that affected prior RLHF models.
Human oversight in this framework is compressed to a small, legible set of natural language principles rather than tens of thousands of human preference labels, making the values governing model behavior more transparent and easier to audit and adjust.

Reflections

The headline is that a model can learn harmlessness from its own AI feedback instead of tens of thousands of human labels. The part I actually care about is that the values end up written down as a short, legible constitution you can read, argue with, and change. That is the difference between governance you can audit and a black box trained to behave for reasons no one can reconstruct. I trust the first and distrust the second, in models and in organizations alike. The other quiet win is the non-evasive posture: a model that engages with a hard request and explains its objection is more useful, and more honest, than one that reflexively refuses. Encoding your values as explicit, editable principles rather than a pile of opaque examples is just good system design, whether the thing you are aligning is a model or a team.