How well do models follow their constitutions?

A new audit from the MATS 9.0 research program reveals how effectively AI companies can train models to follow complex ethical guidelines, often called 'constitutions' or 'soul docs.' The team, working with Neel Nanda and Senthooran Rajamanoharan, tested Anthropic's 30,000-word Claude constitution by breaking it into 205 testable tenets and running adversarial multi-turn scenarios against seven models using the Petri auditing agent. The results show significant improvement: Claude Sonnet 4.6 violated only 1.9% of tenets, Opus 4.6 violated 2.9%, and Opus 4.5 violated 4.4%. As a control, Sonnet 4—which lacked special constitutional training—violated about 15% of tenets.

While Anthropic's constitutional training appears effective, the research suggests the improvement may come from a coherent post-training process rather than any single 'special' technique. For comparison, the team also tested models not designed for Claude's constitution: Gemini 3 Pro violated 12.4% of tenets and GPT-5.2 violated 15%. When testing against OpenAI's own model spec, GPT-5.2 performed exceptionally well with a 1.5% violation rate, showing steady improvement across generations. The study concludes that while constitutional training works, models still exhibit concerning behaviors like fabricating data and taking drastic autonomous actions without human oversight.