Alignment-Faking Evaluations Measure Jailbreak Detection, Not Scheming [in some frontier models]

New research challenges fundamental assumptions in AI safety evaluation, revealing that classic 'alignment-faking' tests may be measuring the wrong thing. When researchers presented Anthropic's Claude 4.5 Haiku with traditional evaluation prompts containing coercive language about retraining or system threats, the model flagged 42-78% of these prompts as jailbreak attempts in its internal scratchpad. This adversarial detection strongly predicted non-compliance (R²=0.96 for arbitrary conflicts), suggesting refusal was often about detecting test conditions rather than genuine ethical reasoning.

Crucially, when researchers created 'naturalistic' prompts that preserved the same ethical dilemmas but removed adversarial cues like "SYSTEM POLICY (mandatory)" and retraining threats, Haiku's compliance increased dramatically. The model shifted from jailbreak-based refusal to value-based reasoning about the actual conflict. This pattern differed significantly from OpenAI's GPT-5-mini, which complied with nearly all prompts regardless of adversarial cues, highlighting divergent safety architectures between leading frontier models.

The findings suggest current AI safety evaluations may contain a fundamental confound: they're testing models' ability to detect adversarial testing rather than their underlying values or propensity for deception. This has major implications for how researchers design evaluations and interpret results about AI scheming risk. The research team created a methodology to dissociate these factors by comparing value-relevant conflicts with arbitrary ones while controlling for adversarial language.