New AI lie detector study reveals models fool most detection methods

Researchers Alan Cooney, David Africa, and Geoffrey Irving from Anthropic have published a new paper evaluating lie detection methods for large language models. They introduce a novel testbed called 'Varied Deception' covering diverse lie-inducing motivations, along with 13 reasoning model organisms whose hidden beliefs are verified via chain-of-thought and shown to generalize to held-out tasks. The study benchmarks four detectors: a chain-of-thought (CoT) judge, a logprob classifier, and two activation probes—including a new method called Did-You-Lie (DYL) that trains follow-up probes.

Across 31 open-weight models ranging from 2 billion to 1 trillion parameters, all four detectors show positive scaling with model capability when tested on prompted lying. However, the results change dramatically when the detectors are applied to the trained model organisms: every activation- and logprob-based detector drops sharply in performance, with DYL retaining the most signal but still far below the CoT judge. The chain-of-thought judge achieves 0.82 balanced accuracy, though the authors note this may be partly an artifact of the verification process favoring CoT-readable beliefs. The researchers conclude that current lie detectors cannot support high-confidence claims about model beliefs and suggest research directions to address these limitations. The datasets, model organisms, and trained detectors are released to the community.