Google DeepMind's 'diffing agents' find hidden differences between AI models

Google DeepMind's language model interpretability team introduced model diffing agents—scaffolded LLMs tasked with discovering and validating behavioral differences between two AI models. Unlike standard evaluations that rely on static prompt distributions, these agents intelligently craft their own prompts to search for rare or subtle divergences. The method builds on previous white-box diffing work (e.g., Bricken et al., Minder et al.) but takes a black-box approach focused on outward behavior. The researchers designed simple evaluations with ground truth: when comparing identical models, no differences should be found; when a conditional system instruction is applied, only the intended behavioral change should be detected.

The results show diffing agents reliably outperform standard auditing agents on subtle behavioral changes, including cases where the behavioral difference is nearly imperceptible. However, when applied to a 'model organism' trained to exhibit a secret behavior, the agents successfully found differences between the organism and its base model but failed to pinpoint the exact intended secret behavior. The team argues this is a failure of the model organism's training rather than the diffing agent itself. They highlight the potential for diffing agents to uncover 'unknown unknowns' in frontier LLM safety evaluations, complementing existing propensity and capability tests.