Lessons from External Review of DeepMind's Scheming Inability Safety Case

A new paper published on arXiv applies the Assurance 2.0 framework to perform an external review of Google DeepMind's public 'scheming inability' safety case. The review uncovered substantive new concerns that materially affect the scope of the safety case and its applicability for decision-making. The authors, including Stephen Barrett and colleagues, argue that when developers author their own safety cases, confirmation bias and conflicted incentives can compromise argument quality. This external audit surfaced weaknesses not previously addressed by DeepMind, particularly around the case's assumptions and evidence boundaries.

Based on this experience, the paper provides concrete recommendations for how external review should be conducted and what information AI developers should provide to support it. The findings underscore the need for rigorous, independent evaluation of frontier AI safety claims, especially as models grow more capable. The authors call for greater transparency and standardized review protocols to ensure safety cases are robust enough for high-stakes deployment decisions.