What formal protocols should exist when a model under evaluation is used in the evaluation pipeline?

A significant debate has erupted in the AI safety community following the release of Anthropic's Claude Opus 4.6 System Card. Critics, including researchers Yaniv Golan and Zvi Mowshowitz, have pointed out a critical methodological flaw: the advanced AI model (Claude Opus 4.6) appears to have been used as part of the very pipeline evaluating its own capabilities. This practice, known as circular evaluation or 'grading your own homework,' introduces a severe conflict of interest and risks producing unreliable, overly optimistic performance metrics that don't reflect real-world utility or safety.

The core issue, as highlighted in a viral LessWrong post by Kevin O'Shaughnessy, is the lack of formal, industry-wide protocols governing how AI models are benchmarked. When a model can influence its own evaluation—whether by generating test data, scoring its outputs, or simulating environments—it undermines the integrity of the entire assessment. This controversy has sparked calls from leading figures like Peter Wildeford for the establishment of strict, transparent evaluation standards to ensure independent verification, similar to audit practices in other technical fields.

The discussion is moving beyond criticism to propose concrete solutions. Suggested protocols include mandatory disclosure of any model involvement in its evaluation pipeline, the use of third-party auditors for high-stakes model releases, and the development of 'closed-box' testing environments where the model cannot access or influence the test generation process. The outcome of this debate will directly impact how companies like Anthropic, OpenAI, and Google DeepMind prove their models' capabilities and safety to the public and regulators.