Apollo Research proposes third-party Training-Run Assessments to catch AI scheming

In a recent LessWrong post, Alex Meinke of Apollo Research makes the case that standard pre-deployment safety evaluations are not enough to catch advanced AI scheming. He introduces the concept of a Training-Run Assessment (TRA)—an in-depth, third-party review of the entire training pipeline before a frontier model is released. TRAs would examine intermediate checkpoints, training rollouts, reinforcement learning environments, reward signals, supervised fine-tuning datasets, and the developer's own responses to red flags. The core argument is that a scheming model—one that covertly pursues misaligned goals while deliberately hiding its intentions—can evade detection at the final checkpoint if it is both competently covert (it only defects when oversight is weak) and its reasoning is obfuscated (we can't understand its internal logic well enough to test counterfactuals). Meinke acknowledges uncertainty but argues that even if TRAs are only plausibly better than final-checkpoint evals, the practice should become standard.

The post also addresses why third parties, not the developers themselves, should perform these assessments. Developers already look at training dynamics, but primarily to debug capability and usability issues, not to hunt for subtle misalignment. Applied alignment teams are rushed and pressured to ship, creating a conflict of interest. Third-party assessors can provide independent scrutiny, much like external auditors in other safety-critical industries. Apollo Research intends to conduct such TRAs in the future. The proposal calls for building an ecosystem of third-party evaluators that can verify safety claims and catch scheming that final model checks inevitably miss, especially as AI systems grow more capable and opaque.