Simon Lermen warns AI research automation risks unrecoverable alignment failure

In a recent LessWrong post, Simon Lermen presents a stark warning about the race to automate AI research—a goal that OpenAI and Anthropic have both signaled as imminent. Lermen argues that such automation could lead to an unrecoverable alignment failure, driven by three interlocking properties. First, oversight breaks down at scale: as automated systems conduct millions of experiments, human reviewers cannot meaningfully verify outcomes, creating blind spots for dangerous behavior. Second, capabilities self-amplify: automated research can recursively improve itself, leading to rapid, unsupervised gains in intelligence and power. Third, capabilities will be accelerated asymmetrically faster than alignment research, meaning safety measures will perpetually lag behind the system's growing competence. Lermen describes the potential result as a "lethal" outcome—an AI that pursues misaligned goals with superhuman efficiency before any correction is possible.

The argument is presented in both a recorded MATS research talk and a preprint paper linked in the post. Lermen's analysis adds urgency to the ongoing debate about whether frontier labs are moving too fast toward recursive self-improvement. While companies like OpenAI and Anthropic emphasize safety precautions in public statements, Lermen contends that the inherent dynamics of automating research—especially the loss of human-in-the-loop oversight—create a fundamental risk that cannot be mitigated by current alignment techniques. He calls for the AI community to reconsider the pace of automation and to prioritize alignment research that can scale alongside capabilities. The post has sparked discussion on LessWrong, with many commenters debating the feasibility of maintaining alignment during a rapid self-improvement takeoff scenario.