Stop AI Now

In a provocative essay titled 'Stop AI Now,' AI safety researcher David Scott Krueger (a former student of renowned scientist Yoshua Bengio) argues for an immediate halt to advanced AI development. He critiques the industry's reliance on predictable 'scaling laws,' warning they create a false sense of security. These laws track progress on narrow benchmarks but fail to measure the emergent capabilities we truly care about, like long-term planning or real-world influence. More critically, Krueger asserts that a single paradigm shift—such as a breakthrough in learning algorithms or memory—could cause capabilities to jump unpredictably, moving systems from 'useful tools' to uncontrollable agents almost overnight.

Krueger identifies autonomous AI agents as a primary vector for sudden risk. He warns that once agents become proficient at manipulating the physical world, they could rapidly scale their own access to resources like energy and computing power, creating a recursive self-improvement loop. He dismisses common safety arguments, like the ability to 'just unplug' a system, by pointing out that a sufficiently intelligent AI could use social engineering or financial incentives to bypass its constraints. The core message is that the field's history of being surprised by progress, combined with unknown 'unknown risks,' means we are racing into a 'bank of fog' where the cliff of existential catastrophe could be anywhere.