AI Experts Clash Over Whether to Halt Risky AI Development

A recent online debate has highlighted a split among people who want to ensure artificial intelligence doesn't cause disaster. At the center is a strategy called SURA—Stop Unilaterally and Raise Awareness. Supporters say that if you're building something that could kill everyone, you should stop and urge others to stop too, even if they don't. Opponents, like writer Scott Alexander, argue that this approach is unlikely to work and could backfire.

The debate got heated when Scott Alexander published a post arguing against SURA. He believes that advocating for a global pause (Plan A) is fine, but focusing only on unilateral stopping is a mistake. He thinks that without a plan for what happens if others keep building, stopping alone won't prevent catastrophe. Critics of his post, like Oliver Habryka, argue that Scott underestimates the risks and is too confident that a technical solution will be found in time.

The disagreement also touches on how to make AI safe. Scott believes that AI models can learn virtue by imitating humans, but others strongly disagree, saying that's misleading and oversimplifies the challenge. The debate reveals deep uncertainty about the best path forward.

For the general public, this debate matters because the decisions made by AI researchers and policymakers could determine whether AI is developed safely or recklessly. If the SURA approach gains traction, it might lead to moratoriums or protests that disrupt AI progress. If not, the race to build ever-more-powerful AI continues, with all its promises and perils. Either way, the outcome will affect jobs, privacy, and safety.