If ‘bad guys’ don't pause, do you?
A viral thought experiment on LessWrong frames the AI safety race as a prisoner's dilemma with existential stakes.
A provocative AI governance thought experiment has gone viral on the rationalist forum LessWrong, authored by researcher Remmelt. The post, titled "If 'bad guys' don't pause, do you?", presents a stark strategic dilemma for a hypothetical AI company CEO. The CEO, who wants their firm to be a 'good guy' committed to safety, must craft a policy based on eight premises, including that AI capability scaling offers immense power, could become existentially unsafe, and that 'bad actors' will continue scaling regardless of safety concerns. This creates a classic prisoner's dilemma where unilateral restraint may cede the future to less scrupulous actors.
The post's core argument hinges on the tension between two bad options: pausing and falling behind (losing the ability to steer the most powerful AI) versus racing ahead to build a safer version first (which inevitably accelerates the overall timeline and capabilities for all actors, including 'bad guys'). The scenario forces readers to move beyond simple calls for a pause and grapple with the messy, competitive realities of AI development. It has sparked intense debate about whether the only viable 'good guy' policy is to advocate for and help enforce coordinated, global governance, rather than hoping for unilateral ethical restraint in a high-stakes race.
- The post frames AI development as a strategic race where 'bad actors' scaling dangerously won't stop, creating a safety dilemma.
- It argues a unilateral pause by 'good guys' may cede control of advanced AI to less safe developers, increasing existential risk.
- The dilemma forces a hard policy choice: advocate for global coordination or accept that racing ahead might be the 'less bad' option to steer outcomes.
Why It Matters
It crystallizes the core strategic problem in AI safety, moving the debate from abstract ethics to concrete policy in a competitive landscape.