Superintelligence race: AI researchers 'don't understand the state', warns LessWrong post
Governments won't let AI companies perform pivotal acts, challenging AI safety assumptions
Alex Amadori argues that even top AI researchers misunderstand the role of government in the race to superintelligence (ASI). They typically frame the future as a competitive game between AI companies, with governments as passive background elements. This leads researchers to join companies they consider the 'good guys,' hoping to influence development from within or to ensure that the winning company performs a 'pivotal act'—a decisive move that prevents all other actors from building dangerous AI.
But Amadori points out a critical flaw: any such pivotal act necessarily requires gaining control over all governments on Earth. Why would sovereign states, each with their own interests and military power, allow a single company to dictate global AI policy? The assumption that a company can unilaterally enforce safety standards ignores geopolitical reality. The post, linking to a longer X thread, suggests that AI safety strategies must account for state actors who will not simply cede control. This challenges the entire framework of 'alignment at the company level' and forces a reconsideration of how power over AGI might actually be distributed.
- AI researchers model the future as competition between companies, ignoring governments as independent actors.
- Performing a 'pivotal act' to prevent irresponsible AI requires controlling all governments, which is unrealistic.
- Strategies like joining 'good' AI companies assume they can steer the endgame, but state power complicates this.
Why It Matters
For AI professionals, this highlights that safety strategy must include government dynamics, not just company-level alignment.