Anthropic warns of AI self-improvement risks, urges industry pause

Anthropic, the AI company behind models like Claude, published a blog post warning of 'massive societal risks' from recursive self-improvement—where AI systems autonomously enhance their own capabilities. The post urged other AI developers to consider slowing down or pausing certain development pathways to prevent a potential 'runaway to superintelligence.' This echoes the 2023 open letter from the Future of Life Institute (FLI), which called for a six-month pause on training AI systems more powerful than GPT-4, signed by figures like Elon Musk and Yoshua Bengio. That earlier call was not heeded.

FLI President and CEO Anthony Aguirre released a statement supporting Anthropic's position, saying, 'We are approaching a runaway to superintelligence that could threaten our shared human future. Both publicly and privately, AI companies are recognizing that a pause or slowdown in certain developmental pathways is crucial to protect lives and livelihoods everywhere.' Aguirre emphasized that this recognition should give hope and that FLI stands ready to work with any company willing to act. The warning comes amid growing concerns about uncontrolled AI development, including the risk of information pollution, job displacement, and loss of human control over civilization.