AI Researchers Warn Superintelligence Could End Humanity

Artificial superintelligence (ASI) — AI far smarter than any human — is no longer science fiction. Major tech companies are openly racing to build it. But many experts, including Eliezer Yudkowsky and Stuart Russell, warn that ASI could pose an existential risk to humanity if we don't know how to control it. The challenge is enormous: we need a mathematical understanding of how to build safe, reliable intelligent agents, and the cost of failure is catastrophic.

For over a decade, researcher Vanessa Kosoy has been working on this problem. In 2018, she created the Learning-Theoretic AI Alignment (LTA) agenda, combining ideas from AI safety research with mathematical learning theory. Since then, she and her collaborators have made steady progress. In 2023, she proposed a safety protocol called Computational Superimitation. Now, she's founded a dedicated lab, CORAL, to push this work forward.

This year, Kosoy's team achieved a breakthrough in compositional learning theory, a key missing piece. This has allowed them to outline a new mathematical framework called computational agent theory. While still theoretical, it's a significant step toward understanding how to build AI that reliably does what we want. The work is complex and not yet a practical solution, but it's a promising direction for making future ASI safe.

For everyday people, this research matters because AI is already changing jobs, economies, and daily life. If superintelligence arrives without safety guarantees, the consequences could be dire. Supporting and understanding this kind of foundational research helps ensure that the AI we build in the coming decades benefits humanity rather than harms it.