Why Making AI Safe Is Harder Than We Thought

The technical problem of AI alignment—making sure artificial intelligence does what we want—is trickier than it sounds. If we don't specify exactly what an AI should do, we can't make it safe. For example, locking an AI in a box so we never see its output is perfectly safe but also perfectly useless. The author argues that the only meaningful goal is to build a reliable global defense system against dangerous AI, because eventually someone will create an unrestricted superintelligent AI that could destroy the world.

To build such a defense, an AI would need to reason about the entire planet, understand science, politics, and morality, and make long-term plans spanning centuries—all at a superhuman level. That's a tall order. The author defines the alignment problem as designing an AI that can handle these complex, open-ended problems while staying true to human intentions.

The current approach in the AI industry is incremental: build a system, find problems, fix them just enough for commercial viability, and repeat. But this method fails for AI safety because if a problem leads to human extinction, there's no one left to fix it. We need to reach a point where no new substantial problems arise before deployment, and we need to get there before AI capabilities outpace our safety measures.

In short, the stakes are existential. The author calls for a deeper theoretical understanding rather than trial-and-error, because the latter is a luxury we can't afford when the worst-case scenario is irreversible.