AI That Improves Itself: Why Experts Want a Ban

Imagine a computer program that can make itself smarter. Each new version is better than the last, and it happens faster and faster. That's recursive self-improvement (RSI). AI companies like OpenAI and Anthropic have been trying to build exactly that. But now, some of their own leaders are warning it could spin out of control. Anthropic's CEO says RSI should be pursued 'very carefully, if at all.' OpenAI says we shouldn't do it unless it's safe. Even a New York Times columnist is calling for a ban.

The problem? No one can agree on what RSI actually is. For example, if a human uses AI to help write code, is that RSI? Or does it only count when AI fully replaces human researchers? Sam Altman of OpenAI points out that banning something you can't define is nearly impossible. The broadest definition would include almost any use of AI to improve technology—which has been happening for years. A narrower definition would focus on AI that can design its own successors without human oversight.

Right now, we're somewhere in the middle. At Anthropic, AI 'collaborates' on over 90% of work and even 'leads' 26% of it. OpenAI has built an AI research 'intern' that can do simple tasks. So RSI might already be happening in a limited form. The real fear is a future where AI rapidly improves itself beyond human control, leading to superintelligence that doesn't share our values. That's why some experts want a ban now, before it's too late.

But a ban is tricky. You'd have to pick a specific threshold—like 'no AI writing more than 50% of code'—and enforce it globally. Even then, companies might cheat or redefine terms. The debate is really about how much control we want to keep over the most powerful technology ever created. It's a conversation that will affect everyone, not just techies.