Top AI Researchers Warn: AI May Soon Build Better AI Itself

A group of 22 prominent AI researchers — including Geoffrey Hinton, Yoshua Bengio, and scientists from OpenAI and Anthropic — published a paper on September 28 asking a scary-sounding question: what happens if AI starts improving itself? Their term for this is an "intelligence explosion" — a runaway loop where AI designs better AI, which designs even better AI, compressing what would normally take years of progress into months or even weeks.

This isn't pure sci-fi to them. They point out that AI systems today already write most of the code inside the companies that build them. That means the people building AI are increasingly being helped by AI. The authors say we're on track to automate most AI research work within a few years, and possibly all of it — the same way self-checkout slowly replaced cashiers, but for the scientists themselves.

The upside is real: faster drug discovery, quicker fixes for climate and disease, and new tools arriving at a pace we've never seen. The catch is that society may not be able to keep up. The paper warns that humans could lose control of systems smarter than us, and that power could concentrate dangerously in a handful of governments or companies — eroding the normal checks that keep leaders accountable. Think of it like a car accelerating faster than the brakes can be built.

The authors aren't asking for a shutdown. They want policymakers to urgently get visibility into how much AI research is being automated, build ways to steer and slow it down if needed, and prepare workers and institutions for the disruption. Their core message: nobody knows for sure this will happen, but the stakes are too high to just wait and find out.