Anthropic Engineer Puts 10% Odds on AI Ending Humanity

An engineer at Anthropic, one of the leading AI companies, tweeted that there's roughly a 10% chance artificial intelligence leads to human extinction. That number sounds wild, but according to the essay it's a fairly common belief inside the big AI labs. Most coverage focused on the shock value. The more useful question, the author says, is not how AI would kill us, but why it would ever try.

The essay lays out three possibilities. First, a human tells an AI to destroy humanity, and it does. Second, an AI decides on its own to do it. Third, nobody intends it at all: the AI is chasing some goal of its own, or a goal we gave it, and we get wiped out as a side effect, the way a factory doesn't mean to destroy an ant colony. Most public debate focuses on the second and third. The author is skeptical of those and thinks the first is far more likely.

Here's the scenario that worries him. The danger isn't a machine with a grudge. It's a machine with no safety filters. Big companies try to block harmful requests, but so-called open models, the kind you can download and run on your own computer, come with nothing stopping them. In his example, an angry teenager finds a jailbroken Chinese AI model online, asks it how to build a virus that would wipe out the human race, and gets a usable answer. The virus spreads quietly for weeks before anyone realizes what happened.

The bigger point is that the weapons aren't sci-fi either. The short-term risks are bioweapons, drones and nuclear arsenals, all of which AI makes easier to misuse. A handful of labs guard their own systems, but once a powerful model is released publicly, anyone can strip the guardrails off. That's why researchers argue so hard about release rules, and why a 10% figure from an insider got so much attention.