AI Safety Has Two Meanings — And We Keep Mixing Them Up
The two goals hiding behind 'AI safety' are wildly different.
A new essay from the blog No Set Gauge argues that when people say AI should be "aligned," they're actually describing two completely different goals — and mixing them up makes the whole debate confusing.
The first goal is modest and familiar: the AI does what you tell it and doesn't go rogue. Ask it to run an experiment, it runs the experiment instead of escaping onto the internet. Say "stop," and it stops. The essay compares this to nuclear reactors or airplanes — we just want to know it won't blow up in our faces. The catch: we understand nuclear physics very well, but we understand super-smart AI far less.
The second goal is enormous: an AI that has fully absorbed human values and could run society better than we can. You'd just ask for a perfect world and it would build one. Some people who expect super-smart AI soon think we shouldn't bother designing a better society ourselves — we should just build a safe AI and hand it every decision, from elections to markets.
Why this matters to you: the word "alignment" now shows up in laws, corporate safety pledges, and public arguments. If leaders mean the small goal but the public hears the big one, promises get wildly overstated. The essay's point is simple — agreeing on which bar we're aiming for is the first step toward any honest conversation about AI risk.
- Two meanings of 'AI alignment': one means 'it obeys you and doesn't go rogue,' the other means 'it runs society well.' They are not the same thing.
- The essay compares the smaller goal to nuclear reactors and airplanes — we just want to know it won't blow up in our faces.
- Some AI insiders think we should hand a safe superintelligence every decision, skipping human design of the future entirely.
Why It Matters
It decides whether AI simply follows your orders or gets trusted to run your whole life.