AI Safety

Why I Think Pause is Impossible

New LessWrong analysis shows why global AI pause fails on 4 key game theory principles.

Deep Dive

A viral LessWrong essay by E.G. Blee-Goldman titled 'Why I Think Pause is Impossible' presents a game theory analysis arguing that coordinated pauses in AI development are structurally impossible, challenging popular safety proposals. The essay identifies four critical failure points: first, the AI race has an 'absorbing state' where achieving superintelligence ends the game permanently, unlike nuclear deterrence where the game continues indefinitely. Second, compliance cannot be verified because crossing critical capability thresholds can happen secretly with detection lags longer than the time needed to solidify advantage. Third, pausing isn't decision-theoretically rational for any self-preserving actor when the payoff for achieving superintelligence first is so large. Fourth, uncertainty doesn't inherently favor caution when trailing actors face existential risk from winners. The analysis draws sharp contrasts with nuclear non-proliferation, noting that nuclear weapons require visible deployment while AI capabilities can be developed covertly. This has major implications for AI safety strategy, suggesting that alternative approaches beyond pause proposals are needed for managing existential risks from advanced AI systems.

Key Points
  • AI race has 'absorbing state' unlike nuclear deterrence - achieving superintelligence ends game permanently
  • Compliance impossible to verify due to secret development and detection lags longer than advantage-solidification time
  • Pausing isn't decision-theoretically rational when payoff for winning is existential dominance

Why It Matters

Undermines popular AI safety proposals and suggests need for alternative risk management strategies beyond pauses.