"What Happens Next?" AI Safety Essay Probes Post-Win Scenarios
Sean Herrington argues AI community overlooks stable endpoints. A chess analogy reveals the flaw.
In a LessWrong essay dated June 23, 2026, Sean Herrington challenges the AI community to rigorously examine what comes after a successful AI development outcome. He uses the game "The Choice Before Us" by Nick Shapiro, where players aim to achieve five wonders (like curing cancer) without unleashing uncontrolled superintelligence. Herrington notes that winning the game leaves players perilously close to creating AGI, yet the game offers no guidance on what to do next—shutting down the GPUs and going home is not a realistic plan.
To illustrate his point, Herrington introduces a chess analogy: evaluating a complex position requires a "quiescence search" that resolves immediate tensions before assessing material advantage. Similarly, AI strategy must project to stable, quiescent futures rather than stopping halfway through a dynamic transition (e.g., an AI takeover of the US government). He warns that declaring victory mid-singularity is like losing your queen in chess. The essay calls for a more disciplined focus on evaluating only futures where things are relatively stationary, ensuring that AI safety plans account for true long-term stability. This perspective aims to counter the common fallacy of assuming that a 90% survival rate means control is retained.
- Herrington critiques the game "The Choice Before Us" for ignoring post-victory scenarios (achieving 5 wonders).
- Uses chess quiescence search to argue AI evaluations must consider stable, stationary futures, not dynamic ones.
- Warns that assuming control after a partial AI win (e.g., 90% survival) is a dangerous oversight—the game may not be over.
Why It Matters
For AI safety professionals: evaluating stable endpoints prevents catastrophic miscalculations in strategy and deployment.