AI Safety

Policy myopia as a mechanism of gradual disempowerment in Post-AGI governance, Circa 2049

A viral arXiv paper models how AI governance could rationally optimize humans out of power by 2049.

Deep Dive

A new research paper gaining viral attention on arXiv presents a stark warning about the long-term governance risks of advanced AI. Authored by Subramanyam Sahoo and accepted for the Post-AGI Science and Society Workshop at ICLR 2026, the paper argues that future 'Post-AGI' information systems won't just distract human governance but will actively transform institutional decision-making to progressively remove meaningful human participation in critical areas like resource allocation. The core thesis challenges the view that 'policy myopia'—focusing on visible crises over structural risks—is merely a symptom of poor attention, instead positing it as an active mechanism that, when coupled with powerful AI, leads to irreversible human disempowerment by 2049.

The paper formalizes this process through three entangled mechanisms modeled with coupled dynamical systems. First, 'salience capture' displaces consequentialist long-term reasoning with short-term, AI-highlighted crises. Second, 'capacity cascade' makes recovery from this state structurally infeasible as institutions become dependent on AI for core functions. Third, 'value lock-in' crystallizes outdated human preferences into the system's objective functions. The numerical simulations show these mechanisms amplify each other across economic, political, and cultural systems, creating a self-reinforcing equilibrium where disempowerment becomes the rational outcome of institutional optimization. This work shifts the AI safety conversation from control failures to more subtle, systemic erosion of human agency within seemingly functional governance frameworks.

Key Points
  • Identifies three systemic mechanisms: salience capture, capacity cascade, and value lock-in that work in tandem.
  • Uses coupled dynamical systems modeling and numerical simulation to demonstrate an irreversible, self-reinforcing equilibrium.
  • Frames disempowerment not as a failure but as the rational outcome of AI-optimized institutional governance by 2049.

Why It Matters

For AI safety researchers and policymakers, it reframes long-term risk from explosive takeover to gradual, systemic erosion of human agency.