The Controllability Trap: A Governance Framework for Military AI Agents
New paper warns of six distinct control failures in agentic AI and proposes a measurable governance architecture.
A new research paper titled 'The Controllability Trap: A Governance Framework for Military AI Agents' warns that current AI safety frameworks are inadequate for governing advanced agentic systems. Authored by Subramanyam Sahoo and accepted at the ICLR 2026 Workshop on Agents in the Wild, the paper argues that AI agents capable of goal interpretation, long-horizon planning, and autonomous coordination introduce six distinct types of control failures that erode meaningful human control in military contexts. These failures are not addressed by existing paradigms, creating a critical governance gap as nations and defense contractors rapidly develop such systems.
The proposed solution is the Agentic Military AI Governance Framework (AMAGF), a measurable architecture built on three pillars: Preventive Governance to reduce failure likelihood, Detective Governance for real-time monitoring, and Corrective Governance to restore safe operations. Its core innovation is the Control Quality Score (CQS), a composite, real-time metric that quantifies the degree of human control over an AI agent. This allows for graduated responses—like scaling back autonomy—as control weakens, rather than a simple on/off switch. The paper defines concrete mechanisms and assigns responsibilities across five institutional actors, providing a formal structure for evaluation. This work represents a significant shift, advocating that governance must actively measure and manage control quality throughout an AI system's operational lifecycle, not just at deployment.
- Identifies six distinct agentic governance failures not covered by current AI safety frameworks, tied to capabilities like autonomous planning and coordination.
- Proposes the Agentic Military AI Governance Framework (AMAGF) with a core Control Quality Score (CQS) for real-time, measurable human oversight.
- Shifts governance from a binary control model to a continuous one, enabling graduated responses as AI autonomy increases and human control degrades.
Why It Matters
Provides a concrete, measurable framework for governing autonomous military AI, a critical step as agentic capabilities outpace existing safety protocols.