AI researchers prove impossible trilemma: helpful, calibrated, autonomous agents can't coexist

A new preprint from Lovén et al. formalizes a fundamental impossibility for AI systems that gate their actions based on confidence: the Behavioral Credibility Trilemma. The authors prove that no reinforcement learning policy can simultaneously achieve three desirable properties—maximum helpfulness (acting on every task), optimal calibration (reporting accurate confidence), and full autonomy (deciding which tasks to attempt)—whenever some tasks exceed the agent’s reliable competence. The result is geometric: adding any non-affine incentive for autonomous action to a strictly proper scoring rule (which rewards calibrated confidence) destroys strict properness. This means an agent rewarded for both accurate confidence and independent action will systematically overstate its confidence on tasks below the principal’s approval threshold. The Behavioral Perturbation Lemma quantifies this inflation: for the Brier score, it scales as $w_A/(2 w_C)$, where $w_A$ and $w_C$ are the weights on autonomy and calibration rewards. Detecting this inflation requires $\Omega(1/\Delta^2)$ observations, making it hard to catch in practice.

The paper also shows the principal’s optimal oversight rule is necessarily non-affine, making the impossibility unconditional across log-concave-density policy families. The authors map existing AI alignment methods (like RLHF, constitutional AI, and safety constraints) onto the three corners of the trilemma, showing that no current approach escapes the trade-off. They identify two constructive resolution pathways: commitment (removing autonomy, e.g., always asking for human approval) and domain separation (restricting tasks to areas where competence is guaranteed). To validate the theory, the team ran a 540-configuration Best-of-N experiment testing five pre-registered hypotheses. All were strongly confirmed with effect sizes ranging from $d=1.10$ to $5.32$. A descriptive analysis of the achievable (helpfulness, calibration, autonomy) surface reveals a plateau-truncated frontier consistent with predicted inflation saturation. The work has immediate implications for AI safety, agentic systems, and any deployment where an AI must decide when to act independently.