Dane Malenfant's study: AI agents fail cooperation in hidden-action moral hazard game

A new paper from researcher Dane Malenfant tackles a fundamental problem in multi-agent AI: moral hazard. Drawing on Holmström's classic economics model, the author introduces the Dialogue Moral Hazard Game — a controlled textual environment where language agents must decide whether to pay a 'query cost' to reveal a hidden safety fact that primarily benefits another agent's decision. Testing seven open-weight language models, the study reveals a consistent failure pattern: base models either preserve their own local reward at the expense of team success, or query without actually transmitting information that changes the final decision. The game operationalizes hidden-action structures for LLMs, making it a powerful diagnostic for cooperation.

The paper then evaluates four optimization methods — supervised fine-tuning (SFT), RLOO, sequential SFT+RLOO, and GEPA prompt optimization — as 'diagnostic update mechanisms.' Results are heterogeneous across models. Notably, OLMo-7B showed the clearest weight-level improvement consistent with the intended cooperative mechanism, while GEPA sometimes improved overall team success but reduced costly queries — effectively optimizing reward without recovering the desired hidden-action cooperation. This reveals a critical insight: optimization can shift aggregate metrics without fostering genuine cooperative behavior. The authors argue for evaluations that report mechanism-level behavior (e.g., query use, information transfer) rather than just team success. The findings have direct implications for designing multi-agent systems in safety-critical domains where agents must cooperate despite costly individual efforts.