AAMAS study: Audio XAI erodes trust in AI agents during Overcooked tasks

Explainable AI (XAI) is often touted as a way to improve human-agent collaboration, but most studies rely on hand-crafted policies in custom settings. Mateus Levi Simões Fernandes and Alberto Sardinha at AAMAS 2026 flipped that script by evaluating XAI support generated from an intrinsically explainable learned policy inside a well-known benchmark. Using the Hierarchical Ad Hoc Agents (HA²) architecture in Overcooked-AI, they created real-time explanations from hierarchical subtask selections, delivered through either text or audio via a novel trigger-based system. This is the first systematic evaluation of its kind in an established benchmark—covering 38 participants in a between-subjects experiment.

Results were nuanced: explanations didn't significantly improve performance, though participants who received them showed trends toward faster improvement over time. The big surprise came from modality—audio explanations caused a significant drop in participants' working-alliance bond with the agent, while text explanations had no such effect. The authors interpret this as spoken explanations activating partnership expectations that the agent's reactive, non-anticipatory policy can't actually meet. They call for matching explanation modality to the underlying policy's ability to sustain the partnership it implies. The paper also delivers the first modality comparison in real-time human-agent collaboration and a baseline methodology for testing intrinsically explainable RL architectures in benchmark environments like Overcooked-AI. For researchers, this is a clear warning: the delivery channel of XAI is not neutral, and designing for the wrong modality can backfire.