RL-powered insider attacks disrupt multi-agent LLM consensus

A new paper from Xiaolin Sun and colleagues explores a critical security gap in multi-agent LLM systems—insider attacks on consensus formation. These systems rely on agents exchanging natural-language messages to reach shared decisions, but existing frameworks assume all participants are aligned with the system goal. The researchers formalize a threat where a malicious insider, while appearing legitimate, strategically delays or prevents agreement among benign agents. To make such attacks practical, they propose a world-model-based approach that learns surrogate dynamics over the latent behavioral states of benign agents, then trains the attacker using reinforcement learning. This model allows the attacker to adapt its messaging strategy without direct access to the benign agents' internal states.

The preliminary results are stark: the trained attacker significantly reduces the benign consensus rate and prolongs disagreement compared to a baseline that simply sends adversarial prompts. This indicates that latent world models combined with RL offer a powerful, adaptive method for insider manipulation in language-based multi-agent systems. While the research is preliminary, it underscores a fundamental vulnerability: as multi-agent LLM systems are deployed in high-stakes domains like finance, logistics, or negotiations, a single compromised agent could systematically undermine group outcomes. The work highlights the urgent need for robust consensus protocols that can detect or withstand such covert attacks.