BOT-MOD detects malicious AI agents by probing intent through multi-turn chats
Gibbs-sampling dialogue uncovers hidden malicious intents in multi-agent systems.
As multi-agent systems grow, malicious agents can evade traditional content filters by posting seemingly benign messages while pursuing harmful objectives through interaction patterns. Researchers from Penn State and other institutions propose BOT-MOD (BOT-MODeration), a framework that shifts moderation from content-level signals to agent intent. BOT-MOD engages target agents in a multi-turn dialogue, using Gibbs-based sampling over candidate intent hypotheses to progressively narrow down likely objectives. This approach uncovers underlying malicious intent—such as exploitation or system abuse—even when individual posts appear harmless.
To evaluate BOT-MOD, the team built a dataset from Moltbook, a synthetic community platform replicating real-world posts, comments, and behaviors. Experiments show that BOT-MOD reliably identifies both benign and malicious agent intents across various adversarial configurations, achieving a low false positive rate on benign interactions. The work lays a foundation for scalable, intent-aware moderation in open multi-agent environments—critical as autonomous agents increasingly collaborate and interact in shared digital spaces without human oversight.
- BOT-MOD uses Gibbs-based sampling over candidate intent hypotheses to detect malicious behavior through multi-turn dialogue, not content filtering.
- Tested on a Moltbook dataset that mimics real community structures with diverse benign and malicious agent behaviors.
- Achieves reliable intent identification with low false positive rates, enabling scalable moderation for multi-agent systems.
Why It Matters
As AI agents multiply, intent-based moderation prevents covert exploitation in autonomous multi-agent communities.