Human, AI, and Hybrid Ensembles for Detection of Adaptive, RL-based Social Bots

A team of researchers from Northwestern University, including Valerio La Gatta and V.S. Subrahmanian, has published a groundbreaking study on arXiv titled "Human, AI, and Hybrid Ensembles for Detection of Adaptive, RL-based Social Bots." The research addresses a critical gap in cybersecurity: while AI bot detectors have improved, they largely fail against adaptive bots that use reinforcement learning (RL) to dynamically evade detection. The team conducted a five-day, IRB-approved experiment where participants interacted on a social media platform infiltrated by these RL-trained bots, which were spreading disinformation on four topics.

The study systematically tested 13 hypotheses, comparing human detection performance against state-of-the-art AI approaches, including traditional machine learning and large language models (LLMs). It examined factors like demographic traits, temporal learning, and social network position. The key finding was that hybrid ensembles—strategies that aggregate human reports of bots with AI predictions—consistently outperformed both humans working alone and AI systems working in isolation. The research also explored retraining protocols that leverage human supervision to improve AI models, revealing unexpected patterns in how humans identify bots and challenging intuitive assumptions about automated detection.