Account history beats content for bot detection against GPT-4 text

The rise of large language models like GPT-4 and Claude has made content-based bot detection increasingly unreliable. These models can generate text with surface statistics nearly indistinguishable from human writing, weakening traditional classifiers that rely on post content. In response, Gaurang Katyal's paper investigates how much of the detection problem can be solved using account-history features that attackers cannot easily manipulate: account age, follower/friend counts and ratios, profile completeness, and handle structure. Using a public corpus of 2,432 Twitter accounts (43% bots), a random forest classifier trained only on these behavioral features achieved a ROC-AUC of 0.977, compared to just 0.830 for a content-only baseline—a statistically significant gap (DeLong's test z=9.36, p<0.001).

The study also tested two adversarial settings. When bot tweets were rewritten to match human statistics for URLs, hashtags, mentions, and casing, the content classifier's AUC dropped from 0.842 to 0.785, while the behavioral classifier remained essentially unchanged. In a more aggressive attack that directly perturbed content features toward the human distribution, the content classifier fell below chance (AUC 0.466), yet behavioral performance stayed invariant. These results, replicated on a sample from TwiBot-20, demonstrate that account-history features are highly robust to LLM-generated text. The paper concludes that operational bot detection should not rely primarily on content signals; behavioral metadata carries most of the detection power and is not eroded by adversarial text rewriting, offering a practical path forward for platforms combatting AI-powered bots.