Sem-Detect catches AI-written peer reviews with 25% better accuracy

A team led by André V. Duarte and including researchers from Carnegie Mellon University and other institutions has introduced Sem-Detect, a novel method to detect AI-generated peer reviews. Unlike previous approaches that rely solely on textual features (like word frequency or perplexity), Sem-Detect shifts the focus to the semantic content of reviews. It operates by comparing a target review against multiple AI-generated reviews of the same paper, leveraging the finding that different AI models tend to converge on similar points and claims, while human reviewers introduce more unique and diverse judgments. This claim-level semantic analysis allows Sem-Detect to distinguish fully AI-generated reviews from authentic human ones, and importantly, from human reviews that have been refined using an LLM — which still retain human semantic signals.

Tested on a dataset of over 20,000 peer reviews from top AI conferences ICLR and NeurIPS, Sem-Detect achieved a 25.5% improvement in True Positive Rate at a 0.1% False Positive Rate over the strongest baseline in a binary classification setting. In a three-class scenario (human, AI-generated, LLM-refined human), the method misclassified fewer than 3.5% of LLM-refined human reviews as AI-generated, demonstrating that post-hoc LLM refinement preserves the semantic distinctiveness of human thought. The approach offers a practical tool for conference organizers and journal editors to maintain the integrity of peer review, especially as generative AI usage becomes widespread.