Georgia Tech study: Frontier LLM debate evaluators miss the mark

Researchers Anthony Miyaguchi and Conor Johnston, from Georgia Tech's DS@GT ARC group, submitted a working-note paper to the Touché 2025 Retrieval-Augmented Debate task. The competition challenges AI systems to both generate the next utterance in a simulated debate and evaluate debate responses using Gricean maxims: Quantity, Quality, Relation, and Manner. Their submission used a retrieval-augmented prompting pipeline that leveraged six leading LLMs from three providers, combining retrieval of relevant evidence with prompts to generate coherent and persuasive arguments.

The paper's analysis reveals two key findings. First, frontier LLM systems are strong response generators in debate settings, producing utterances that hold up well competitively. Second, when the same models are used as evaluators, they agree strongly within model families, but this consensus does not reliably track the official evaluation target—with the largest discrepancy occurring on the Quality maxim. This means LLM-based judges may consistently agree with each other while still missing what the official evaluation deems high-quality, pointing to a fundamental limitation of using LLM ensembles for debate scoring. The paper, accepted for CLEF 2026 Best of Labs proceedings, includes open-sourced code and provides a cautionary data point for anyone building multi-LLM evaluator systems.