RankJudge benchmarks LLM judges on multi-turn conversations with flaw injection

LLM-as-a-judge auto-evaluation has become essential for assessing conversational AI, but existing benchmarks only cover simple Q&A tasks, missing the complexity of multi-turn dialogues. To address this, researchers from academia and industry introduce RankJudge, a synthetic benchmark generator that creates paired conversations grounded in reference documents. Each pair differs by a single flaw injected into one turn, allowing unambiguous labeling of which conversation is better. This design isolates failure categories to specific turns and enables a strict joint correctness criterion. RankJudge is implemented across three domains—machine learning, biomedicine, and finance—and evaluated on 21 frontier LLM judges.

The researchers rank the judges using the Bradley-Terry model, a statistical approach for paired comparisons. They also assign difficulty ratings to each conversation pair, which allows dynamic curation of the evaluation slice to reduce label noise—a result confirmed by human annotation. Critically, judge rankings remain stable under partial observability, coarser correctness criteria, and an alternative random-walk rating algorithm. These findings suggest RankJudge provides a reliable and scalable method for assessing LLM judgment in realistic multi-turn settings, addressing a key bottleneck in conversational AI development.