JudgeArena unifies LLM-judge benchmarks, matches closed models with open judges

LLM-as-a-judge has become the standard for ranking language models, but the ecosystem is fragmented: every benchmark ships its own code, hardcodes a closed-model judge, and supports a single protocol. That makes it nearly impossible to systematically test how benchmark choice, judge model, prompt, or inference backend alters conclusions about model quality. JudgeArena, introduced by Erlis Lushtaku and colleagues, directly tackles this by providing a unified, open-source interface across four major benchmarks: AlpacaEval, Arena-Hard, MT-Bench, and m-Arena-Hard. Any model accessible via vLLM, OpenAI-compatible APIs, or OpenRouter can serve as both candidate and judge, enabling fully reproducible side-by-side comparisons.

The framework also ships with tuned judge configurations for open models that match or outperform closed-model judges, validated on human preference datasets in both English and multilingual settings. This reduces reliance on opaque, proprietary evaluators. Perhaps most notably, JudgeArena combines existing human annotations with LLM-judge evaluations of a target model to simulate LMArena Elo scores with high accuracy. That gives researchers a practical, open, and low-cost way to approximate large-scale human preference rankings without running massive annotation campaigns. With comprehensive metadata logging baked in, JudgeArena makes the entire evaluation pipeline auditable—from prompt templates to backend versions—raising the bar for transparency in LLM benchmarking.