RoPoLL: Robust LLM jury beats biased judges with geometric median

RoPoLL (Robust Panel of LLM Judges) tackles a critical flaw in LLM-as-judge evaluation: when you ask multiple LLMs to rate outputs and average their scores, a single biased judge can skew the entire jury. The authors formalize this under a Huber contamination model and prove that standard panel-of-LLMs (PoLL) incurs unbounded bias under any positive contamination—even with a large jury.

Their fix is elegantly simple: replace the mean aggregation with the geometric median (GM), a robust estimator known for its optimal finite-sample breakdown point of 1/2. The paper provides both finite-sample error bounds and matching minimax lower bounds, revealing a statistical-computational gap where polynomial-time RoPoLL pays a factor of √d relative to the intractable Tukey halfspace median. Empirically, across 13 open-weight judges (4B–675B) on three reward-model benchmarks under four corruption regimes (up to 50% contamination), RoPoLL dominates PoLL on every biased corruption type—by ~19% on cross-dimensional attacks and orders of magnitude on heavy-tailed Byzantine adversaries. The headline result: a 3-judge RoPoLL committee totaling 38B parameters beats Mistral-Large-3 (675B) by 1.31x on HelpSteer-2 under 30% bimodal-random corruption, an 18x parameter efficiency gain.