Metric Match cuts LLM judge validation costs by 32.5%

LLM judges are increasingly used to automate evaluation of open-ended text generation, but their reliability depends on costly human annotations. Researchers from Stanford (Alyssa Unell, Natalie Dullerud, Naomi Boneh, et al.) introduce Metric Match, a method that selects a subset of samples for human annotation by matching the population reliability metric using synthetic labels. This avoids annotating irrelevant samples while preserving correlation-based metrics like Pearson or Spearman.

Empirically, Metric Match achieves a win-rate of 0.838 against random subset selection across four correlation metrics and 15 datasets, with an 18.7% reduction in average estimation error and a 32.5% decrease in annotation requirements. In a medical case study, the method saved $1,041.67 over random selection for expert annotation. The method also extends to reliability classification—determining whether an LLM judge exceeds a deployment threshold—where it again outperforms random selection. All code is publicly available with an installable package.