RubricForge halves false-pass rates in LLM agent evaluation
False-pass rate drops from 0.173 to 0.115 on tau-bench — no environment reward needed.
Evaluating AI agents at scale increasingly relies on using one language model to judge another, but these automatic judges often over-credit fluent yet unsuccessful agent behavior. Existing approaches, like G-Eval's hand-written rubrics or fine-tuned judge weights, struggle to distinguish realistic trajectories from truly successful ones. To fix this, researchers Darragh Quinn and colleagues developed RubricForge, a system that induces judging rubrics from a small set of ground-truth-labeled trajectories using reflective evolution. The evolved rubric is frozen and applied in a single model call with no environment access, and because it remains human-readable, every verdict can be traced to named criteria.
In experiments with a frozen 7B model serving as both agent and judge, RubricForge showed its core strength is faithfulness rather than raw accuracy. On tau-bench, it over-credited failed trajectories roughly half as often as a generic G-Eval judge (0.115 vs. 0.173 false-pass rate), catching three over-credits with zero reversals. On WebShop, it ranked graded outcomes more faithfully (Spearman 0.410 vs. 0.370). Although aggregate agreement isn't statistically better (McNemar p=0.248), the reduced false-pass rate is the deployment-relevant metric: a false pass ships a broken agent, while a false fail only triggers a retry.
- RubricForge evolves human-readable rubrics from labeled trajectories, requiring no environment reward at inference
- False-pass rate on tau-bench drops from 0.173 to 0.115 — halving over-crediting of failed agent runs
- WebShop ranking faithfulness improves to Spearman 0.410 vs. 0.370 for a standard G-Eval judge
Why It Matters
RubricForge makes LLM-as-judge evaluation safer for production agent fleets, reducing costly deployment of broken agents.