Auto-Rubric as Reward replaces opaque RLHF with explicit multimodal criteria

Aligning multimodal generative models with human preferences traditionally relies on RLHF with scalar or pairwise reward signals, which collapse nuanced human judgment into opaque proxies and are vulnerable to reward hacking. To address this, researchers propose Auto-Rubric as Reward (ARR), a framework that externalizes a VLM's internalized preference knowledge into prompt-specific, interpretable rubrics before any pairwise comparison. These rubrics decompose holistic intent into independently verifiable quality dimensions, significantly reducing biases like positional bias and enabling both zero-shot and few-shot setups with minimal supervision.

To extend these gains into training, the authors introduce Rubric Policy Optimization (RPO), which distills ARR's multidimensional evaluation into a robust binary reward by conditioning preference decisions on the rubrics. This stabilizes policy gradients compared to opaque scalar regression. On text-to-image generation and image editing benchmarks, ARR-RPO outperforms existing pairwise reward models and VLM judges. The results show that the key bottleneck is not a lack of knowledge in models, but the absence of a factorized interface to externalize that knowledge into structured, inspectable criteria.