Small Reward Models via Backward Inference

Researchers from the University of Washington and Carnegie Mellon introduced FLIP, a new method for training small reward models. Instead of judging responses directly, FLIP works backward by inferring what prompt would have generated a given response. In tests across four domains, FLIP outperformed standard 'LLM-as-a-Judge' baselines by an average of 79.6% using 13 small language models. It's particularly effective for longer outputs and resistant to reward hacking.