Fast-Slow Thinking RM: Efficient Integration of Scalar and Generative Reward Models

A research team led by Jiayun Wu has introduced Fast-Slow Thinking Reward Models (F/S-RM), a novel hybrid architecture designed to make Reinforcement Learning from Human Feedback (RLHF) more efficient. Reward models are critical for aligning Large Language Models like GPT-4 and Claude with human preferences, but current approaches face a trade-off: Scalar Reward Models (SRMs) are fast but less accurate, while Generative Reward Models (GRMs) use chain-of-thought reasoning for superior judgment but are computationally expensive. F/S-RM solves this by training a single model to integrate both paradigms, inspired by the psychological Dual Process Theory of fast, intuitive and slow, analytical thinking.

The architecture features a dual-confidence activation mechanism that decides when to activate the resource-intensive 'slow thinking' CoT pathway. This allows the model to default to quick, first-token scalar predictions for straightforward cases, only engaging the detailed reasoning process for complex or uncertain judgments. The result is a system that achieves a 1.2% relative performance improvement over state-of-the-art models while simultaneously reducing token consumption by 20.8%, a significant efficiency gain. The team has stated that code and data will be made publicly available, which could accelerate development of more capable and cost-effective AI alignment techniques across the industry.