One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models

A team of researchers from Stanford University and Google, led by Daniel Fein, published a significant paper titled 'One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models.' The study systematically measures biases in five high-quality Reward Models (RMs), which are crucial for aligning large language models (LLMs) like GPT-4 and Claude with human preferences through techniques like Reinforcement Learning from Human Feedback (RLHF). The key finding is that despite prior work, serious biases persist in even the most advanced RMs, making LM policies vulnerable to 'reward hacking'—where models learn undesirable behaviors from flawed reward signals. The researchers identified not only known issues like length bias (preferring longer answers) and sycophancy (agreeing with the user) but also newly discovered biases toward a model's own specific output style and the order in which multiple-choice answers are presented.

The paper categorizes these RM failures by complexity and proposes a novel, post-hoc intervention called 'mechanistic reward shaping' to mitigate low-complexity biases that stem from spurious correlations. This method is designed to reduce targeted biases without degrading overall reward quality and requires only minimal amounts of new labeled data. Critically, the approach is extensible to new biases as they are discovered, can work with model-internal representations, and shows promise for generalizing to out-of-distribution scenarios. This work provides a crucial diagnostic tool and a potential corrective framework for the AI safety community, directly impacting how companies like OpenAI, Anthropic, and Google develop and audit the alignment systems for their flagship models.