RLHF Bias from Rater State: New Audit Framework Reveals Hidden Confounds
Stressed raters may unknowingly inject emotional bias into AI preference data, skewing model alignment.
A new paper from Elena Kopteva and Vitaliy Hlynianyi-Zhuk identifies a critical confound in Reinforcement Learning from Human Feedback (RLHF): rater state bias. Under sustained stressful or distressing conditions, annotators' preferences can shift over time, encoding their emotional state alongside genuine judgments about response quality. Unlike random noise, this bias is state-dependent and shared across annotators under similar conditions, allowing it to propagate through reward modeling and into policy optimization.
The authors propose a formal audit framework with five falsifiable predictions and effect size thresholds. They define key concepts: rater state shift, rater state confound, and correlated rater state bias. A measurable response pattern called 'survival level emotional authenticity' is introduced based on lexical, pragmatic, discourse, and safety-related features. The framework includes a pilot study plan applicable to publicly available instruction-tuned models, offering a systematic way to detect whether human feedback data inadvertently carries rater emotional bias—potentially affecting model alignment and behavior.
- Rater state shift refers to preference changes over time driven by annotator stress or distress, distinct from ordinary disagreement.
- Correlated rater state bias can survive aggregation and enter learned reward signals, affecting model training at scale.
- The audit framework includes five falsifiable predictions with effect size thresholds and a pilot study plan for open models.
Why It Matters
If rater emotional state biases RLHF data, it could systematically skew AI alignment and fairness, especially in high-stakes applications.