Mitigating LLM biases toward spurious social contexts using direct preference optimization

A new research paper from Stanford University tackles a critical flaw in large language models (LLMs): their susceptibility to bias from irrelevant social context. Researchers Hyunji Nam and Dorottya Demszky found that when evaluating tasks like teacher instructional quality, adding spurious details—such as a teacher's experience level, education, or demographic identity—could shift an LLM's prediction by up to 1.48 points on a 7-point scale. Surprisingly, larger models sometimes showed greater sensitivity to this irrelevant information, despite having higher baseline accuracy. Standard mitigation techniques, including prompt engineering and conventional Direct Preference Optimization (DPO), proved largely ineffective.

To solve this, the team developed **Debiasing-DPO**, a novel, self-supervised training method. The technique works by having the model generate two reasoning paths: one using only the core query (neutral) and another using the query plus the spurious context (biased). The model is then trained to prefer the neutral reasoning. This objective is combined with supervised fine-tuning on ground-truth data to prevent a loss in predictive performance. When applied to models like Llama 3B & 8B and Qwen 3B & 7B Instruct, Debiasing-DPO dramatically reduced bias by 84% and, crucially, improved the models' predictive accuracy by an average of 52%. The findings demonstrate that robustness to misleading context is not an automatic result of model scaling and requires targeted intervention.