Debiasing NLP models can backfire, ACL 2026 study finds

A study presented at ACL 2026 Findings reveals that common preprocessing methods for debiasing NLP models can backfire, creating stereotype shifts for demographic groups not originally targeted. Researchers from Dartmouth and other institutions tested encoder-only and decoder-only models using three preprocessing strategies: removing stereotypical sentences, removing group mentions, and swapping group references. Across pre- and post-training on Wikipedia at different data scales, they found that while measured stereotypes decreased for the targeted groups, stereotyping or counter-stereotyping increased for other demographics – including across completely unrelated categories.

Standard benchmarks failed to detect these side effects, which the authors attribute to the narrow focus of existing evaluation metrics. Using attention-rollout analysis, they observed that the unwanted shifts were not accompanied by large changes in attention flow, making mechanistic explanations elusive. The paper argues for side-effect-aware, transparent mitigation practices and provides actionable diagnostics for practitioners. This work challenges the assumption that preprocessing-based debiasing is safe and effective, urging the field to adopt more holistic evaluation frameworks.