Does Reasoning Make Search More Fair? Comparing Fairness in Reasoning and Non-Reasoning Rerankers

A team from Johns Hopkins University led by Saron Samuel, Benjamin Van Durme, and Eugene Yang conducted the first systematic comparison of fairness between reasoning and non-reasoning rerankers in search systems. Their paper, "Does Reasoning Make Search More Fair?" analyzed six reranking models including reasoning-based approaches like Rank1, using the TREC 2022 Fair Ranking Track dataset across multiple retrieval settings and demographic attributes. The key finding was that reasoning capabilities neither improve nor harm fairness compared to traditional non-reasoning approaches, with fairness metrics remaining remarkably stable (AWRF 0.33-0.35) even as relevance performance varied substantially (nDCG scores ranging from 0.247 to 1.000).

Demographic breakdown analysis revealed persistent fairness gaps for geographic attributes regardless of model architecture, indicating that current reasoning implementations simply preserve the fairness characteristics of their input rankings rather than actively correcting biases. The researchers used their novel fairness metric, Attention-Weighted Rank Fairness (AWRF), to measure how equitably different demographic groups are represented in search results. These results suggest that specialized reasoning models explicitly designed to be aware of fairness attributes could lead to improvements, but current off-the-shelf reasoning rerankers don't inherently address fairness concerns despite their sophisticated capabilities.