RecSys 2026 study: Position bias undermines LLM reranking consistency

A new paper from Ethan Bito, Yongli Ren, and Estrid He, accepted at RecSys 2026, exposes a critical flaw in how large language models are used for listwise reranking in recommender systems. Because recommendation candidates are an unordered set, an ideal reranker should produce identical outcomes regardless of the order in which candidates are serialized. But decoder-only LLM rerankers — the kind used in production pipelines — let input order influence model scores, pairwise preferences, and final rankings. The researchers introduce an evaluation framework that treats rankings generated under equivalent candidate permutations as observations of an induced preference system, measuring pairwise preference instability, global preference inconsistency, and listwise output consistency across multiple LLMs, datasets, and list lengths.

Their findings are sobering: these three consistency measures align closely with each other but often diverge from recommendation effectiveness and marginal position-exposure bias. Simply improving relevance or flattening exposure across positions does not restore stable pairwise preferences, globally coherent preference structures, or consistent ranked outputs. This means that reducing marginal exposure skew — a common mitigation tactic — is insufficient to establish ranking-function validity in LLM-based reranking. The study challenges the reliability of LLMs as drop-in rerankers for production recommendation systems and suggests that order-robustness must be explicitly designed and validated, not assumed. Code is available via the paper's arXiv entry, inviting further research into bias-resistant reranking methods.