Statistical Impossibility and Possibility of Aligning LLMs with Human Preferences: From Condorcet Paradox to Nash Equilibrium

A team of researchers (Kaizhao Liu, Qi Long, Zhekun Shi, Weijie J. Su, Jiancong Xiao) have published a paper in the Annals of Statistics that rigorously examines the fundamental limits of aligning LLMs with diverse human preferences. They focus on two central questions: when can preferences be represented by a single reward model, and when can aligned models avoid collapsing to a single response?

The first key result is a statistical impossibility theorem: reward-based approaches like reinforcement learning from human feedback (RLHF) can only fully align preferences if there are no Condorcet cycles—situations where majority preferences are intransitive (A beats B, B beats C, C beats A). The authors prove that under a standard probabilistic model of human preferences (the Luce model), such cycles occur with probability approaching 1 exponentially fast as the number of responses grows. This means that for most real-world preference distributions, no single reward function can capture what humans truly want, and RLHF will inevitably fail to preserve all preferences.

On the positive side, the paper identifies a possibility result for non-reward-based alignment, specifically Nash learning from human feedback (NLHF). They prove that LLMs will naturally adopt mixed strategies (i.e., not collapse to a single deterministic response) if and only if there is no response that is majority-preferred over all others. Remarkably, this condition holds with high probability under the same Luce model, meaning NLHF can statistically preserve minority preferences without needing explicit diversity regularizers. This offers a principled alternative to RLHF that may produce fairer and more representative AI systems.