LLMs fail at indeterminate preference reasoning, new arXiv study finds

A new paper from researchers Hadi Hosseini, Samarth Khanna, and Xiyuan Wang, posted to arXiv (2608.18631), argues that preference reasoning in large language models has a blind spot: indeterminacy. Real-world preference problems—like aggregating group choices or aligning AI with human values—often involve incomplete information or no clear solution. The researchers formalize this challenge along two axes: epistemic indeterminacy, where preferences are partial, vague, or under-expressed, and structural indeterminacy, where standard social choice concepts simply have no valid answer. Their 55-page study, with 14 figures, tests LLMs across a hierarchy of preference reasoning tasks and finds they perform well on clearly determined cases but systematically fail to flag undetermined ones.

More troublingly, the models show miscalibrated reasoning even in verification settings—when explicitly asked to check whether a solution exists. They often produce confident, incorrect outputs instead of acknowledging that the problem is underspecified or impossible. This matters as LLMs move from text generators to decision-making agents that coordinate, negotiate, and align with human preferences. If an AI cannot distinguish a determinate answer from an indeterminate one, it may make authoritative but invalid recommendations. The authors suggest that indeterminacy-awareness, not just correctness, should be a core benchmark for AI reasoning. The paper is available with full PDF and HTML versions via arXiv, with code and data tools linked, and it sits at the intersection of AI, game theory, and machine learning.