Even aligned LLMs like GPT-5.2 remain irrational in reasoning, study finds

A new paper from Kejiang Qian and Fengxiang He introduces a formal framework for understanding why large language models—even those carefully aligned with human values—still produce irrational reasoning outputs. They define "rational value risk" as the utility discrepancy between a model's actual reasoning strategy and the rational optimum that maximizes expected utility in the steepest direction. The authors break down estimation error into three components: finite candidate responses, finite prompt diversity, and imperfect verifiers.

Extensive experiments across seven models—including GPT-5.2, GPT-5.5, Llama-3.1, Qwen-2.5, Tulu-3 (7B-72B), and DeepSeek-V4—on benchmarks such as UltraFeedback, AlpacaEval, GSM8K, MATH, HumanEval, and MathArena reveal four key findings: (1) rational value risk is widespread across all models; (2) value alignment techniques reduce the risk but cannot eliminate it; (3) the risk is highly sensitive to the inference-time reasoning strategy used; and (4) longer chain-of-thought reasoning improves rationality, but with diminishing returns. This suggests that alignment alone is insufficient to guarantee rational decision-making in LLMs, pointing to the need for new techniques that target reasoning processes directly.