BayesBench reveals LLMs struggle to rationally update beliefs across conversations

Large language models (LLMs) are increasingly used in multi-turn conversations where each exchange provides new evidence. Yet most evaluations only check the final answer, ignoring how beliefs evolve. A new paper introduces BayesBench, a suite of simulation environments probing whether LLMs update their beliefs like a rational Bayesian reasoner. The benchmark includes three progressively complex tasks: (i) Bayesian estimation — inferring an unknown parameter from sequential evidence; (ii) Bayesian prediction — using those inferred beliefs to forecast outcomes; and (iii) latent-framed Bayesian prediction — where observations are filtered through a user persona, requiring joint inference over both the latent state and the persona.

Testing across seven LLMs from 3B to 70B parameters, the researchers found that scaling model size improves the ability to infer latent variables and accumulate evidence — occasionally matching Bayesian posterior updates. However, these gains did not reliably carry over to downstream prediction tasks. For example, a larger model might correctly identify a hidden parameter but then fail to use it rationally to forecast the next observation. This exposes a critical gap: LLMs can learn to compress evidence into latent representations, but they struggle to use those representations for coherent belief updating about the target outcome. The work highlights a fundamental limitation in current LLMs' reasoning over time — one that simple final-answer benchmarks miss entirely.