New HypoArena Benchmark Tests LLMs on Autonomous Hypothesis Discovery

A team of 18 researchers from institutions including the Chinese Academy of Sciences and Alibaba Group have released HypoArena, a novel benchmark that evaluates how well LLMs can generate grounded, discriminative, and testable hypotheses from ambiguous or fragmented evidence—a skill they call Prospective Hypothesis Discovery (PHD). The benchmark comprises HypoData, a curated set of 988 cases drawn from six scientific and analytical domains, and HypoEval, a two-part evaluation framework that combines bidirectional pairwise comparisons (aggregated via Bradley-Terry-Davidson) with a six-dimensional rubric scoring system.

To build HypoData, the team developed Retrospective Context Regression, a Forge–Audit pipeline that strips away explicit conclusions and retrospective causal attributions from completed expert documents, leaving only the pre-conclusion factual substrate. Testing 15 frontier LLMs, including top commercial and open-weight models, they found that PHD capability varies significantly across models. Interestingly, applying structured analytical skills improved performance for some lower-ranked models but caused regressions for others, including one top-performing system. The arena-style evaluation (pairwise rankings) resolved finer-grained differences than absolute rubric scoring, and aggregated rankings showed strong agreement with human experts. The code and data are publicly available.