LitXBench: A Benchmark for Extracting Experiments from Scientific Literature

Researchers Curtis Chong and Jorge Colindres have published LitXBench, a new framework designed to benchmark AI systems that extract complete experimental data from scientific literature. This addresses a growing need in fields like materials science, where aggregating data from papers is crucial for building better property prediction models and accelerating discovery. The team also released LitXAlloy, a dense benchmark comprising 1,426 total measurements extracted from 19 published alloy papers. A key innovation is storing benchmark entries as Python objects rather than traditional text formats like CSV or JSON, which significantly improves auditability and enables programmatic data validation.

In their evaluation, the researchers tested frontier language models against existing multi-turn extraction pipelines. They found that Google's Gemini 3.1 Pro Preview significantly outperformed these specialized pipelines, achieving up to a 0.37 higher F1 score. The analysis suggests this performance gap exists because current extraction pipelines incorrectly associate measurements primarily with material compositions, rather than with the specific processing steps that actually define a material's properties. This finding highlights a fundamental flaw in how many automated systems interpret scientific experiments, pointing toward more context-aware AI approaches for literature mining.