SciTaRC: Benchmarking QA on Scientific Tabular Data that Requires Language Reasoning and Complex Computation

A research team led by Hexuan Wang and Philipp Koehn from Johns Hopkins University and the University of Edinburgh has introduced SciTaRC, a new benchmark designed to test AI models on their ability to answer complex questions about tabular data from scientific papers. Unlike simpler table QA tasks, SciTaRC requires both deep language understanding and multi-step computation, such as calculating growth rates from time-series data or deriving statistical relationships. The benchmark contains 1,200 expert-authored questions across physics, biology, and environmental science, presenting a significant challenge even for advanced models.

The researchers tested current state-of-the-art AI systems and found they fail on at least 23% of SciTaRC questions, with the gap widening dramatically for more capable models. Surprisingly, the open-weight Llama-3.3-70B-Instruct model failed on 65.5% of tasks, highlighting a critical weakness in current AI capabilities. The analysis reveals what they term an 'execution bottleneck'—models can often devise correct strategies but cannot faithfully execute the necessary calculations or reasoning steps. Code-based methods prove brittle when handling raw scientific tables with irregular formatting, while natural language models primarily fail due to initial comprehension issues and calculation errors.

This research exposes a fundamental limitation in current AI systems: they struggle with tasks requiring both precise numerical computation and contextual understanding of scientific data. The findings suggest that simply scaling model size or training data won't solve this problem, pointing instead to architectural limitations in how AI systems process and execute multi-step reasoning. The team's work provides a concrete measurement tool for tracking progress in this critical area of AI capability, with implications for scientific research automation and data analysis tools.