New Financial Benchmark Exposes AI Reasoning Flaws

A team of 13 researchers from Peking University and Zhejiang University introduced FinIndices, a benchmark designed to test Large Language Models (LLMs) on financial reasoning tasks that require numerical precision and multi-step logic. Unlike existing benchmarks that rely on simple QA or cropped tables, FinIndices uses uncropped financial statements up to 32K tokens to evaluate real-world complexity. The evaluation exposed two critical vulnerabilities: first, a 'Knowledge Bottleneck' where models fail on tasks requiring temporal de-cumulation and stock-flow caliber matching (e.g., Gemini-3.1-Pro's accuracy plummeted from 70.70% to 38.22% when formula hints were removed); second, a 'Structural Bottleneck' where cognitive load from multi-metric, multi-period tables forces models to regress to shallow heuristics like incorrect column fetching.

The researchers also found that Supervised Fine-Tuning (SFT) could partially restore structured logic, boosting performance by up to 8.54% on single-index tasks and 3.82% on table tasks. FinIndices is now publicly available, offering a more rigorous testbed for financial reasoning in LLMs. The findings highlight the limitations of current benchmarks and underscore the need for improved alignment techniques to handle high-stakes financial applications.