From Comprehension to Reasoning: A Hierarchical Benchmark for Automated Financial Research Reporting

A research team led by Yiyun Zhu and Dawei Cheng has published a new benchmark called FinReasoning, designed to rigorously test the ability of large language models (LLMs) to generate reliable financial research reports. The benchmark was created in response to real-world deployments revealing persistent failures like factual errors, numerical inconsistencies, and fabricated references in AI-generated analysis, which can lead to severe economic losses. Unlike previous benchmarks focused on comprehension, FinReasoning decomposes the report generation process into three stages aligned with actual analyst workflows: semantic consistency, data alignment, and deep insight generation. It introduces a fine-grained evaluation framework that strengthens hallucination detection and incorporates a detailed 12-indicator rubric to measure core analytical skills.

The evaluation results exposed a critical 'understanding-execution gap' in most models. While LLMs could often identify errors in existing reports, they struggled significantly to generate accurate corrections themselves. Similarly, models could retrieve relevant data but frequently failed to return it in the correct, usable format. The benchmark ranked several leading models, finding that no single AI achieved overwhelming superiority. Doubao-Seed-1.8, OpenAI's GPT-5, and Kimi-K2 emerged as the top three in overall performance, but each exhibited a distinct and uneven distribution of capabilities across the different analytical tracks. This highlights that current AI systems are not yet reliable primary producers of financial analysis and require careful, task-specific evaluation.