Toward Reliable Evaluation of LLM-Based Financial Multi-Agent Systems: Taxonomy, Coordination Primacy, and Cost Awareness

A new academic paper by researchers Phat Nguyen and Thang Pham tackles the growing but chaotic field of AI-powered financial trading agents. The study analyzes 12 different multi-agent systems (where multiple AI agents collaborate) and two single-agent baselines, introducing a four-dimensional taxonomy to categorize them by architecture, coordination, memory, and tool use. Its central argument is the Coordination Primacy Hypothesis (CPH), which posits that the design of how agents communicate and coordinate is a more significant driver of trading performance than simply using a larger or more expensive AI model. The authors present this as a testable hypothesis, noting that definitive proof requires evaluation infrastructure that currently doesn't exist.

The paper's most critical contribution is its documentation of five pervasive and damaging evaluation failures common in the field: look-ahead bias, survivorship bias, backtesting overfitting, transaction cost neglect, and regime-shift blindness. The authors demonstrate that these methodological flaws are severe enough to completely reverse the sign of reported returns, turning paper profits into real-world losses. To address this, they propose a new metric called the Coordination Breakeven Spread (CBS), designed to measure whether the complexity of multi-agent coordination genuinely adds value after accounting for transaction costs. The work concludes by advocating for the adoption of minimum evaluation standards as a prerequisite for credible research and validation of claims about AI trading system performance.