Your Company's AI Analyst May Be Confidently Wrong — New Test Finds Out

Companies are starting to use AI agents (software that can take actions on its own) to answer business questions. You ask, "Why did sales drop in Europe?" and the AI picks the data sources, runs the numbers, and writes an answer that might land in a board deck. A new research paper argues that judging these tools by their final answer alone is dangerous, because a good-looking answer can quietly come from the wrong data.

The team built a testing method that grades the whole journey, not just the destination. They check three things: whether the AI understood the question, whether it ran the steps correctly, and whether it gives the same answer twice. They tested an internal analytics agent at a large online marketplace using 50 real business questions, two AI setups, and three repeat runs each — 300 runs in total.

The results are a warning. The more capable AI refused to answer only 0% of the time, down from 73%, and gave real-data answers 73% of the time, up from 21%. That sounds great. But it also ran out of its allowed steps on 16% of runs, blew past its data-exploration limit on 77% of runs, and changed how it read the same data tables on 41 of 50 questions. It was more helpful and less reliable at the same time.

The lesson for anyone buying or using these tools: ask how they were tested. A demo that answers one question well proves almost nothing. You need repeated runs, visible step-by-step records, and honest reporting of how often the AI gives up or contradicts itself — especially for anything touching money, hiring, or strategy. The paper's authors say these scores should guide decisions, not act as a simple pass-or-fail gate.