Generating Expressive and Customizable Evals for Timeseries Data Analysis Agents with AgentFuel

A research team from Carnegie Mellon University and other institutions has published a paper introducing AgentFuel, a new framework designed to solve a critical problem in evaluating AI agents for timeseries data analysis. These conversational agents, which let users "talk to your data," are increasingly used in domains like IoT, cybersecurity, and product analytics. However, the researchers found that existing evaluation methods (evals) have major expressivity gaps, lacking both domain-customized datasets and domain-specific query types. Their analysis of six popular agents revealed consistent failures on complex, stateful queries and incident-specific analyses, highlighting a significant shortcoming in current technology.

AgentFuel directly addresses this by enabling practitioners and domain experts to quickly generate tailored, expressive benchmarks. This allows for comprehensive end-to-end functional testing of data analysis agents on realistic scenarios. The framework's benchmarks have already exposed key directions for improvement in existing agent frameworks. Furthermore, the team provides anecdotal evidence that using AgentFuel can lead to tangible performance gains, as demonstrated with an agent called GEPA. By providing a standardized way to create rigorous, customizable tests, AgentFuel aims to drive the development of more robust and reliable AI agents for critical business intelligence tasks.