AgentEval mines workflow graphs to catch LLM agent failures
Automated testing finds 23-38 hidden boundaries per agent without source code
Deep Dive
AgentEval is a black-box testing framework that mines conversational workflow graphs to identify and stress state-dependent failures in LLM agents. By enumerating guards and prerequisites, AgentEval generates tests covering 23–38 distinct boundaries per agent, outperforming a prompt-only baseline (23 vs. 12) with lower duplicate and false-alarm rates. It uses only conversation turns, with no access to source code.
Key Points
- AgentEval mines a conversational workflow graph from agent interactions to identify hidden state-dependent boundaries.
- It discovers 23–38 distinct boundaries per agent, outperforming prompt-only baselines (23 vs 12) with fewer duplicates and false alarms.
- Uses only conversation turns and no source code access, matching a white-box auditor's coverage in benchmark tests.
Why It Matters
This makes LLM agent testing more reliable, preventing costly errors like unauthorized transactions in production systems.