IBM's Tangent study: LLM agent testing dominated by shallow unit tests
2,572 test methods analyzed, 23 patterns found—and most are narrowly scoped.
IBM Research researchers (Rangeet Pan, Tyler Stennett, Divya Sankar, and colleagues) released Tangent, an empirical study of how LLM-based agent applications are actually tested in practice. The team mined a large corpus of open-source agent projects, manually labeling 2,572 test methods across 240 modules to derive a taxonomy of 23 testing patterns spanning fixtures, data, objectives, and assertions. Their findings: agent tests are overwhelmingly narrow unit tests that rely on simplistic inputs, aggressive mocking, and shallow validation—while complex multi-step interactions, realistic external tool use, and non-functional requirements (like latency and cost) receive minimal coverage.
Interestingly, structured interviews with 10 senior industry practitioners showed they place more emphasis on non-functional testing than open-source developers do, but both groups struggle with unclear test objectives and generating high-quality test data. The paper argues for formal foundations for agent testability, explicit test objectives, and fault-based testing techniques—urgent guidance as agentic AI moves from demos to mission-critical enterprise workflows.
- IBM researchers manually labeled 2,572 test methods from 240 modules in open-source agent projects to build a 23-pattern testing taxonomy.
- Tests lean on narrow unit scopes, heavy mocking, and shallow assertions, leaving complex interaction and non-functional coverage sparse.
- Interviews with 10 senior industry practitioners confirmed shared gaps: unclear objectives and hard-to-generate realistic test data.
Why It Matters
As agentic AI enters production, engineers need rigorous testing patterns—this study gives the first large-scale map of what's broken.