AI agents write better edge-case tests but introduce more flakiness

A team of researchers from Stevens Institute of Technology and other institutions has published a large-scale empirical analysis on arXiv comparing the quality of AI agent-generated unit tests against human-written ones. The study, titled 'Beyond Test Presence: Assessing the Quality and Robustness of Agent-Generated Tests in Open-Source Projects,' analyzed 204,673 test artifacts from the AIDev dataset, including 24,941 human-authored and 179,732 agent-generated files. Using AST parsing with Python's ast module, they evaluated three dimensions: assertion strength, edge-case coverage, and flakiness potential.

The results reveal a nuanced trade-off. AI agents excelled in edge-case coverage—nearly doubling the variety of boundary checks (0.62 vs. 0.32) and showing higher null-safety testing (13.4% vs. 8.3%). However, human developers maintained a slight edge in assertion strength (88.1% strong assertions vs. 85.37% for agents). The bigger concern: agent-generated tests were more flaky (0.41 candidate flakiness rate vs. 0.30), largely due to reliance on file I/O and non-deterministic logic. The authors warn this creates 'stealth technical debt'—test suites that pass but lack robustness. The findings suggest AI needs better 'environmental awareness' to write hermetic, stable tests.