New Metric Exposes 17.5% of Expected Code Behaviors Go Untested

Traditional test adequacy metrics like code coverage and mutation testing measure what the code does, not what it should do. In a new study published on arXiv, researchers Partha Protim Paul and Reid Holmes propose a method to detect behavioral gaps—discrepancies between expected behaviors (extracted from natural language documentation and source code) and those actually validated by existing test cases. Their approach maps these expectations to test cases and flags any that are missing.

Evaluating the approach across ten popular open-source Java libraries, they extracted 20,729 behaviors with 93.1% precision. Their empirical analysis conservatively estimates that 17.5% of detected expected behaviors remain entirely untested, terming this the 'behavioral gap.' To rule out that these gaps are just an artifact of human-written tests, they tested state-of-the-art automated test generators EVOSUITE and ASTER, which similarly failed to validate at least 20.6% and 27.1% of expected behaviors respectively. Importantly, behavioral gaps were not predicted by traditional structural metrics: the majority of untested behaviors occurred in methods that already had high line coverage, and over half persisted in methods with high mutation kill scores. These results suggest that behavioral coverage is an independent dimension of test suite adequacy that can complement—not replace—traditional metrics, giving developers a new way to catch missing test cases that even automated generators miss.