PROGRESS: New AI testing method catches 58% of Java bugs

PROGRESS (PROperty-Guided REgression Search for Semantic Falsification) is a new testing framework from UVA researchers that fuses two testing approaches. Search-based regression testing generates tests that cover complex code paths but derives oracles from the system-under-test, so existing bugs get encoded as expected behavior. Property-based testing provides independent semantic oracles (properties that should always hold), but it needs high-quality properties and offers little guidance for reaching deep program states. PROGRESS bridges this gap by integrating intent-driven properties into coverage-guided evolutionary search.

The framework works in three steps: it extracts intent-bearing context from code and uses a language-model pipeline to generate executable jqwik properties while limiting implementation leakage; it extends EvoSuite's DynaMOSA algorithm with a per-property fitness function that rewards progress through preconditions and prioritizes falsifying executions; and it binds property parameters to evolving test sequences via jqwik generators. Evaluated on 25 large-scale Java systems, PROGRESS detected 328/562 current-version bugs (58%) versus none for regression-test generation, and satisfied all preconditions for 70/150 hard-to-reach properties compared to 18 for standalone jqwik. Ablations confirmed documentation and caller/callee context are critical for generating valid executable properties, and the team released a full artifact package.