SemaDiff uses LLM-generated tests to spot semantic-changing commits
Detects behavior changes with 100% precision by comparing test results across versions.
Traditional refactoring detectors struggle to confirm that a commit contains only semantic-preserving changes—no hidden behavior modifications. This gap complicates tasks like bug localization, dataset construction, and rollback analysis. SemaDiff fills it by performing a behavior-based analysis: for a given commit, it examines the diff to identify modified code and extracts unchanged dependent code that calls it.
SemaDiff then uses an LLM to generate additional class that exercises the changed code in both versions, along with automatic tests for that dependent code. Running the same tests on both versions reveals any behavioral differences. If all tests produce identical outcomes, the commit is classified as semantic-preserving. Evaluated on 183 annotated commits from popular open-source Java projects, SemaDiff achieves 76% overall accuracy and 100% precision in identifying semantic-changing commits, making it a precise tool for software repository mining.
- SemaDiff generates LLM-based test code that calls the modified portion of a commit, enabling behavioral comparison.
- Achieves 100% precision in detecting semantic-changing commits, ensuring no false positives for behavior-changing commits.
- Evaluated on a manually labeled dataset of 183 commits from well-known open-source Java projects.
Why It Matters
Boosts reliability of commit classification for debugging, rollback, and bug dataset construction.