A Ground-Truth-Based Evaluation of Vulnerability Detection Across Multiple Ecosystems
Researchers built a ground-truth dataset from OSV to compare detection tools.
A new empirical study by researchers Peter Mandl, Paul Mandl, Martin Häusl, and Maximilian Auch evaluates automated vulnerability detection tools across multiple software ecosystems. The study uses a carefully curated ground-truth dataset derived from the Open Source Vulnerabilities (OSV) database, which explicitly maps vulnerabilities to specific package versions. This allows for a systematic comparison of detection results across different tools and services, revealing significant inconsistencies and systematic differences between detection systems.
The dataset represents a snapshot of the vulnerability landscape at the time of evaluation, as OSV is continuously updated. To support reproducibility and future studies, the authors provide an open-source tool that automatically reconstructs the dataset from the current OSV database using the described methodology. The study underscores the importance of transparent dataset construction for reproducible empirical security research, offering a foundation for improving automated vulnerability detection.
- Uses a curated ground-truth dataset from the OSV database mapping vulnerabilities to concrete package versions.
- Highlights systematic differences between vulnerability detection systems across multiple ecosystems.
- Provides an open-source tool to automatically reconstruct the dataset for reproducibility.
Why It Matters
This study reveals critical gaps in vulnerability detection, urging more reliable tools for securing software dependencies.