Independent audit reveals 99% of anomaly detection metrics still gameable by no-skill detectors

A new independent adversarial stress-test evaluates 12 metrics adopted after Kim et al. (2022) showed that point-adjustment (PA) awards perfect F1 to random scores. The paper tests trivial and adversarial no-skill score generators against 12 replacement metrics on the 250-series UCR Anomaly Archive plus SMD, SMAP, MSL, NAB, PSM. The criterion for 'gamed' is a no-skill generator reaching >=90% of the best real detector's score. Results show the fix is only partial: affiliation-F1 is gamed on 99% of series, ROC-based metrics (e.g., VUS-ROC) on 62-64%, while PR-based metrics and PA%K resist (14-18%). A decisive McNemar split shows VUS-ROC is gamed on 119 series where VUS-PR is not, with no reverse cases.

The findings recommend a metric-selection protocol: prefer a PR-based metric or PA%K, treat affiliation-F1 and ROC-AUC variants as gameable, and verify per benchmark rather than assuming robustness. No single metric stays safe across all six benchmarks (e.g., VUS-PR is gamed on NAB). The paper releases a pip-installable stress-test harness with vendored metric implementations. For practitioners, this means that using ROC-based metrics can give false confidence, while PR-based metrics provide a more reliable foundation for evaluating time-series anomaly detectors.