ICML 2026 paper exposes instability in class-split anomaly detection

A new paper accepted at the ICML 2026 Workshop on Hypothesis Testing reveals a critical flaw in how anomaly detection (AD) models are evaluated using class-split benchmarks. Authors Ascarate, Lebrat, Santa Cruz, Fookes, and Salvado demonstrate that when the held-out anomaly class overlaps the normal data distribution in representation space, anomaly scores can become unstable—collapsing toward random chance or even inverting direction. This means a model that appears to detect anomalies correctly on one split could fail or reverse its predictions on another, depending on which class is held out.

The team proposes a simple, training-free diagnostic called "neighborhood class leakage" that accurately predicts score-direction instability across three datasets: Fashion-MNIST, CIFAR-10, and Imagenette, tested in both pixel space and VAE latent spaces. Their findings suggest that class-split AD benchmarks should be treated as geometry-dependent stress tests rather than unconditional evidence of anomaly-detection ability. This work has direct implications for practitioners using class-split evaluations to select or validate AD models, especially in high-stakes applications like manufacturing defect detection or medical imaging, where misclassification of rare anomalies can be costly.