VigilFormer detects surveillance anomalies at 41.5 FPS with record AUC scores

Video anomaly detection in surveillance has long faced a trade-off between accuracy and real-time performance. Existing methods either rely on heavy feature extractors or efficient but less accurate architectures. VigilFormer, introduced by Xinze Zhang in a new arXiv paper, tackles this by combining a Deformable Spatio-Temporal Encoder (DSTE) with a Causal Anomaly Classifier (CAC) and an Adaptive Confidence Scheduler (ACS). The DSTE attends to only a sparse set of informative locations across frames, avoiding the quadratic cost of dense self-attention while capturing irregular motion patterns. The CAC uses dilated causal convolutions over snippet-level features and optimizes a contrastive multiple-instance learning objective, enabling separation of anomalous and normal representations without requiring frame-level labels. To further boost inference speed, the ACS dynamically skips low-information frames in static scenes, reducing redundant computation.

Evaluated on three standard benchmarks—UCF-Crime, ShanghaiTech, and CUHK Avenue—VigilFormer achieves AUC scores of 87.83%, 97.21%, and 89.74% respectively, while running at 41.5 FPS on a single GPU. This represents a significant improvement over recent weakly-supervised methods in both accuracy and throughput. The combination of sparse deformable attention, causal temporal modeling, and adaptive frame skipping offers a practical path for deploying high-accuracy anomaly detection in real-world surveillance systems where compute resources are limited. The paper is available on arXiv under the identifier 2606.14724.