Pseudo-label distillation compresses SSL audio models without accuracy loss

Current discriminative anomalous sound detection (ASD) methods rely on expensive machine-information labels to train feature extractors, while self-supervised learning (SSL) alternatives are computationally heavy. To bridge this gap, Takuya Fujimura and Tomoki Toda introduce a pseudo-label distillation framework that extracts cheap labels from SSL features and uses them to train a compact discriminative extractor. This transfers SSL's high performance to a much smaller model, achieving state-of-the-art ASD accuracy at a fraction of the cost.

To further improve robustness, the authors propose lightweight Noise-Robust Feature Transformation (NRFT) methods that leverage a small amount of clean or isolated noise data. Evaluations on the DCASE 2020–2025 Task 2 datasets using four different SSL models show that pseudo-label distillation not only replicates SSL performance but often exceeds it when combined with coarse labels and data augmentation. NRFT provides additional gains, making the approach practical for real-world industrial monitoring where labeled data is scarce and edge deployment is key.