Trade-offs Between Capacity and Robustness in Neural Audio Codecs for Adversarially Robust Speech Recognition

A team from the University of Southern California's SAIL lab, led by Jordan Prescott, has published a pivotal study on securing automatic speech recognition (ASR) systems. The research focuses on neural audio codecs—AI models that compress speech into discrete tokens—and their innate ability to filter out adversarial perturbations. These are subtle, inaudible noises crafted to trick voice assistants like Siri or Alexa into executing unauthorized commands. The core discovery is a non-monotonic trade-off governed by the codec's quantization depth.

By manipulating the Residual Vector Quantization (RVQ) layers, the researchers found that shallow quantization acts as a blunt filter, suppressing attack noise but also degrading legitimate speech content, leading to high transcription errors. Conversely, very deep quantization preserves perfect audio fidelity but also lets the adversarial perturbations through unscathed. The breakthrough is identifying an intermediate depth that optimally balances these effects, creating a 'Goldilocks zone' that minimizes word error rate under attack. This configuration reduced transcription errors by approximately 40% compared to vulnerable systems.

Crucially, the defense proved robust. The team demonstrated that the gains hold even under 'adaptive attacks,' where a hacker knows the defense is in place and tries to circumvent it. The method outperformed traditional compression-based defenses like MP3 or AAC. The researchers also identified a strong correlation between changes in the discrete codebook tokens and final transcription error, providing a new metric for evaluating system vulnerability. This work, submitted to Interspeech 2026, provides a blueprint for building more inherently robust voice AI by design, rather than relying on brittle add-on filters.