New CFKD framework boosts speaker verification from audio codecs

Neural audio codecs (NACs) are widely used for efficient audio compression and have excelled in tasks like speech synthesis. However, their discrete representations have historically lagged behind traditional spectral features (e.g., Fbanks) in automatic speaker verification (ASV). A new paper accepted at Interspeech 2026, authored by Zheng Liang, Junjie Li, and Kong Aik Lee, empirically demonstrates that speaker cues are implicitly present in these discrete tokens but are poorly leveraged by standard ASV training methods.

The team introduces Cross-Feature Knowledge Distillation (CFKD), which guides a codec-based student model to mimic the embedding space of a strong Fbank-based teacher. This structured supervision allows the student to make fuller use of speaker information encoded in discrete audio tokens. Experiments on the VoxCeleb benchmarks show that CFKD substantially closes the accuracy gap, with codec-based systems approaching the performance of Fbank-based models. This work highlights the untapped potential of discrete audio tokens for diverse speech tasks beyond compression.