Reconstruct! Don't Encode: Self-Supervised Representation Reconstruction Loss for High-Intelligibility and Low-Latency Streaming Neural Audio Codec

A research team from Johns Hopkins University and the University of Southern California has introduced a breakthrough in neural audio codec training with their paper "Reconstruct! Don't Encode." They developed a novel Self-Supervised Representation Reconstruction (SSRR) loss that fundamentally changes how codecs are optimized. Unlike traditional methods that focus on mel-spectrogram reconstruction—which often sacrifices intelligibility—SSRR ensures the codec's output can accurately reconstruct semantic, content-rich representations from models like WavLM. This shift directly targets preserving the actual meaning and clarity of speech.

The result is JHCodec, a state-of-the-art model that delivers three key advantages. First, SSRR dramatically accelerates training convergence, enabling researchers to achieve competitive, high-performance results using only a single GPU—a significant reduction in cost and barrier to entry. Second, it produces superior intelligibility by ensuring critical speech content is preserved. Third, and crucially for real-world applications, it allows for a zero-lookahead architecture in streaming Transformer-based codecs. This eliminates the need for future audio frames to process the current one, achieving minimal latency for real-time communication and deployment. The team has open-sourced the full code, training pipeline, and demos, inviting further development and application.