BAMU boosts neural speech codec quality 10% without retraining

Pretrained neural speech codecs typically assign a fixed residual vector quantization (RVQ) depth to every audio frame, ignoring the fact that some frames are harder to compress than others. This wastes bits on easy frames and under-allocates on difficult ones. In a new arXiv paper, Mingyu Zhao and eight colleagues from Tsinghua University and industry labs propose BAMU (Bitstream-Aware Marginal-Utility Allocation), a framework that works entirely on frozen, off-the-shelf codecs — no retraining required.

BAMU uses a lightweight, rate-independent predictor to estimate how much each additional RVQ layer will reduce latent distortion for each frame. A constrained allocator then chooses prefix-valid RVQ depths under an exact serialized-bit budget, adapting bitrate frame-by-frame. Experiments on LibriSpeech and VCTK show consistent quality gains for EnCodec and improvements for DAC at medium and high bitrates. In a 30-listener subjective test, BAMU raised the MOS from 3.449 to 3.780 compared to matched fixed-depth coding. That's a meaningful jump in perceived speech quality, achieved purely by smarter bit allocation instead of heavier models or larger bitrates.