RFSQ compression method beats RVQ with 3.6% audio quality boost

Finite Scalar Quantization (FSQ) simplifies neural compression training by using fixed codebooks, but it suffers from residual magnitude decay in multi-stage settings—each subsequent stage receives exponentially weaker signals, limiting representational power. A new paper from Xiaoxu Zhu and colleagues at arXiv introduces Robust Residual Finite Scalar Quantization (RFSQ), which tackles this bottleneck with two novel conditioning strategies: learnable scaling factors and invertible layer normalization.

RFSQ's experiments span both audio and image modalities. For audio reconstruction at 24 bits/frame, RFSQ-LayerNorm achieves a DNSMOS score of 3.646, a 3.6% improvement over the state-of-the-art RVQ method (3.518). On ImageNet, RFSQ reaches 0.102 L1 loss and 0.100 perceptual loss; the LayerNorm variant delivers 9.7% better L1 and 17.4% better perceptual performance compared to unconditioned baselines. The LayerNorm strategy maintains normalized input statistics across stages, effectively preventing the exponential magnitude decay that limits naive residual approaches. RFSQ combines FSQ's training simplicity with multi-stage quantization's representational power, positioning itself as a new standard for neural compression across diverse modalities.