NSVQ: New method solves codebook collapse in vector quantization

Vector quantization (VQ) is a cornerstone of modern generative AI, enabling models like VQGAN and DALL-E to compress images into discrete tokens. However, large-codebook VQ models frequently suffer from codebook collapse—where many code vectors become unused, degrading reconstruction quality. In a new paper on arXiv, researchers from multiple institutions identify a previously overlooked cause: encoder drift. As the encoder shifts the latent distribution during training, sparsely updated code vectors fail to keep up, lose assignments, and create a feedback loop that amplifies quantization error.

The team introduces NSVQ (Non-Stationary Vector Quantization), a three-pronged strategy. First, a dense non-stationary embedding loss helps the codebook track encoder drift in early training. Second, they freeze the encoder to let the codebook consolidate under a fixed latent geometry. Finally, adversarial refinement is reintroduced to sharpen outputs. On ImageNet-1k at 128x128 resolution with 65,536 codes, NSVQ reduces rFID from 2.39 to 2.10 compared to SimVQ while maintaining full codebook utilization. The method also improves FID scores in downstream latent diffusion models, suggesting real-world gains for image generation pipelines.