New AI codec MS-VQ-VAE compresses video 7.6x below H.265

Learned video codecs based on continuous latent representations (like traditional autoencoders) struggle below 0.1 bits per pixel (bpp) because the Lagrangian optimization technique used to balance quality and bitrate fails without a differentiable rate signal. This new paper from Manikanta Kotthapalli and Banafsheh Rekabdar on arXiv demonstrates that discrete latent representations entirely sidestep this limitation. Using a vector-quantized (VQ) approach, the codebook size imposes a hard information ceiling, while a learned autoregressive prior exploits the power-law distribution of code usage to push actual bitrates far below that ceiling—without any rate-penalty tuning. The architecture builds on MS-VQ-VAE, sweeping codebook sizes from 128 to 1024.

The results are dramatic: on 500 UCF101 test clips (64x64, 32 frames), the codec operates at 0.043–0.064 bpp, which is 3.3-5× below H.264's practical floor and 5-7.6× below H.265's floor at this resolution. Every configuration outperforms H.265 CRF 36 on perceptual quality (LPIPS). At K=1024, the model beats H.265 by 0.072 absolute LPIPS while using 5.1× fewer bits. The authors also resolved a critical training instability: gradient-based VQ collapses at K ≤ 512, but EMA-stabilized codebook updates with dead-code restart maintain full utilization. Codebook analysis confirms power-law index distributions and 70-85% entropy efficiency, establishing this pipeline as a principled learned entropy coder.