JOMP framework cuts neural video codec operations by 87.6%

Neural video codecs based on variational autoencoders achieve impressive rate-distortion performance but rely on floating-point arithmetic, hindering real-world adoption due to high computational complexity and limited cross-platform interoperability. Converting these models to integer-based networks is challenging because inter-dependent coding components have varying sensitivity to precision across different codec designs. This paper from Yu-Hsiang Lin and colleagues addresses these obstacles with JOMP (Jointly-Optimized Mixed-Precision Quantization), a framework that jointly optimizes quantization parameters and bit widths during training.

JOMP is the first mixed-precision quantization framework specifically designed for neural video codecs. It allows different modules within a codec to operate at different precision levels, optimizing the rate-distortion-complexity trade-off. The researchers validated JOMP across multiple coding frameworks and temporal buffering strategies, providing a unified understanding of their combined effects. When applied to their best-performing model, JOMP achieved rate-distortion performance comparable to the state-of-the-art DCVC-FM while slashing bit operations by 87.6%. The team also developed a complete integerization pipeline for deterministic decoding, marking a significant step toward practical, efficient neural video codecs.