SEC-VCM cuts video bitrate by up to 97.6% for machine vision tasks

Electrical Engineering and Systems Science researchers led by Yuxiao Sun have introduced SEC-VCM (Symmetric Entropy-Constrained Video Coding for Machines), a framework designed specifically for video transmission to AI systems rather than human viewers. Traditional video codecs like H.266/VVC optimize for perceptual quality, but machine vision systems (MVS) don't need every pixel—only the semantic information that drives understanding. Existing VCM approaches often bind codecs to specific downstream models, requiring retraining. SEC-VCM solves this by establishing a symmetric alignment between the codec and a visual backbone (VB), allowing the codec to leverage representation capabilities from foundation models to preserve semantics while discarding MVS-irrelevant data.

SEC-VCM introduces two key innovations: a Bi-directional Entropy-Constrained (BiEC) mechanism that enforces symmetry between decoding and VB encoding by suppressing conditional entropy, and a Semantic-Pixel Dual-Path Fusion (SPDF) module that injects pixel-level priors into the final reconstruction, suppressing artifacts harmful to machine vision. The results are dramatic: compared to the VVC reference software VTM, SEC-VCM achieves bitrate savings of 37.4% on video instance segmentation, 29.8% on video object segmentation, 46.2% on object detection, 44.9% on multiple object tracking, and an eye-popping 97.6% on MLLM-based video grounding. The paper has been accepted by IEEE Transactions on Image Processing.