Multi-task Just Recognizable Difference for Video Coding for Machines: Database, Model, and Coding Application
New model slashes video file sizes for surveillance and autonomous systems while preserving AI accuracy.
A research team including Junqi Liu, Yun Zhang, and others has published a breakthrough paper on arXiv introducing a new method for optimizing video compression specifically for machine consumption. Their work tackles Video Coding for Machines (VCM), a field focused on efficiently streaming video to AI systems—like those in autonomous vehicles or security cameras—rather than human eyes. The core innovation is the Multi-Task Just Recognizable Difference (MT-JRD) concept, which identifies the precise threshold of visual detail an AI model needs to perform tasks accurately. By understanding this 'just recognizable difference,' engineers can strip away superfluous data that machines ignore, dramatically reducing file sizes.
To build this system, the team first created a substantial dataset of 27,264 JRD annotations across three core machine vision tasks: object detection, instance segmentation, and keypoint detection. They then developed the Attribute-assisted MT-JRD (AMT-JRD) prediction model. This model uses specialized modules, including a Generalized Feature Extraction Module (GFEM) and an Attribute Feature Fusion Module (AFFM), which incorporates prior knowledge about object size and location to enhance predictions beyond raw image data alone.
The results are significant for real-world deployment. The AMT-JRD model achieved a mean absolute error of 3.781 in its predictions, outperforming previous single-task models by 6.7%. When applied to video compression, this precision translates directly into bandwidth savings. Compared to standard codecs like VVC and JPEG, the AMT-JRD-based VCM system improved coding efficiency by an average of 3.861% and 7.886% respectively, as measured by the Bjontegaard Delta-mean Average Precision (BD-mAP) metric. This means AI systems can receive high-quality video feeds using significantly less data, lowering costs and latency for applications from traffic monitoring to robotic vision.
- The AMT-JRD model predicts the 'Just Recognizable Difference' for AI across 3 tasks (detection, segmentation, keypoints) with a mean error of 3.781, beating prior models by 6.7%.
- It uses a novel Attribute Feature Fusion Module (AFFM) that incorporates object size and location data to improve prediction accuracy beyond standard image features.
- Applied to video compression (VCM), it reduces bitrates by an average of 7.886% (BD-mAP vs. JPEG) while preserving AI task accuracy, enabling more efficient streaming for machines.
Why It Matters
Enables efficient, low-bandwidth video feeds for city-scale surveillance, autonomous vehicles, and industrial IoT, reducing cloud costs and latency.