GOPAgen uses video codec GOPs for efficient long-video understanding

Despite advances in long-video understanding, existing agentic methods struggle with detailed motion comprehension and efficient memory. GOPAgen, from researchers Haozhe Chi, Yang Jin, and Yadong Mu, introduces a breakthrough by directly integrating video codec into the understanding pipeline. The system uses a motion agent trained on Groups of Pictures (GOPs) — the structural units of video compression — to capture fine-grained motion. A GOP tree reasoning algorithm, naturally aligned with codec hierarchy, enhances local motion understanding. Additionally, a structural memory mechanism stores local motion with detailed captions in structured pages, and a coarse-to-fine zoom-in algorithm efficiently navigates this memory. A motion vector database enables retrieval at multiple granularities.

GOPAgen achieves state-of-the-art Video Question Answering (VQA) results on MotionBench and Egoschema, demonstrating superiority in motion-aware long-video tasks. The approach dramatically reduces computational overhead by leveraging existing codec outputs, making it practical for real-world video analytics. This work opens new possibilities for tasks like surveillance, sports analysis, and video summarization where detailed motion understanding is critical.