Researchers' VideoKR dataset powers AI with 315K expert video reasoning examples

Video understanding has long struggled with tasks that require external knowledge and deep reasoning, often defaulting to shallow textual cues. Enter VideoKR, a breakthrough dataset introduced by Lin Fu and five co-authors, accepted as an ICML 2026 Spotlight. The corpus comprises 315K reasoning examples drawn from 145K carefully curated, CC-licensed videos spanning expert domains like science, medicine, and engineering. Unlike existing datasets, VideoKR is built via a human-in-the-loop, skill-oriented generation pipeline that targets progressively deeper reasoning capabilities—from basic comprehension to multi-step inference—while ensuring example difficulty, diversity, and reliability. Each example includes detailed Chain-of-Thought (CoT) rationales, forcing models to reason step-by-step rather than memorize patterns. The team also releases VideoKR-Eval, a new expert-annotated benchmark designed to measure genuine video understanding without textual shortcuts.

Experiments show that models post-trained on VideoKR using a standard SFT (supervised fine-tuning) followed by GRPO (group relative policy optimization) pipeline significantly outperform prior post-training approaches on knowledge-intensive video reasoning benchmarks. Remarkably, these models also remain competitive on general video reasoning tasks, indicating that VideoKR's focus on deeper reasoning does not come at the cost of broader capabilities. Comprehensive ablation studies isolate the contributions of each design choice, providing actionable insights for future work. The findings underscore that data quality and deliberate skill-oriented design are key drivers of progress in video AI, opening the door to truly intelligent video analysis systems capable of understanding complex, expert-level content.