AV-Flamingo: Open model understands long videos with audio and visual reasoning

Audio-Visual Flamingo (AV-Flamingo) is a fully open, state-of-the-art audio-visual large language model designed for joint understanding and reasoning over audio, images, and long-form videos. Unlike prior models that focus on short clips, AV-Flamingo handles complex, real-world videos lasting minutes or more. The model introduces three key innovations: Audio-Visual-Skills, a dataset of ~7M caption and question-answer training instances emphasizing temporal, compositional, and cross-modal reasoning; a three-stage curriculum that progressively trains from short-range perception to long-horizon multi-event reasoning; and Temporal Audio-Visual Interleaved Chain-of-Thought, which explicitly grounds intermediate reasoning steps to timestamps, improving temporal alignment and interpretability.

Extensive experiments across 15+ audio-visual, omni-modal, audio, and vision benchmarks show that AV-Flamingo outperforms similarly sized open models by clear margins and remains highly competitive with—and in some cases surpasses—much larger open-weight and closed models, particularly on long and complex real-world audio-visual understanding and reasoning tasks. Beyond benchmarks, AV-Flamingo demonstrates strong real-world utility and transfers well to unseen tasks, highlighting its robustness and generalization ability. The model is fully open, with training data, code, and weights to be released, enabling researchers and developers to build on top of it.