OmniMem's new memory compression boosts audio-visual LLMs by 4%
Video LLMs now process 2x longer streams with same memory using OmniMem.
OmniMem addresses a critical bottleneck in streaming audio-visual large language models: the quadratic growth of key-value (KV) caches as video length increases. Unlike prior methods that compress all tokens uniformly, OmniMem introduces a modality-aware memory allocation strategy that separately manages visual and audio contexts. This tackles the severe token imbalance between the two modalities—audio typically has far fewer tokens but carries crucial information. The framework then applies perturbation-aware memory selection to preserve only the most informative and non-redundant KV states, enabling compact memory without breaking long-range dependencies.
To further optimize for real-world deployment, the team developed budget-aware fine-tuning, which encourages models to consolidate useful information into retained memory. Tested on VideoMME Long, LVBench, and LVOmniBench with video-SALMONN 2+ and Qwen-2.5-Omni, OmniMem consistently outperforms strong training-free compression baselines by 2-4% absolute accuracy under the same memory budgets. After fine-tuning, gains increase to an additional 1-2%. The approach is model-agnostic and open-sourced, making it practical for any streaming audio-visual application that needs longer context without exploding memory costs.
- OmniMem uses modality-aware memory allocation to balance visual and audio tokens, addressing token imbalance in streaming video.
- Perturbation-aware selection retains only informative, non-redundant KV states, cutting memory without harming long-range understanding.
- On benchmarks like VideoMME Long and LVBench, OmniMem achieves 2-4% higher accuracy at the same memory budget, with up to 6% after fine-tuning.
Why It Matters
Enables long-form video understanding on existing hardware, unlocking real-time streaming analysis for surveillance, media, and robotics.