Frequency-Modulated Visual Restoration for Matryoshka Large Multimodal Models

A team of researchers has published a paper on arXiv introducing FMVR (Frequency-Modulated Visual Restoration), a breakthrough technique designed to solve a major bottleneck in Large Multimodal Models (LMMs). These models, which process both images and text, traditionally generate a massive number of "visual tokens" from an image, leading to high computational costs. Previous methods to reduce these tokens often resulted in a significant loss of visual detail and semantic information. FMVR offers a novel, plug-and-play solution that disentangles visual representations into distinct frequency components, allowing for smarter compression.

Specifically, FMVR uses two simple operations—AvgPool and MaxPool—to split the visual data from a reduced set of tokens into low-frequency and high-frequency components. It then applies lightweight, learnable parameters to modulate these frequencies. The high-frequency data acts as a saliency filter to enhance key details, while the low-frequency data strengthens weaker, background semantics. This dual-action approach enables the model to preserve critical information even with far fewer visual tokens. The researchers integrated FMVR with Matryoshka Representation Learning, enabling the model to dynamically adjust the number of visual tokens used during inference based on available computational budget.

The results are striking. In experiments, their implementation, FMVR-LLaVA, applied to the popular LLaVA-1.5-7B model achieved an 89% reduction in FLOPs (floating-point operations), a direct measure of computational cost. Crucially, this massive efficiency gain came with almost no performance drop, maintaining close to 100% of the original model's accuracy across a rigorous evaluation of 10 image-based and 4 video-based benchmarks. The team has stated the code will be open-sourced, paving the way for more efficient and accessible multimodal AI.