AdaVFM: Adaptive Vision Foundation Models for Edge Intelligence via LLM-Guided Execution
New adaptive vision model cuts FLOPs by 77.9% while boosting accuracy on key benchmarks.
A research team led by Yiwei Zhao has introduced AdaVFM, a novel framework designed to solve a critical bottleneck in edge AI: deploying powerful, language-aligned vision foundation models (VFMs) on devices with strict power and latency constraints. The core innovation is a hybrid system where a lightweight model on the edge device is dynamically controlled by a more powerful, cloud-based multimodal large language model (LLM). This LLM acts as a context-aware agent, analyzing the visual scene and task complexity to instruct the edge model to execute only the most relevant, efficient sub-network from a larger pre-trained VFM backbone. This process, guided by neural architecture search (NAS), allows the system to adapt its computational footprint in real-time.
Extensive testing shows AdaVFM achieves a state-of-the-art accuracy-efficiency trade-off. On the ImageNet-1K zero-shot classification benchmark, it surpassed prior models of comparable size by up to 7.9% in top-1 accuracy. For the more demanding open-vocabulary segmentation task on ADE20K, it achieved a 5.2% mean Intersection over Union (mIoU) improvement. Most impressively, for models delivering similar accuracy, AdaVFM reduced the average computational cost (measured in FLOPs) by up to 77.9%. This dramatic efficiency gain comes from its key insight: the impact of shrinking a model is highly task-dependent, and an intelligent runtime controller can exploit this variability.
The framework represents a significant shift towards adaptive, agentic AI systems for the edge. By offloading complex reasoning about *how* to process a scene to a cloud LLM, the on-device component can remain extremely lean. This synergy enables versatile visual understanding—like identifying objects or segmenting scenes based on natural language queries—on resource-limited hardware such as smartphones, drones, or IoT sensors, paving the way for more sophisticated and responsive always-on contextual AI applications.
- Dynamically reduces compute by up to 77.9% FLOPs using LLM-guided sub-network selection.
- Boosts zero-shot accuracy by 7.9% on ImageNet-1K and segmentation mIoU by 5.2% on ADE20K.
- Uses a cloud-based multimodal LLM agent to analyze context and control a lightweight edge model.
Why It Matters
Enables powerful, language-aware computer vision on smartphones and IoT devices, making advanced AI practical for real-time applications.