RUTA uses 2% of visual tokens with 88% performance retention

High-resolution images and long videos give vision-language models rich context, but the resulting visual token sequences make LLM-side computation expensive. Existing token reducers often work at fixed rates or rely on learned thresholds. Now, a new approach from researchers including Jian Zou, Xiaoyu Xu, and Balu Adsumilli introduces RUTA (Rate-Utility Token Allocation), a principled method that learns both which tokens to retain and how many to allocate for each image-query pair. RUTA builds query-conditioned candidate tokens and predicts a retention probability for each one, trained via differentiable Bernoulli gates. Retained tokens act as anchors that aggregate information from non-retained tokens based on semantic affinity and spatial proximity, all optimized by a penalized rate-utility objective balancing task loss against expected token usage.

Across five benchmarks, RUTA delivers dramatic compression while preserving performance. Relative to full-token baselines, it uses only 2.0% of visual tokens on LLaVA-NeXT-7B and 4.2% on Qwen3-VL-8B, while keeping 88.2% and 94.4% of task performance, respectively. This translates to roughly 50x reduction in token count, directly cutting memory and latency for high-resolution and long-video multimodal reasoning. The method is model-agnostic, requiring no architectural changes, and is particularly compelling for production systems relying on image-heavy workloads. It addresses a key bottleneck in deploying large vision-language models in real-world applications, making them more economical and responsive.