New scaling law predicts VLM accuracy from text benchmarks across 7 model families

Choosing an LLM backbone for a vision-language model (VLM) has been a costly trial-and-error process. Compute-based scaling laws fail across model families, leaving teams to sweep through candidates empirically. In a new arXiv paper, Ziran Li and colleagues from Meituan propose the Capability-Driven Multimodal Scaling Law—the first cross-family framework that predicts VLM benchmark accuracy directly from observable textual capability. They apply PCA to LLM text-benchmark scores to extract a low-dimensional capability score S, then model VLM performance as a function of S, with per-backbone transfer rates and an absorption rate quantifying data-scaling efficiency.

The team validated the framework rigorously, training over 150 VLMs on 34 LLMs spanning 7 model families under a controlled recipe, testing on 200+ textual and 50+ multimodal benchmarks. The law accurately extrapolates from models up to 8B parameters to 72B-scale backbones, predicts full VLM training trajectories, and generalizes to held-out model families. Several actionable insights emerged: certain textual benchmarks negatively correlate with multimodal performance, exposing latent benchmark-gaming; base LLMs outperform instruction-tuned counterparts as VLM backbones due to higher absorption rates and lower data-scaling decay; and model families occupy distinct positions in the transfer–absorption space. The framework turns backbone selection from expensive empirical sweeps into a quantitative, principled decision—potentially saving teams significant compute and time.