Beyond Dominant Patches: Spatial Credit Redistribution For Grounded Vision-Language Models

A research team led by Niamul Hassan Samin has published a paper introducing Spatial Credit Redribution (SCR), a novel method to combat a core failure mode in vision-language models (VLMs) like LLaVA and Qwen-VL. The researchers identified that VLMs frequently hallucinate objects because of 'spatial credit collapse,' where early transformer layers focus activation on just a few dominant image patches, ignoring broader visual context and over-relying on language priors. SCR is a training-free, inference-time intervention that addresses this by intelligently redistributing hidden-state activation from these high-attention source patches to their surrounding context, guided by low-entropy inputs.

The technique was rigorously evaluated across six VLM families (including Chameleon, LLaVA, and Qwen models at 7B, 13B, and 30B scales) on standard benchmarks. Results show SCR reduces object hallucination by ~4.7-6.0 percentage points on the challenging POPE-Adversarial benchmark and achieves a 42-51% relative reduction on the CHAIR-s metric. Crucially, it preserves the model's descriptive capability (CIDEr score) within 0.8 percentage points. With an overhead of just 43-56 ms—3-6x lower than comparable methods like OPERA and VCD—SCR Pareto-dominates alternatives, making it highly practical for real-time applications. An ablation study confirmed the mechanism, showing that random patch selection instead of attention-guided selection slashes the hallucination reduction gains by nearly half, proving credit collapse is the key driver of the problem.