PV-TAM sharpens VLMs by filtering answer-side attention drift

A new paper on arXiv proposes PV-TAM (Prompt-Vision Token Activation Map) to improve how large vision-language models (VLMs) align visual attention with semantic meaning. The authors—Yiyang Chen, Yixin Tan, and Binrui Shen—identify two key problems with existing attention-based methods: decoding drift (where language priors from previously generated answer tokens accumulate and mismatch with visual attention) and distortions from structural tokens like modality boundary markers, which can encompass the entire context and generate high attention to irrelevant areas.

PV-TAM sidesteps these issues by shifting the attention analysis to prompt-side tokens instead of answer-side ones. It introduces a filter to remove the systematic bias induced by modality boundary markers. Unlike traditional overlap metrics that rely solely on masks and ignore activation intensity, PV-TAM uses the peak distribution of attention to measure alignment between prompts and visual regions. Experiments show consistent improvements across multiple datasets on both attention-based and IoU-style localization metrics, offering a more reliable consistency evaluation for VLMs.