HeRA: New method aligns attention heads to slash MLLM hallucinations

Representation alignment has emerged as a key technique to improve Multimodal Large Language Models (MLLMs) by regularizing their internal representations toward those of an external vision encoder. However, existing methods typically align a fixed layer of the language backbone, overlooking the fine-grained structure of Transformer models. In this work, Davide Caffagni and colleagues propose Head-Wise Representation Alignment (HeRA), a method that enforces cross-modal alignment at the level of individual attention heads. Their approach is grounded in the Platonic Representation Hypothesis, focusing on preserving the topological structure of representations—local neighborhood relationships—across modalities. Using the Mutual K-Nearest Neighbor (MKNN) alignment metric, they introduce a contrastive objective that acts as a differentiable proxy for matching local structures. HeRA applies this objective during multimodal training to specific attention heads in the LLM, selected by their alignment score according to the MKNN metric. Counterintuitively, the authors find that aligning the least aligned heads yields the largest gains.

Extensive evaluations across multiple MLLMs and 18 benchmarks demonstrate that HeRA consistently improves performance on challenging vision-centric tasks and serves as an effective regularizer against visual hallucinations. The method naturally curbs the over-reliance on linguistic priors that often causes models to hallucinate. The findings highlight that attention head granularity matters more than previously thought for multimodal alignment. The code has been publicly released, allowing the community to build on this work. By focusing on the least aligned heads, HeRA efficiently reduces cross-modal discrepancies without excessive computational overhead, making it a practical addition to existing multimodal training pipelines.