New MGAP Method Reduces MLLM Hallucinations Without Sacrificing Coherence
Training-free decoding technique adaptively suppresses harmful language priors using subspace projection.
Multimodal Large Language Models (MLLMs) often hallucinate objects not present in visual inputs, typically due to over-reliance on language priors. Existing training-free decoding methods address this by blindly suppressing those priors, but they overlook that language priors can be both helpful and harmful depending on visual alignment. This blind suppression disrupts the model’s semantic manifold, a phenomenon the researchers term Manifold Departure, leading to degraded performance and coherence.
To solve this, the team introduces Manifold-Guided Adaptive Projection (MGAP). First, MGAP constructs a language-prior subspace from blind hidden states via SVD. During decoding, each multimodal hidden state is projected onto this subspace, and a consistency-aware gate adaptively attenuates only the projected prior component. This subspace-selective update preserves orthogonal semantic components. Extensive experiments on POPE (hallucination detection) and CHAIR (caption evaluation) show MGAP outperforms prior decoding baselines, achieving stronger hallucination suppression without sacrificing coherence. The method is training-free, geometry-aware, and will be presented at ICML 2026.
- MGAP is a training-free decoding method that selectively suppresses harmful language priors in MLLMs.
- It uses SVD to construct a language-prior subspace and a consistency-aware gate for adaptive attenuation.
- Outperforms prior baselines on POPE and CHAIR benchmarks, reducing hallucinations while maintaining coherence.
Why It Matters
More reliable MLLMs for vision-language tasks like captioning and VQA, enabling trustworthy AI in critical applications.