PaLMR: Towards Faithful Visual Reasoning via Multimodal Process Alignment

A research team led by Yantao Li and Fang Zhao has introduced PaLMR (Process-Aligned Multimodal Reasoning), a novel framework designed to tackle a critical flaw in current multimodal AI models: process hallucinations. These occur when models like Qwen2.5-VL-7B arrive at a correct final answer but do so by misinterpreting or hallucinating the visual evidence in the reasoning chain. Traditional reinforcement learning rewards only final-answer correctness, which inadvertently tolerates these flawed internal steps. PaLMR directly addresses this by enforcing alignment not just on outcomes, but on the entire reasoning process itself.

The framework consists of two core, complementary components. First, a perception-aligned data layer constructs training data with structured pseudo-ground-truths and verifiable visual facts, ensuring the model has a reliable foundation for its analysis. Second, a process-aligned optimization layer implements a hierarchical reward fusion scheme. This includes a process-aware scoring function that evaluates and rewards each logical step in the model's chain-of-thought, encouraging visually faithful reasoning and improving overall training stability.

In experiments, PaLMR applied to the Qwen2.5-VL-7B model substantially reduced reasoning hallucinations and improved visual reasoning fidelity. It achieved state-of-the-art results on the HallusionBench, a benchmark specifically designed to test for such errors, while maintaining robust performance on other major benchmarks like MMMU, MathVista, and MathVerse. The findings, accepted for CVPR 2026, indicate that PaLMR offers a principled and practical path toward more reliable and interpretable multimodal AI systems where users can trust not just the answer, but the logic that produced it.