ConVBench & ConVLM: New Benchmark and RL Method Improve LVLM Logical Consistency

Large Vision-Language Models (LVLMs) excel at perception but often fail at logical consistency in visual reasoning. Existing benchmarks focus on symbolic math or simple vision tasks, leaving a gap in evaluating complex reasoning. To address this, researchers from multiple institutions created ConVBench, a benchmark where each image is paired with two logically equivalent questions across six categories: action and state, complex counting, spatial reasoning, causal and intent understanding, commonsense reasoning, and temporal perception. Two metrics, logical consistency and robust accuracy, jointly measure correctness and consistency, offering a more rigorous test of whether models truly understand visual scenarios.

To boost performance, the team developed ConVLM, which applies Group Relative Policy Optimization (GRPO)-based reinforcement learning with a novel consistency reward. It uses automatically generated logically equivalent QA pairs and a dual-reward design combining accuracy and consistency signals, encouraging agreement between paired responses. The framework functions effectively with or without strict answer supervision, making it adaptable to various data settings. ConVLM represents a significant step toward reliable LVLMs for high-stakes applications like autonomous driving and medical imaging.