MultiView-Bench reveals VLMs can't integrate 3D views; ViewNavigator boosts 3-5x

A team of researchers including Hantao Zhang and Zhuoran Yang from institutions such as Yale and Princeton have released MultiView-Bench, a novel diagnostic benchmark designed to evaluate vision-language models (VLMs) on their ability to integrate observations from multiple viewpoints into a coherent, allocentric (world-centric) 3D mental model. Unlike existing benchmarks that focus on single-image tasks or camera-relative navigation, MultiView-Bench forces models to decouple object positions from transient perspectives and ground them in a fixed global coordinate system—a prerequisite for real-world tasks like mechanical part assembly.

Evaluations of frontier VLMs reveal consistent failure modes: strong performance on 2D planar relations from single images, but marked difficulty with 3D spatial reasoning and aggregating information across views. Further analysis shows biases such as struggles with unconventional axis directions and sensitivity to object colorways and texture variations. To address these limitations, the authors propose ViewNavigator, a multi-agent framework that actively selects informative viewpoints, perceives, and fuses multi-view evidence. In budget-matched comparisons, ViewNavigator improves diverse base models by up to 5x on the MultiView-Bench, suggesting a promising path toward more robust 3D understanding in VLMs.