SpaR3D-MoE achieves 3D spatial reasoning from sparse RGB views with MoE

Current multimodal LLMs struggle to bridge 2D semantic understanding with 3D spatial geometry. Existing approaches either require costly 3D-specific training data or rely on heuristic sampling and monolithic fusion that disrupts spatiotemporal connectivity and induces modality contention. To solve this, Haida Feng and colleagues propose SpaR3D-MoE, an end-to-end framework that equips MLLMs with geometry-aware capabilities using only sparse RGB inputs. The system first employs an adaptive spatiotemporal manifold sampling mechanism that constructs a graph over frames, extracting informative keyframes while preserving topological connectivity and reducing redundancy. Second, a heterogeneous geometry-inductive Mixture-of-Experts (MoE) driven by an instruction-pose aware router adaptively routes multimodal tokens to specialized experts, resolving cross-modal contention inherent in monolithic fusion.

Extensive experiments on VSI-Bench, ScanQA, and SQA3D demonstrate state-of-the-art performance. Notably, SpaR3D-MoE achieves an average score of 63.5 on VSI-Bench, outperforming the strongest baseline by 7.8 absolute points, with relative improvements of 35.4% on Route Plan tasks and 51.4% on Relative Direction tasks. The method is accepted to ECCV 2026, highlighting its impact on computer vision and AI for 3D reasoning from minimal input.