Ground3D-LMM answers spatial queries with exact metric measurements
New model grounds 3D points and outputs real-world sizes, distances, and clearances.
Ground3D-LMM bridges a critical gap in 3D large multimodal models (LMMs): previous systems either answered conversational queries without explicit grounding or performed 3D grounding without interactive measurement capabilities. This new model integrates both, taking a point cloud and optional RGB image as input and producing responses that are both spatially grounded (pointing to the exact 3D region) and metric-aware (outputting real-world measurements such as size, thickness, clearance, and distance). It operates at object and part granularity and can handle multi-object queries, making it suitable for detailed scene understanding.
To train and evaluate this capability, the team created a large-scale dataset based on ScanNet and ScanNet++ with dense object and part annotations, containing roughly 2.5 million question-answer pairs spanning eight distinct tasks. A manually verified test set ensures reliability. Experiments across multiple datasets show that Ground3D-LMM provides a strong baseline for grounded, measurement-aware 3D conversational AI. The paper has been accepted at ECCV 2026, and both the dataset and model are publicly available, enabling further research in spatial reasoning for robotics, AR/VR, and 3D scene understanding.
- Unifies conversational AI with explicit 3D point grounding and metric measurements (size, thickness, clearance, distance) in a single model.
- Trains on a dataset of ~2.5M question-answer pairs across eight tasks, built on ScanNet and ScanNet++ with dense object/part annotations.
- Supports multi-object queries and part-level granularity, enabling precise spatial reasoning beyond simple object-level answers.
Why It Matters
Brings verifiable, measurable AI reasoning to 3D environments, enabling precise spatial understanding for robotics and AR.