New LEGO model builds 3D scene hierarchies for LLMs
LEGO breaks 3D scene understanding with hierarchical semantic reasoning
A team of researchers from Wuhan University, China Geological Survey, and other institutions has introduced **LEGO: Leveled Language Gaussian Splatting**, a groundbreaking approach for open-vocabulary 3D scene understanding that goes beyond traditional object recognition.
LEGO uniquely captures intrinsic semantic hierarchies within scenes—such as resolving a flowerpot into a bouquet, then into buds, and finally into individual petals—using a self-adaptive re-grading mechanism that unifies multi-view segmentation outputs from models like SAM into a single, 3D-consistent hierarchy. By grounding these segments with CLIP embeddings and incorporating spatial relationships, LEGO constructs level-wise language scene graphs. This enables large language models to perform complex, context-aware spatial reasoning and precise visual grounding.
The method achieves new state-of-the-art results on both promptable and open-vocabulary 3D segmentation benchmarks, demonstrating superior hierarchical decomposition and reasoning capabilities. The team has also released a project page and is preparing for presentation at ECCV 2026.
- LEGO builds hierarchical 3D scene understanding (e.g., flowerpot → bouquet → petal) using self-adaptive re-grading of multi-view SAM outputs
- Converts 3D segments into language scene graphs using CLIP embeddings and spatial relationships, enabling LLMs to perform context-aware spatial reasoning
- Outperforms state-of-the-art on 3D segmentation benchmarks; accepted to ECCV 2026
Why It Matters
LEGO enables AI systems to understand and reason about complex 3D environments with human-like semantic depth, unlocking advanced spatial AI applications