NAIMA: Semantics Aware RGB Guided Depth Super-Resolution
New vision model uses DINOv2's semantic understanding to fix blurry depth boundaries in 3D reconstruction.
A research team has introduced NAIMA (Neural Attention for Implicit Multi-token Alignment), a novel AI architecture that significantly improves the quality of depth map super-resolution. The system tackles a core problem in guided depth super-resolution (GDSR), where a high-resolution RGB image is used to enhance a low-resolution depth map. Traditional methods often fail because color and texture cues in the RGB image can be misleading, causing artifacts and blurry object boundaries in the final 3D depth map. NAIMA solves this by injecting global semantic context, using knowledge distilled from the powerful, pretrained DINOv2 vision transformer.
The key innovation is the Guided Token Attention (GTA) module, which iteratively aligns encoded RGB spatial features with depth encodings using cross-attention. This allows the model to selectively incorporate semantic understanding—like recognizing object edges and surfaces—from different layers of DINOv2. By understanding the scene's semantics, NAIMA can ignore misleading visual textures and focus on true geometric boundaries. The architecture demonstrates substantial performance gains over previous state-of-the-art methods on standard benchmarks, enabling cleaner, more accurate upscaling of depth maps for applications in robotics, augmented reality, and 3D reconstruction.
- Uses DINOv2 vision transformer embeddings to provide global semantic priors, preventing artifacts from misleading RGB textures.
- Introduces a novel Guided Token Attention (GTA) module for iterative, cross-modal alignment between RGB features and depth encodings.
- Achieves significant quantitative improvements over existing GDSR methods across multiple scaling factors (e.g., 4x, 8x) and datasets.
Why It Matters
Enables higher-fidelity 3D perception for robots, AR/VR, and autonomous systems by producing sharper, more accurate depth maps from limited sensor data.