DVPSFormer Unifies Depth, Segmentation, Tracking in One Real-Time Model

Safe autonomous navigation demands real-time understanding of dynamic environments—metric depth, semantic classes, and instance trajectories. Existing approaches use computationally expensive multi-stage pipelines or offline tracking, limiting real-time use. To solve this, researchers from ETH Zurich, UC Berkeley, and other institutions introduce DVPSFormer, an efficient online architecture for 4D scene understanding. The key innovation is explicit scene discretization (ESD), which uses segmentation queries to represent both foreground and background regions. This enables a discrete-to-continuous (D2C) depth head to decode metric depth in a single forward pass, tightly coupling semantic and geometric learning while reducing latency drastically.

In addition, DVPSFormer employs an online majority voting (OMV) mechanism that leverages temporal consistency across frames to refine instance classification during tracking, eliminating the need for offline post-processing. The model achieves state-of-the-art results on Cityscapes-DVPS and SemKITTI-DVPS benchmarks, demonstrating superior performance in both segmentation and depth estimation. By unifying depth perception, semantic segmentation, and instance tracking in one streamlined online framework, DVPSFormer makes real-time holistic perception feasible for self-driving cars and robotic platforms, moving closer to safe autonomous operation.