OV3D-Bench exposes semantic bottleneck: 3D detectors collapse 3.5x on prompt tweaks

Open-vocabulary monocular 3D detectors—models that identify and localize objects in 3D from a single image without predefined classes—report strong results, but those results often rely on unrealistic evaluation protocols. Researchers from TU Munich and Carnegie Mellon built OV3D-Bench to fix that. It tests 7 detectors under deployment-realistic conditions across 7 indoor and outdoor datasets, replacing per-image category oracles with dataset-level prompts and decoupling accuracy into localization, semantic robustness, and cross-domain transfer.

The findings are striking: detectors localize objects well but frequently mislabel correctly placed boxes with semantically adjacent categories. Prompt phrasing is a hidden killer—WildDet3D's AP plummets from 18.6 to 5.4 when 'car' becomes 'a detailed high-resolution photo of a car.' The widely used target-aware protocol inflates DetAny3D's AP by 1.9x on ScanNet, hiding these failures. Strikingly, simply remapping a frozen closed-vocabulary detector's output using SigLIPv2 matches purpose-built open-vocabulary methods. That suggests geometric localization is largely solved, leaving open-vocabulary semantics as the primary bottleneck for real-world 3D perception.