3D-VCD: Hallucination Mitigation in 3D-LLM Embodied Agents through Visual Contrastive Decoding
New decoding framework improves robot reliability without retraining, tackling object and spatial hallucinations.
A team of researchers led by Makanjuola Ogunleye and Eman Abdelrahman has developed 3D-VCD, the first inference-time framework specifically designed to combat hallucinations in Large Multimodal Models (LMMs) that act as the 'brains' for embodied agents in 3D environments. Unlike existing methods for 2D images, 3D-VCD targets the unique failure modes of 3D reasoning—such as misidentifying object presence, misunderstanding spatial layouts, or making incorrect geometric assumptions—which can lead to unsafe and ungrounded actions by physical or virtual robots.
The core innovation is a contrastive decoding technique that operates on structured 3D scene representations. The system creates a distorted version of the agent's 3D scene graph by applying semantic perturbations (like swapping object categories) and geometric corruption (altering coordinates or sizes). By contrasting the AI's predictions under the original, correct scene context versus the distorted one, 3D-VCD identifies and suppresses token predictions that are insensitive to this grounded evidence, meaning they are likely driven by the model's internal language biases or 'priors' rather than the actual environment.
Evaluated on standard benchmarks like 3D-POPE and HEAL, the method consistently improved the grounded reasoning accuracy of 3D-LLM agents. Crucially, it achieves this as an inference-time intervention, requiring no expensive retraining or fine-tuning of the underlying large model. This establishes a practical and effective pathway toward more reliable embodied intelligence, where an agent's decisions are firmly anchored in its perceptual reality.
- Targets 3D-specific hallucinations in embodied agents, like object presence and spatial layout errors, not addressed by 2D methods.
- Uses contrastive decoding on perturbed 3D scene graphs to suppress language-prior-driven, ungrounded token predictions.
- Improves performance on 3D-POPE and HEAL benchmarks without any model retraining, offering a plug-and-play safety enhancement.
Why It Matters
Enables safer, more reliable AI robots and virtual assistants by ensuring their decisions are grounded in real-world 3D perception, not internal biases.