Sonic4D: Spatial Audio Generation for Immersive 4D Scene Exploration

A research team from multiple institutions has introduced Sonic4D, a groundbreaking framework that addresses a critical gap in 4D scene generation—the lack of synchronized spatial audio. While recent AI models have excelled at creating photorealistic dynamic 3D scenes (4D) from monocular videos, they have universally overlooked sound, limiting immersion. Sonic4D proposes a novel, training-free solution that generates realistic 3D audio aligned with visual events, moving beyond simple stereo to audio that changes based on a user's virtual position and the scene's physics. This marks a significant step toward complete audiovisual synthesis for virtual and augmented reality.

The technical pipeline operates in three key stages. First, it uses pre-trained expert models to generate both the 4D visual scene and a basic monaural audio track from a single video. Second, it employs a pixel-level visual grounding strategy to localize and track sound sources within the 3D space over time, estimating their precise coordinates. Finally, it uses physics-based simulation to synthesize plausible spatial audio that varies across different viewpoints and timestamps. Extensive experiments show the method produces audio consistent with the visual scene, enabling truly immersive exploration where sounds appear to come from specific locations, like a bouncing ball or a moving car, enhancing the sense of presence in virtual environments.