New Semantic Communication Framework Boosts Real-Time Mobile 3D Reconstruction
Pioneering method uses confidence maps to preserve geometric accuracy under noisy channels.
Real-time mobile 3D reconstruction is critical for applications like autonomous navigation and digital twin construction, where a moving platform captures images and transmits them to a server for scene understanding. Unlike offline reconstruction, camera poses and geometry are estimated on-the-fly, making multi-view consistency a real-time requirement and rendering geometric estimation highly sensitive to communication-induced distortions. Existing semantic communication (SemCom) designs operate at the image or single-view level without providing explicit reliability information for geometric estimation, limiting their utility.
The proposed framework introduces a semantic transceiver that outputs both the reconstructed image and a pixel-wise confidence map, quantifying reliability per region. This confidence map is integrated into RANSAC-based pose initialization and bundle adjustment to down-weight unreliable regions, enhancing robustness under noisy channels. Simulations demonstrate that the approach maintains high image quality while significantly improving pose estimation accuracy and 3D structural consistency compared to prior SemCom and separate source-channel coding methods.
- Framework includes a semantic transceiver that outputs pixel-wise confidence maps for reliability quantification.
- Confidence-guided RANSAC and bundle adjustment reduce influence of unreliable regions under noisy channels.
- Simulations show improved pose estimation accuracy and 3D structural consistency over existing methods.
Why It Matters
Enables robust mobile 3D reconstruction in real-time, critical for autonomous navigation and digital twins.