Selective depth supervision boosts sparse-view neural reconstruction by 10%
Depth Anything V2 priors improve Splatfacto's PSNR from 14.9 to 15.9 in sparse driving scenes
A new study from Wei-Teng Chu, Yashasvini Gopalan, and Changju Yuan tackles a key challenge in 3D reconstruction: generating accurate neural representations from sparse camera views, such as those found in autonomous driving footage. The team uses Depth Anything V2, a monocular depth estimator, to produce dense geometric priors, then aligns them to metric depth with scale-shift fitting. Critically, they apply depth supervision only to pixels flagged as reliable by photometric masks derived from an RGB-only baseline model. This selective approach prevents noisy predictions from degrading the reconstruction.
On KITTISeq02 under an every-2-frame sparse-view setup, the method yielded modest improvements for Mip-NeRF-360 but significant gains for Splatfacto, a 3D Gaussian splatting representation. PSNR rose from 14.903 to 15.932 and RMSE dropped from 0.542 to 0.100—a 71% reduction in depth error. Additional experiments on KITTISeq05 and the Bicycle scene confirmed that the benefit comes from selecting low-error regions, not simply reducing supervision density. However, when multi-view overlap is already strong (e.g., Bicycle), depth supervision can hurt RGB rendering quality, suggesting practitioners should apply it cautiously and at moderate weight.
- Depth Anything V2 priors boosted Splatfacto's PSNR by 1.03 points (14.9 → 15.9) on sparse KITTI driving data
- Using photometric masks to filter unreliable depth regions reduced RMSE from 0.542 to 0.100 (71% improvement)
- Mip-NeRF-360 saw little benefit from the same supervision strategy, while Splatfacto improved clearly
Why It Matters
Selective monocular depth supervision improves sparse-view 3D reconstruction—critical for self-driving, AR, and robotics with limited camera overlap.