RegimeVGGT speeds up 3D scene reconstruction 6.7x by removing layer redundancy

VGGT (Visual Geometry Grounded Transformer) recovers dense 3D scene structure from multi-view images in a single forward pass, but its quadratic cross-frame attention severely limits scalability on larger inputs. Existing training-free accelerators compress uniformly along one axis, ignoring layer-specific behavior—missing a key opportunity. Researchers from UPenn, UC Irvine, and NTU set out to fix this by first probing VGGT's internal dynamics through spectral, causal, and attention analyses.

Their findings reveal three distinct regimes: shallow layers lack cross-view structure, middle layers drive cross-view alignment, and deep layers are redundant for dense geometry but still require cross-frame attention for accurate pose estimation. RegimeVGGT applies a layer-wise U-shaped compression along two axes: Saliency-Guided Banded Merging protects geometry- and edge-salient tokens, while Selectively Protected K/V Downsampling preserves cross-frame spatial coverage and the pose-critical path through a phase-shifted spatial grid, a reference-frame anchor, and uncompressed camera/register tokens. The result: a 6.7x speedup over VGGT* with no loss in reconstruction quality, all without any fine-tuning or retraining. This makes high-resolution 3D scene reconstruction practical for real-time robotics, AR/VR, and autonomous driving applications.