STAC: Plug-and-Play Spatio-Temporal Aware Cache Compression for Streaming 3D Reconstruction

A research team led by Runze Wang has introduced STAC (Spatio-Temporally Aware Cache Compression), a novel framework designed to overcome a critical bottleneck in real-time 3D reconstruction. The problem stems from transformer-based models that use a key-value (KV) cache mechanism—this cache grows linearly with video stream length, creating unsustainable memory demands that force early data eviction and degrade reconstruction quality. STAC's breakthrough comes from observing that attention in these 3D reconstruction transformers exhibits inherent spatio-temporal sparsity, meaning most cached data is redundant.

STAC implements three core components to exploit this sparsity: a Working Temporal Token Caching mechanism that preserves only the most informative long-term tokens using decayed attention scores; a Long-term Spatial Token Caching scheme that compresses spatially redundant tokens into efficient voxel-aligned representations; and a Chunk-based Multi-frame Optimization strategy that processes consecutive frames together for better temporal coherence and GPU efficiency. The results are dramatic—accepted for CVPR 2026, STAC achieves state-of-the-art reconstruction quality while reducing memory consumption by nearly 10x and accelerating inference by 4x compared to existing methods.

This advancement fundamentally improves the scalability of streaming 3D reconstruction, a technology crucial for applications like autonomous vehicles, augmented reality, and robotics. By making real-time 3D scene building from continuous video feeds dramatically more efficient, STAC removes a major barrier to practical deployment of these AI systems in memory-constrained environments.