DSSA: New method decouples appearance and identity for stable video object tracking
Solves slot swapping by separating what an object looks like from who it is
Unsupervised video object-centric learning aims to decompose dynamic scenes into persistent object representations, but existing slot-based methods suffer from slot swapping—where an object's identity jumps between slots during rapid motion or occlusion. The core problem: a single slot vector must simultaneously encode per-frame appearance (which changes with lighting, pose, etc.) and cross-frame identity (which should remain invariant), creating an objective conflict. Additionally, token renormalization in Slot Attention amplifies weakly attending slots, letting them absorb tokens from other objects.
DSSA solves this by decoupling each slot into a local state for per-frame appearance and an identity state for temporally stable object information. The identity state is updated via a learned recurrent transition that acts as a temporal filter on the local state. A competition-modulated aggregation (CMA) mechanism down-weights updates from weakly matching slots, preventing them from stealing tokens. Experiments on MOVi-C, MOVi-D, and YouTube-VIS show consistent improvements in segmentation quality and temporal consistency over prior methods, with stronger downstream object recognition and video dynamics prediction. Code and models will be released upon acceptance.
- DSSA separates appearance (local state) from identity (identity state) to eliminate slot swapping in video object-centric learning
- Competition-modulated aggregation (CMA) prevents weakly attending slots from absorbing tokens from other objects, stabilizing training
- Outperforms prior methods on MOVi-C, MOVi-D, and YouTube-VIS benchmarks for both segmentation quality and temporal consistency
Why It Matters
Enables AI systems to track objects consistently across video without supervision, improving autonomous driving, video understanding, and robotics.