Sparse2Act boosts robot manipulation with action-aligned 3D representations
86.9% success in 500 steps—cross-domain and sim-to-real transfer work.
Sparse2Act, developed by Yu Guo and colleagues, tackles a key bottleneck in robot manipulation: learning 3D representations that generalize across tasks, robots, and environments. Traditional sparse 3D encoders are tied to specific data distributions and action spaces. The innovation is to use task-space end-effector actions as self-supervised training signal—masked sparse 3D tokens are trained to organize scene features around the workspace motion paired with each observation. After pretraining, only the encoder initialization is reused by downstream policies, which can retain their own architectures and action parameterizations (including joint-space commands).
Results are striking. On LIBERO-10, Sparse2Act hits 86.9% average success in just 500 fine-tuning steps. The same pretrained encoder works across domains, achieving 73.4% on Meta-World-5. Ablations confirm the masked action-alignment objective drives gains. In real-world tests, simulation pretraining plus limited real-data fine-tuning yields 72.5% across four diverse tasks. This suggests robot actions themselves can provide compact geometric supervision for building reusable 3D representations—a step toward generalizable manipulation.
- Sparse2Act achieves 86.9% average success on LIBERO-10 after only 500 fine-tuning steps.
- The same encoder enables cross-domain transfer: 73.4% from LIBERO to Meta-World-5.
- Real-world sim-to-real transfer succeeds at 72.5% across four tasks with minimal real data.
Why It Matters
A reusable 3D perception backbone that adapts across robots and tasks, slashing retraining time.