SCORE framework boosts real-world robot manipulation success from 38% to 90%

Robots trained on real-world data are often slow, imprecise, and brittle to perturbations. Reinforcement learning (RL) could improve them, but real-world RL is expensive and risky. Simulation provides a cheap alternative, but unconstrained RL in simulation exploits contact and dynamics mismatches, leading to unsafe behaviors that don't transfer back to hardware. Standard regularization overconstrains to imperfect priors, limiting improvement.

To solve this, the authors introduce SCORE (Support-Constrained Off-Domain REinforcement). The key insight: constrain simulation-based RL to only explore actions that the base real-world policy can already produce. They implement this via flow steering, which restricts the policy to the support of a generative pretrained policy. This ensures transferable behaviors while maximizing improvement. SCORE learns from sparse rewards, avoids distillation, and leaves the base policy untouched. On eight real-world dexterous multi-fingered manipulation tasks, SCORE raised average success from 37.8% to 89.9% (best baseline scored 59.5%) and achieved success in 36.8% fewer steps. This demonstrates a new paradigm for real-to-sim-to-real policy improvement.