Tune to Learn: How Controller Gains Shape Robot Policy Learning

A team from MIT's Improbable AI Lab, led by Pulkit Agrawal with researchers Antonia Bronars and Younghyo Park, has published a foundational study titled 'Tune to Learn' that challenges a core assumption in robot learning. The conventional approach to tuning robot controller gains—parameters that determine how aggressively a robot moves to reach a commanded position—has focused on achieving desired task stiffness or compliance. The new research argues this logic breaks down when controllers are paired with learned, state-conditioned policies, as the effective stiffness emerges from the complex interplay between the policy's reactions and the control dynamics.

Through systematic experiments across three major learning paradigms—Behavior Cloning (BC), Reinforcement Learning (RL) from scratch, and Sim-to-Real transfer—the team discovered that controller gains should be selected primarily for 'learnability.' They found that Behavior Cloning performs best with compliant and overdamped (slower, less oscillatory) gain settings. In contrast, Reinforcement Learning algorithms, given proper hyperparameter tuning, can succeed across a wide range of gain regimes. Most critically for real-world deployment, they demonstrated that stiff and overdamped gains actively harm the success of transferring policies from simulation to physical robots.

These findings have immediate practical implications for roboticists. The study effectively re-frames controller gain selection from a task-specific engineering decision to a core hyperparameter in the machine learning pipeline. The optimal setting is not about the final desired robot behavior but about which setting makes the policy easiest to train effectively with a given algorithm. This provides a data-driven guide to accelerate training convergence and improve the reliability of learned policies, especially for sim-to-real transfer which is crucial for cost-effective robot training.