FlowPRO: New reward-free RL fine-tuning for robot VLAs from arXiv

FlowPRO tackles the challenge of post-training Vision-Language-Action (VLA) models for reliable robot deployment. Traditional methods like supervised fine-tuning (SFT) or DAgger only indirectly exploit failure signals, while reward-based reinforcement learning struggles with real-world reward design and training reliable critics. The researchers propose FlowPRO, a reward-free offline reinforced fine-tuning framework designed specifically for flow-matching VLAs. At its core is RPRO (Robotic Flow-matching Proximalized Preference Optimization), a new preference-optimization objective that pairs a contrastive optimizer with an explicit proximal regularizer. This regularizer anchors the magnitude of the implicit reward, eliminating the reward-hacking failure mode that plagues plain Flow-DPO.

On the data side, the authors introduce a teleoperated intervention-and-rollback paradigm: a human operator teleoperates a robot, naturally generating paired positive and negative trajectories (τ^w, τ^l) from a single action. A Smooth Interpolation procedure combined with batch mixing then converts these sparse corrections into dense per-state supervision while preserving the base policy's capabilities. In experiments across four long-horizon bimanual tasks, FlowPRO achieved the highest success rate, outperforming four representative baselines. Ablation studies confirmed the contribution of each loss component. The work represents a practical step toward scalable robot learning without hand-crafted reward functions.