Q-VGM boosts robot success rates 17-27% with value-gradient matching

Researchers from an undisclosed team (Ziqian Wang et al., arXiv 2026) introduced Q-VGM, a novel off-policy reinforcement learning approach for fine-tuning flow-matching vision-language-action (VLA) policies. The core challenge: flow-matching policies are highly expressive but notoriously difficult to improve with a learned Q-function because backpropagating through the multi-step denoising process is numerically unstable at scale, and policy-gradient methods lack tractable action likelihoods. Q-VGM sidesteps this by leveraging VGG-Flow, which converts the value gradient into a denoising-time value-gradient field—avoiding end-to-end backpropagation entirely. The critic is an action-sensitive Cal-QL ensemble over compact RLT features with per-layer action injection, enabling efficient learning from a fixed replay buffer.

In experiments, Q-VGM demonstrated dramatic improvements across multiple benchmarks. On LIBERO, average success rate rose from 75.0% to 92.5%; on RoboTwin 2.0 from 76.4% to 87.2%; and on two real-robot tabletop tasks from 40.0% to 67.5%, outperforming all same-backbone, same-critic baselines. The method follows a practical few-shot initialization then learn-from-experience paradigm: starting from a few-shot-SFT pi0.5 VLA, it uses self-generated rollout data to improve without additional expert supervision. This breakthrough could accelerate the deployment of robust, adaptable robot policies in real-world environments where expert demonstrations are scarce.