Drift-Based Policy Optimization: Native One-Step Policy Learning for Online Robot Control

A research team from multiple institutions, led by Yuxuan Gao, has published a paper on arXiv introducing Drift-Based Policy Optimization (DBPO), a novel framework designed to solve a critical bottleneck in robotic AI. Current state-of-the-art methods, like diffusion policies, model complex, multimodal action distributions but require tens to hundreds of iterative network evaluations (NFEs) to generate a single action. This makes them too slow for high-frequency, closed-loop control and unstable for online reinforcement learning (RL), where policies must learn and adapt in real-time.

The proposed DBPO framework operates in two key stages to shift computational burden from inference to training. First, it trains a Drift-Based Policy (DBP) backbone using fixed-point "drifting" objectives. This process internalizes the iterative refinement typically done during a model's runtime directly into its parameters, creating a powerful, one-step generative model by design. Second, the team developed the DBPO algorithm, which equips this pretrained backbone with a specialized stochastic interface, allowing for stable, on-policy updates during online fine-tuning without breaking the one-step deployment property.

Extensive experiments validate the approach. The DBP matches or exceeds the performance of slower multi-step diffusion policies while achieving up to 100x faster inference. It also consistently outperforms other one-step baselines on challenging manipulation benchmarks. Crucially, DBPO enables effective online policy improvement, a notoriously difficult task for one-step models. The team demonstrated this on a real-world dual-arm robot, achieving reliable, high-frequency control at 105.2 Hz, a rate necessary for dynamic, real-time interaction.