QPILOTS: New method steers flow policies at test time, 90% success on 50 tasks
Researchers from U of Toronto and NVIDIA achieve 90% success across 50 offline-to-online RL tasks without retraining.
Deep Dive
Researchers propose QPILOTS, a method that steers diffusion/flow-based policies at inference time without modifying the original model. By projecting noisy intermediate states to clean-action estimates and using critic gradients, QPILOTS avoids unstable backpropagation. Two variants—QPILOTS-U (fast single-point) and QPILOTS-M (learned auxiliary)—achieve 90% average success across 50 RL benchmarks and improve a frozen VLA robot foundation model.
Key Points
- QPILOTS avoids modifying the frozen policy; it steers the denoising process by projecting noisy intermediates to clean-action estimates for reliable critic gradients.
- Two variants: QPILOTS-U (fast single-point approximation) and QPILOTS-M (learned auxiliary network for differentiable posterior sampling).
- Achieves 90% average success rate across 50 offline-to-online RL tasks and improves a frozen VLA robot model on six simulated manipulation tasks.
Why It Matters
Enables efficient RL steering of large frozen diffusion policies, reducing retraining cost and unlocking scalable robot learning.