DiPOD: UC Berkeley's fix for drift in diffusion policy training

Reinforcement learning post-training is crucial for improving diffusion policies, but existing policy-gradient methods often suffer from instability. A new paper from UC Berkeley (including Pieter Abbeel, Angjoo Kanazawa, and others) identifies the root cause: the double-drift phenomenon. When optimizing a variational surrogate, the ELBO can drift away from the true log-likelihood, causing proxy gradients to misalign with the true policy gradient. The authors introduce DiPOD (Diffusion Policy Optimization without Drifting Apart), which maintains tight ELBO behavior throughout training. DiPOD interleaves self-distillation with standard policy-improving gradient updates and augments each step with an on-policy ELBO regularizer. This simple yet effective approach keeps the surrogate bound aligned with the true objective.

DiPOD is evaluated across two domains: post-training of diffusion language models and continuous-control diffusion policies. In both settings, it substantially stabilizes training—preventing the performance collapses seen in prior methods—while consistently reaching higher cumulative rewards. The framework is practical: it requires only a small modification to existing diffusion policy-gradient pipelines. By addressing a fundamental stability issue, DiPOD could make RL fine-tuning for diffusion models more reliable, potentially accelerating progress in text generation, robotics, and other areas where diffusion policies are applied.