Lipschitz Budget K Governs Diffusion Policy Expressivity and Statistical Trade-offs

Diffusion-based policies have emerged as powerful tools for representing complex, multimodal action distributions in reinforcement learning, but until now the mathematical drivers of their expressivity and the trade-offs when learning from finite data were unclear. In a new preprint on arXiv, researchers Viet Vu, Renyuan Xu, Jiacheng Zhang, and Yufei Zhang identify the drift Lipschitz budget K as a central quantity governing both expressivity and statistical behavior. They show that diffusion policies with K-Lipschitz drifts can approximate optimal deterministic policies with a value approximation error on the order of 1/K, and prove a matching lower bound under nondegenerate noise.

Increased expressivity comes with a statistical cost: when drifts are parameterized by neural networks, higher K improves approximation but raises statistical complexity. Balancing these terms yields a finite-sample performance gap of order ~O(n^{-2/(m+6)}) for generic neural-network drifts, and a sharper ~O(n^{-2/(m+4)}) for one-sided dissipative drift classes, where n is sample size and m is state dimension. Numerical experiments confirm this sample-dependent trade-off, and the authors propose a practical implementation principle: choose the diffusion budget K according to available sample size, then select a neural-network architecture with the corresponding fixed Lipschitz coefficient.