DP-DiT: 3M-param diffusion model matches 50M-param foley generators

Recent diffusion models can generate high-fidelity Foley sound effects directly in the waveform space, but they typically rely on heavy architectures like CNN-based U-Nets or frequency-domain Transformers, requiring large model capacities and substantial compute. In a new arXiv paper, Runwu Shi and colleagues from multiple institutions propose a Dual-Path (DP) architecture that performs dimension-wise self-attention along both subband and frame axes in the time-frequency domain. This design captures fine-grained temporal-spectral dependencies while maintaining high efficiency, addressing a gap in compact waveform diffusion models.

The team builds two variants on the DP backbone: DP-DiT and DP-U-Net. Evaluated on the DCASE and FSD-Kaggle2018 datasets, both variants deliver superior performance. Notably, the 3M-parameter configuration achieves results comparable to models exceeding 50M parameters, a >16x reduction in parameters. This makes high-quality Foley generation feasible for edge devices and real-time applications. Audio samples are available online. The work highlights how attention-based designs over time-frequency representations can replace large convolutional stacks for efficient audio synthesis.