Audio Imitator framework separates timbre and tempo for precise video audio style control
Forget holistic prompts—AI now tunes voice color and pace independently from silent video.
Video-to-audio generation has improved dramatically, but fine-grained control over stylistic attributes like timbre (voice color) and tempo (pace) remained elusive because reference audio was treated as a monolithic prompt. AudioIM, introduced in a new arXiv paper by Zhao et al., breaks this limitation. The system uses two specialized encoders—one for timbre-related features, one for tempo-related features—and injects them via global conditioning. A masking-based training strategy also enables the model to work with partial or latent audio prompts at inference, giving creators flexible control over the final sound.
Tested on the VGGSound dataset, AudioIM achieves higher style similarity while maintaining semantic consistency and temporal synchronization with the original silent video. The approach separates concerns that previous holistic methods muddled, making it possible to, for example, match the timbre of a specific speaker while adjusting the speaking rate independently. This could unlock new applications in video dubbing, virtual character voicing, and accessibility tools where both identity and delivery pace matter. Audio samples are available at the project page linked in the paper.
- Dual encoders separately extract timbre and tempo representations from reference audio instead of using a holistic prompt.
- Masking-based training enables latent prompt conditioning at inference without needing a full reference clip.
- Outperforms prior methods on VGGSound in style similarity while preserving semantic alignment and synchronization.
Why It Matters
Separate control over voice identity and pacing enables more natural video dubbing and character voicing without trade-offs.