MMAG benchmark exposes gaps in mixed audio generation with 4,000 tested clips

A team led by Zihao Zheng at Shanghai Jiao Tong University released MMAG, the Multi-control Mixed Audio Generation benchmark, designed to evaluate AI systems that generate complex audio scenes combining speech, music, and sound effects. The benchmark contains roughly 4,000 human-verified audio clips with rich annotations covering speech content, speaker identity, music attributes, sound events, and temporal relationships. It also includes dedicated subsets for voice cloning and timestamp-conditioned generation, pushing beyond existing benchmarks that focus on isolated domains or coarse-grained descriptions.

Using a systematic evaluation protocol measuring acoustic fidelity, speech quality, semantic alignment, and temporal accuracy, the researchers tested representative agentic orchestrators, unified audio-visual generation models, and native mixed-audio generators. Results show substantial performance trade-offs across capabilities—no existing model performs consistently well. For example, some models excel at semantic alignment but struggle with speaker consistency, while others handle temporal control but sacrifice acoustic fidelity. The authors frame MMAG as a comprehensive benchmark to spur future research into controllable mixed audio generation, particularly highlighting voice cloning and timestamp-conditioned tasks as open challenges.