New framework uses audio-aware LLMs to improve text-to-audio instruction following

Researchers have developed a novel method to improve text-to-audio models, which often struggle to follow instructions involving multiple sound events and temporal sequences. The team, including contributors from Meta, proposes using audio-aware large language models (ALLMs) as fine-grained judges that verify whether target events and their temporal relationships are correctly present in generated audio. They validated ALLM judgments on existing benchmarks and through human verification, then used the feedback to construct preference pairs for direct preference optimization.

The approach introduces S3Bench, a narrative benchmark specifically designed to evaluate multi-event temporal instruction following. Experiments show the method significantly improves event completeness, temporal ordering, and joint instruction-following accuracy across existing benchmarks and S3Bench, all while preserving audio quality. The work was accepted as a long paper at Interspeech 2026 and addresses a critical gap in current text-to-audio generation systems.