MIDI-Informed Singing Accompaniment Generation in a Compositional Song Pipeline

A research team from Academia Sinica and National Taiwan University has introduced a novel compositional pipeline for AI song generation that addresses key limitations of current end-to-end models. Their approach, detailed in the paper "MIDI-Informed Singing Accompaniment Generation in a Compositional Song Pipeline," breaks down the complex task of creating full songs (with both vocals and accompaniment) into three manageable components: melody composition, singing voice synthesis, and singing accompaniment generation. This modular design specifically tackles the data/compute intensity and limited editability of existing systems, offering a more practical alternative for music production.

The core innovation is MIDI-SAG (MIDI-Informed Singing Accompaniment Generation), which conditions the instrumental backing track on symbolic vocal-melody MIDI data to ensure precise rhythmic and harmonic alignment between the singing and the accompaniment. The system uniquely handles intermittent vocals (common in real songs) by combining explicit rhythmic/harmonic controls with audio continuation techniques, maintaining musical consistency across vocal and instrumental sections. Remarkably efficient, the pipeline's newly trained components required only 2.5k hours of audio data and were trained on a single RTX 3090 GPU, yet achieve perceptual quality competitive with recent open-source end-to-end baselines. The team plans to open-source the model, potentially democratizing high-quality AI music production.