MeloDISinger edits sung lyrics while preserving melody and duration
A flow-matching model that rewrites vocals without changing the tune or timing
Researchers from KAIST have proposed MeloDISinger, a novel flow-matching-based model for text-based singing voice editing (SVE). The system allows users to revise sung lyrics while preserving the original melody, total duration, and all unedited regions of an audio track. Traditional SVE approaches often struggle with maintaining natural timing and melodic flow when words are changed—MeloDISinger tackles this by introducing a dedicated duration-control module and a generative audio infilling pipeline.
At the core of MeloDISinger is the MeloDRP module, which predicts fixed-budget duration ratios for each phoneme. By fusing phonetic cues with pseudo-MIDI melodic context through cross-attention, MeloDRP enables explicit span-wise duration control—ensuring the edited line fits the original melody's rhythm. A temporal-overlap supervision mechanism encourages soft phoneme–note correspondences, making the alignment more robust. The model then uses a flow-matching mel decoder to infill synthesized audio into the edited regions, seamlessly blending with the unmodified background. Experiments show state-of-the-art performance in both objective metrics (e.g., pitch accuracy, duration consistency) and subjective listening tests. The paper has been accepted to Interspeech 2026.
- MeloDISinger uses flow-matching to synthesize edited vocal regions while preserving original melody and total duration.
- The MeloDRP module predicts fixed-budget duration ratios via cross-attention between phonetic input and pseudo-MIDI melodic features.
- Achieved state-of-the-art results on objective and subjective evaluations; accepted to Interspeech 2026.
Why It Matters
Seamless text-based editing of vocals without losing musical timing—useful for music production, dubbing, and content creation.