New AI Edits Music from Simple Text Commands
You could soon edit songs as easily as editing a document.
Researchers present ProCraft, a lightweight 0.5B-parameter autoregressive codec language model that edits 30-second music excerpts end-to-end from free-form instructions, including compound requests. It supports seven operations across eight instruments, from track removal to accompaniment generation.
To preserve content outside the edit, it aligns intermediate representations with pretrained music embeddings and pitch and instrument targets, and uses gated cross-attention in every decoder layer to keep generation tied to the instruction. Trained with On-the-fly Editing-pair Synthesis, which generates paired instructions and audio combining track-level operations with effect changes, ProCraft performs competitively against task-specific systems on single-task editing and outperforms four free-text editors on compound instructions for track removal, extraction, and denoising on real (MoisesDB) and synthetic (Slakh) recordings.
In a 16-listener user study, it was rated highest on removal and denoising and on par with a 16-times larger model on extraction and instrument transfer. Ablations show representation alignment and cross-attention reduce leakage from unwanted tracks and raise editing success rates, especially on out-of-domain data.
- ProCraft lets you edit music by typing instructions like 'remove the vocals' or 'add a guitar'.
- It can handle multiple edits at once, saving time compared to doing them one by one.
- The AI is small and efficient, but only works on 30-second clips for now.
Why It Matters
This could make music editing accessible to anyone, saving time and money on production.