Ryan's DEMON open-sources real-time AI music generation at 12.3 generations/sec

Ryan, a contributor to open-source generative audio, has released DEMON (Diffusion Engine for Musical Orchestrated Noise). Inspired by StreamDiffusion but for audio, it uses ACEStep 1.5 and achieves 12.3 generations per second of 60-second audio on an NVIDIA 5090, 8.9 on 4090, and 4.2 on 3090. Responsiveness is tunable via ringbuffer depth, with first-effect latency as low as 112ms. The system supports real-time remixing of songs, allowing adjustments to denoising, structure, timbre strength, reference track swapping, prompt blending with parameter curves, and LoRA hotswapping at runtime. It also includes a research preview of latent channel feedback and vocal stem cutting/pasting with Melformer. Built using TensorRT, mixed precision, and W8A8 quantization, DEMON is designed for interactive musical creation.

Despite its capabilities, DEMON has limitations. ACEStep correctly "begins" and "ends" the song, making it optimized for remixing entire songs or loops rather than continuous autoregressive generation. For pure continuous music, autoregression would be more suitable. Ryan trained around 200 LoRAs/DoRAs for ACEStep, to be released in batches. The project is open-source on GitHub with a demo app built by the Daydream team. For tech-savvy musicians and AI enthusiasts, DEMON represents a leap in real-time generative audio, enabling instrument-like interaction with AI. It matters because it brings diffusion-based audio generation into the live performance realm, previously dominated by slower batch processing.