STRUM: AI turns any song into playable Guitar Hero charts

STRUM (Spectral Transcription and Rhythm Understanding Model) is a groundbreaking open-source pipeline that turns any raw audio recording into fully playable rhythm-game charts for drums, guitar, bass, vocals, and keys — no metadata or manual annotation required. Developed by Joshua Opria and built for the Clone Hero and YARG communities, it combines a two-stage CRNN onset detector with a six-model ensemble for drums, neural onset detectors plus monophonic pitch tracking for guitar/bass, word-aligned ASR for vocals, and spectral keyboard detection for keys. The system was evaluated on a carefully curated 30-song benchmark screened for audio quality.

Results show strong performance: drums onset F1=0.838, bass F1=0.694, guitar F1=0.651, and vocals F1=0.539 at a ±100ms tolerance with per-song global offset optimization. The paper also includes a thorough ablation study of seven drum-pipeline components using paired Wilcoxon tests, a timing distribution analysis of community charts, and a per-class confusion matrix for drums. By releasing code, model weights, and the full benchmark manifest, STRUM democratizes chart creation — letting anyone turn any song into a playable rhythm-game experience without manual transcription.