MIDI-RAE-JEPA: Self-supervised AI learns hierarchical musical structures

MIDI-RAE-JEPA, developed by Scott Hawley, is a novel self-supervised learning framework for symbolic music representation. It combines a pitch- and time-shift equivariance objective with the LeJEPA architecture and a Swin Transformer V2 encoder, processing symbolic music encoded as piano roll images. The model is trained purely on self-supervised tasks, including a masked embedding predictor (MEP), with collapse prevention via SIGReg. This approach encourages the encoder to internalize temporal musical relationships and hierarchical structures, enabling rich internal representations without any labeled data.

Results demonstrate impressive capabilities: a separate decoder trained on frozen encoder embeddings achieves a reconstruction F1 of 0.995, and a flow matching generative model conditioned on those embeddings produces generations that closely match the pitch register and rhythmic density of the conditioning excerpt. When conditioning is mismatched, the model still generates musically plausible but unrelated outputs. Learned representations also outperform a Haar scattering transform baseline on a downstream emotion classification task, and embedding distances increase monotonically with pitch and time shift magnitude, confirming measurable equivariance. These findings suggest that equivariance-based SSL objectives, combined with sufficient encoder capacity, offer a viable path toward semantically rich, generatively useful representations of symbolic music for applications like AI-assisted co-writing.