H2H Music Improv dataset gives AI a communication model for live improvisation

Current real-time AI improvisation systems treat communication as an afterthought, layering interaction strategies onto generative algorithms via explicit controls and predefined modes. This approach fails to capture the nuanced, spontaneous negotiation that human musicians use in free improvisation. To bridge this gap, researchers Aleksandra Teng Ma, Anthony Cammarota, Jiayi Wang, Alexandria Smith, Cheng-Zhi Anna Huang, Jeffrey Albert, and Alexander Lerch conducted a collaborative co-design process with expert improvisers, deriving a communication model that describes how free improvisers exchange musical ideas and transition into stable musical spaces. The model is formalized as a machine-readable annotation scheme, making it directly usable for AI systems.

Their companion H2H (Human-to-Human) Music Improvisation dataset is the first of its kind for free improvisation: six hours of audio-visual expert duo performances with clean per-player stems and per-player annotations of both the musicians' own intentions and their perception of their partner's intentions. This dual annotation provides a unique window into the interpersonal dynamics of improvisation. Published at ISMIR 2026, the model and dataset offer a foundational resource for studying musician communication and could inform the design of AI musical partners that communicate by design rather than by post-hoc control, opening new possibilities for live performance and human-AI creative collaboration.