PlayWrite: A Multimodal System for AI Supported Narrative Co-Authoring Through Play in XR

A research team from Autodesk and the University of Toronto has introduced PlayWrite, a novel mixed-reality system that fundamentally reimagines AI-assisted storytelling. Published on arXiv, the system addresses the limitation of current text-prompt-based AI writing tools, which fail to capture the spatial and interactive essence of how stories often emerge. Instead of typing commands, users author narratives by directly manipulating virtual characters and props within an extended reality (XR) environment. This physical play is then interpreted by an AI pipeline to create a co-creative writing experience.

The technical core of PlayWrite is a multi-agent AI pipeline that translates users' physical interactions into structured narrative components called 'Intent Frames.' These frames are visualized as rearrangeable 'story marbles' on a timeline, giving writers a tangible overview of their plot. A final large language model weaves the sequenced marbles into a cohesive narrative. In a user study with 13 writers, the system proved highly effective at fostering improvisation and play, with participants treating the AI as a true collaborative partner that could spark unexpected ideas. This research demonstrates a significant shift toward embodied, playful interaction as a core modality for human-AI co-creation, moving far beyond the chatbox.