CanvasAgent and CanvasCraft enable AI to orchestrate visual tools for complex image creation

Complex image creation often requires combining multiple tools—synthesis, localization, segmentation, editing, compositing, text reading, and enhancement. Existing multimodal agents focus on perception or single-domain editing, lacking supervision for multi-step image-creation trajectories. To address this, researchers from multiple institutions developed CanvasCraft, a large-scale dataset with 140K fully annotated executable trajectories and 10K RL task specifications, and CanvasAgent, a tool-augmented multimodal agent that learns to orchestrate these tools through multi-turn interaction.

CanvasAgent is first trained with supervised fine-tuning (SFT) on executable reasoning-action trajectories, then optimized using GRPO with a hybrid reward combining outcome- and process-level signals. During rollout, the agent inspects intermediate results, tracks visual assets, and adapts tool decisions to the evolving state. Experiments demonstrate superior final image quality and trajectory behavior, marking a shift from perception-augmented reasoning to manipulation-centered visual creation. This work advances AI's ability to handle complex, multi-step creative workflows.