Study: Visualization researchers struggle with manual experiment stimuli management

A new study by Hyeok Kim and Jeffrey Heer, accepted to IEEE VIS 2026, provides the first in-depth look at how visualization researchers handle experiment stimuli. Through interviews with 19 researchers of diverse backgrounds, the authors map the entire lifecycle of stimuli—from exploration and selection to shipment, deployment, and analysis. They find that stimuli management is often a tedious, manual process that doesn't scale for experiments with many levels and complex conditioning. Researchers reported significant time wasted on manual inspection, version control, and ensuring stimuli quality, which can invalidate results if done poorly.

The study also surfaces mixed feelings about AI-assisted visualization experiment design. While some researchers see potential for automating stimuli generation and inspection, others worry about introducing biases or reducing experimental control. The authors call for future work on automated stimuli inspection, scalable deployment tools, and better experimental apparatus support. This paper highlights a critical bottleneck in visualization research—one that, if addressed, could accelerate scientific discovery and improve reproducibility.