ICML 2026 paper: Preregister AI agent experiments to fix hidden biases
AI agent studies have dangerous 'researcher degrees of freedom'—here's how to fix them.
The rise of LLMs and autonomous AI agents has spawned a new research paradigm: 'in silico' behavioral experiments, where AI proxies substitute for human participants. These studies promise unparalleled scalability and control, but as Michelle Vaccaro highlights in her ICML 2026 Spotlight paper, they also inherit—and worsen—classic methodological pitfalls. The low cost of iteration allows researchers to unknowingly (or knowingly) exploit degrees of freedom: from model selection and prompt tweaking to redesigning experiments based on outcomes. Without preregistration, these choices remain invisible, threatening the credibility of findings that increasingly inform real-world AI deployment.
Vaccaro systematically catalogs these vulnerabilities and proposes a standardized preregistration template tailored for AI agent experiments. The template forces researchers to pre-commit to model choices, prompts, settings, and analysis plans before data collection. The paper calls on top conferences, journals, and funding agencies to make preregistration mandatory for this emerging field. As AI agents negotiate, transact, and make decisions on behalf of people, ensuring that behavioral research on them is reproducible and trustworthy becomes a matter of safety and ethics—not just academic rigor.
- Paper accepted at ICML 2026 as a Spotlight (top 5%), signaling high impact.
- Identifies 5+ researcher degrees of freedom: model selection, prompt wording, settings, outcome-contingent redesign.
- Proposes a new, AI-agent-specific preregistration template and urges top conferences to standardize it.
Why It Matters
As AI agents make decisions for people, credible behavioral research becomes critical for safety, fairness, and regulatory trust.