Scientists Can't Repeat AI Experiments, And That's a Big Problem

Scientists often use AI programs to simulate human societies—for example, to study how people cooperate or form conventions. But a new paper shows that most of these simulations are documented so poorly that other researchers can't reproduce them. In fact, when the authors tried to recreate a published simulation, they found that a single missing detail—whether the AI was told to act as a player or as an outside observer—completely changed the outcome. The AI's behavior shifted from mostly one pattern to always another, and it reached that pattern three times faster.

The problem is that AI simulations involve many hidden choices: how prompts are written, how the AI's memory is handled, and how its responses are interpreted. These details are rarely reported, making it impossible to check or build on the work. The authors reviewed 28 published simulations and found that 77% left out crucial details about how the AI was prompted and how its outputs were processed.

To fix this, they propose ODD+LLM, an extension of an existing standard for describing simulations. It adds five new sections to document exactly how the AI is set up and used. This is like adding a detailed recipe to a scientific paper, so others can follow it precisely.

Why does this matter to you? AI is increasingly used to make decisions and study human behavior. If the research behind it can't be reproduced, we can't trust the results. This new standard aims to make AI research more transparent and reliable, which ultimately leads to better AI tools and policies.