ICML 2026 paper: RL researchers must stop conflating simulator scores with real-world performance

In a new position paper accepted at ICML 2026, researchers Vandergrift, Elelimy, and White argue that the reinforcement learning community needs to explicitly separate two fundamentally different research questions: solving simulators (achieving high scores in a specific simulated environment) and using simulators as a proxy for training agents that will later be deployed in real-world settings. The distinction matters because the constraints, algorithmic choices, and evaluation metrics differ drastically. For example, solving a simulator often allows unlimited resets and direct manipulation of the environment, while using a simulator as a proxy requires algorithms that can handle distribution shift, limited data, and partially observable dynamics—much like real deployment. The paper demonstrates through simple experiments how conflating these goals can lead to algorithms that look impressive in benchmarks but fail when transferred to the real world. The authors call for a community-wide shift in empirical practices, urging researchers to explicitly state which use case they are addressing and to adopt evaluation protocols tailored to that setting. This work sparks a necessary conversation about the rigor and reproducibility of RL experimentation.