Dynamics models pick RL hyperparameters in first real-world water plant trial

Hyperparameter selection remains a major bottleneck for deploying reinforcement learning in real-world systems, especially when simulators are unavailable and online experimentation is costly. Previous work proposed calibration models trained on offline data to approximate environment dynamics, enabling offline hyperparameter selection, but these were only tested in simple simulated settings. In this paper, accepted to the 2026 Reinforcement Learning Conference, Jordan Coblin, Han Wang, Martha White, and Adam White present the first industrial application of these calibration models: a municipal water treatment plant. They evaluate several approaches, including a k-nearest neighbors model with a Laplacian distance metric, on high-dimensional, non-stationary sensor data for nexting prediction tasks.

The results show that the models can generate realistic long-horizon rollouts and recover meaningful hyperparameter sensitivity trends, effectively enabling offline tuning without costly online trials. The researchers also examined scalability to year-long datasets, the selection of fine-tuning learning rates for pre-trained agents, and robustness under distribution shift. Overall, this work provides a proof of concept for using offline dynamics models to support RL deployment in real-world environments, while highlighting practical challenges such as non-stationarity and the need for careful metric design. It's a significant step toward making RL viable in safety-critical industrial settings where trial-and-error learning is impractical.