New arXiv Paper Proposes Robust A/B Testing Under Model Misspecification

A new paper on arXiv (2605.12899) tackles a critical flaw in modern A/B testing: most experimental designs assume the statistical model is correctly specified, but real-world data often violates those assumptions. The authors—Qianglin Wen, Xiangkun Wu, Chengchun Shi, Ting Li, Niansheng Tang, Yingying Zhang, and Hongtu Zhu—introduce a robust sequential experimental design framework that explicitly handles model misspecification. Their approach unifies two previously separate domains: contextual bandit settings (where algorithms adaptively choose treatments based on user context) and dynamic settings (where treatment effects evolve over time). The key theoretical contribution is a proof that their design minimizes the worst-case mean squared error of the estimated treatment effect, providing a statistical guarantee even when the model is wrong.

To validate their approach, the researchers tested it on both synthetic data and real-world datasets from a leading technology company. The results demonstrate that the robust design maintains high sample efficiency and accurate treatment effect estimates under a range of misspecification scenarios, outperforming traditional methods that fail when assumptions break. This work has immediate practical implications for any organization running large-scale A/B tests—from product teams optimizing features to data scientists evaluating marketing campaigns. By ensuring reliable inference even with imperfect models, the framework can reduce costly errors and improve decision-making in dynamic, real-world environments.