Starting Off on the Wrong Foot: Pitfalls in Data Preparation

A new research paper from Jiayi Guo, Panyi Dong, and Zhiyu Quan exposes a fundamental weakness in how AI models for insurance are built. The study demonstrates that conventional data preparation steps, especially random train-test splitting, produce unreliable and unstable results when applied to highly imbalanced insurance loss data. This common practice can undermine the entire statistical validity of downstream risk and pricing models, leading to poor performance in real-world, high-stakes scenarios.

To solve this, the team developed a novel, statistically rigorous data preparation framework. It leverages two key advancements: 'support points' for creating representative data partitions that maintain distributional consistency, and the non-parametric Chatterjee correlation coefficient for initial feature screening to better capture complex dependencies. They integrated these methods with missing-data handling into a unified pipeline called InsurAutoML.

Evaluation on both simulated and academic benchmark datasets showed definitive improvements. The proposed framework not only enhanced model robustness and interpretability but also delivered a substantial efficiency gain, reducing computational resource requirements. This work provides a crucial methodological upgrade, ensuring that the critical first step of data preparation no longer sabotages the reliability of AI-driven insurance applications.