Sample Size Calculations for Developing Clinical Prediction Models: Overview and pmsims R package

A research team from King's College London and University College London has published a groundbreaking paper and released pmsims, an open-source R package that addresses one of the most persistent challenges in clinical AI development: determining adequate sample sizes for prediction models. The work, led by Diana Shamsutdinova and six co-authors, introduces a novel simulation-based framework that moves beyond traditional heuristic rules and closed-form formulas, which often fail with complex machine learning models and data structures. The researchers propose distinguishing between mean-based criteria (targeting average performance) and assurance-based criteria (achieving performance with high probability), with pmsims implementing the latter through an innovative combination of learning curves and Gaussian Process optimization.

The pmsims package represents a significant methodological advancement by providing model-agnostic, computationally efficient sample size calculations that accommodate user-defined performance metrics and explicitly account for variability in model performance. Through case studies, the team demonstrates how sample size estimates vary substantially across methods, performance metrics, and modeling strategies, highlighting the limitations of existing approaches. The software's flexibility allows researchers to avoid common pitfalls like overfitting and poor generalizability that result from inadequate sample sizes. Future development will extend these methods to hierarchical and multimodal data, incorporate fairness and stability metrics, and address challenges like missing data and complex dependency structures, potentially transforming how clinical AI models are developed and validated.