Corani et al. reveal mean score best for short time series model selection

In a recent preprint (arXiv:2606.24715), Giorgio Corani, Stefano Damato, Dario Azzimonti, and Lorenzo Zambon tackle the challenge of selecting between probabilistic models when evaluating on multiple time series. They focus on three common aggregation methods for proper scoring rules: the mean score, the median score, and the mean rank across time series. Previous work noted these can lead to contradictory decisions—a problem the authors trace to the skewness of score distributions.

Using synthetic data and the M5 competition's intermittent demand time series, the team demonstrates that as the test set size grows, all three criteria converge toward the same optimal model. For short test sets (common in practice), only the mean score consistently picks the true underlying model. Additionally, model selection based on mean ranks proves insensitive to scaling factors—a potential advantage or pitfall depending on the context. The work underscores the importance of careful evaluation design and warns against relying on median-based or rank-based criteria when test data is limited.