New framework picks 5 datasets to replace 112 with 95% ranking accuracy

Evaluating machine learning models on massive benchmarks—like 112 datasets for time series classification or 57 tasks from MTEB—is expensive and slow. A new paper from Rostislav Gusev and Alexey Zaytsev (accepted at KDD 2026) tackles this by introducing a systematic framework to select a small, representative subset of datasets that preserves the original model ranking. The authors compare several selection strategies: clustering, A/D-optimality, random baselines, and greedy farthest-first (FAFI). They provide theoretical bounds on ranking errors for FAFI and use bootstrap aggregation to generate valid confidence intervals for principled strategy comparison.

Empirically, the best strategy on the 112-dataset time series benchmark achieved a Spearman rank correlation of 0.95 with the full ranking using only 5 datasets. For the 57-task NLP benchmark derived from MTEB, several strategies (including simple FAFI) significantly outperformed random selection. However, on a recommender systems benchmark (30 datasets), the improvement over random was slight and often statistically insignificant. The authors note that the effectiveness of selection approaches depends on both the quality of dataset representations and the scale of the benchmarking regime. This work offers a practical path to drastically reduce evaluation costs while maintaining high fidelity in model comparisons.