New Saturation Index Tells You When to Stop Collecting Training Data

Arnav Gupta's new paper tackles a fundamental but undertheorized problem in applied machine learning: when to stop collecting labeled examples. The proposed saturation index S(K) measures the ratio of the effective rank of the pooled within-class sample covariance to the shot count. Gupta proves that when S(K) falls below a threshold, the covariance estimator is well-concentrated around the population covariance and the linear discriminant has stabilized. The index is computable in O(d^3) time from support features alone, requiring no test labels or trained classifier. Across 246 doubling-pair observations from 17 binary tasks and six datasets, the index showed a median within-task Spearman correlation with marginal accuracy gain of ρ = 0.811. The pooled Spearman correlation was ρ = 0.548 (p = 1.1 × 10⁻²⁰).

A three-phase diagram—exploration, transition, and saturation—emerged with mean marginal gains of 3.48%, 2.40%, and 0.82%, all pairwise significance tests passing at p ≤ 0.008. As a binary stopping rule, the index achieves AUC = 0.752, providing meaningful probabilistic guidance for annotation decisions. Notably, a small saturation index paired with low accuracy diagnoses representational inadequacy, helping practitioners identify when their model architecture or features are insufficient. Asymptotic effective rank and peak accuracy showed no significant monotone relationship across tasks (Spearman r_s = 0.380, p = 0.133). All results are for binary classification with a fixed linear classifier; extensions to N-way settings and pretrained backbone representations are discussed as future work. This work offers a practical, theoretically grounded tool to reduce labeling costs and diagnose model limitations early.