New paper reveals how to spot influential data points in high-dimensional ML models
Leave-one-out influence behaves differently when data has many dimensions—here's the math.
In classical statistics, a training sample's influence on a model is measured by leave-one-out—how the model changes when that point is removed. For low-dimensional settings (d fixed, n large), the properties of these influences are well understood. But in modern high-dimensional regimes where the number of features d scales with the number of samples n (n ≍ d), the picture gets messy: a point's influence depends on all other training points in complex ways.
Hugo Cui's new paper tackles this exact problem for convex M-estimation under Gaussian design. The key result: the distribution of leave-one-out influences across the training set converges to a limiting measure that can be sharply characterized. Moreover, the analysis shows that highly influential samples tend to lie close to the decision boundary—a result that formalizes a long-used heuristic in active learning, where points near the boundary are prioritized for labeling. The work provides rigorous theoretical foundations for data selection and model diagnostics in high dimensions.
- Proves convergence of influence distribution to a limiting measure when n ≍ d
- Shows influential points cluster near the decision boundary, validating active learning intuition
- Provides precise asymptotic formulas for leave-one-out influence under Gaussian design
Why It Matters
Gives practitioners a principled way to identify critical training points, improving data curation and model debugging.