New ML law reveals limits of statistical shift detection
A new scale law shows why traditional stats fail to detect subtle data drift
Adel Kaleche's paper introduces a mathematical scale law that fundamentally limits how well traditional statistical tests (mean, covariance, kurtosis) can detect subtle distribution shifts in high-dimensional embeddings. The law states that certifying changes at spatial scale ε with mass fraction f requires polynomial tests of degree N* ≥ log(1/f)/(2ε). Kaleche proves this via Chebyshev extremal problems and shows the bound is tight using Gaussian test functions and RBF kernel RKHS witnesses.
The research demonstrates a practical calibration rule: using an RBF kernel with bandwidth matching the feature scale achieves superior detection performance. In experiments across 26 real embedding streams, this kernel method reached AUC ≥ 0.95 while resisting adversarial optimization. Topological methods like persistent homology showed mixed results, with total persistence achieving 75% recall at 1% FPR but requiring 116x more computation than kurtosis-based methods.
- New scale law proves statistical tests need N* ≥ log(1/f)/(2ε) to detect subtle distribution shifts
- Kernel tests with RBF bandwidth matching feature scale achieve 95% AUC vs 75% for topological methods
- Topological approaches cost 116x more computational resources while offering comparable performance
Why It Matters
This breakthrough provides a mathematically grounded approach to detecting data drift, enabling more reliable AI systems in production environments.