AI Trimming Gets a Reality Check: New Test Shows How to Trust Smaller Models

Structured pruning relies on cheap surrogate objectives because directly evaluating every possible pruning mask is too expensive. Most evaluations only report average surrogate error or rank correlation, but PruneShift—a new evaluation framework—separates broad predictive fidelity, fidelity near selector outputs, and the quality of the final selected pruning decision. The authors prove that standard agreement measures like Spearman and Kendall can look nearly perfect while the actual selected decision is maximally bad, and they derive conditions that make a selection trustworthy. Four studies test the framework: results are mixed, with some tests favoring the surrogate-chosen mask, some favoring a fixed comparator, and many endpoints inconclusive. The takeaway is that predictive fit, decision reliability, and pruning method quality each