Demystifying When Pruning Works via Representation Hierarchies

A team from Northeastern University and MIT has published a pivotal paper titled 'Demystifying When Pruning Works via Representation Hierarchies' (arXiv:2603.24652). The research tackles a major puzzle in AI efficiency: why does network pruning—the process of removing less important parameters to shrink models—work brilliantly for some tasks but catastrophically fail for others? The authors discovered that the answer lies in the model's internal 'representation hierarchy,' which they decompose into three sequential spaces: embedding, logit, and probability.

Their key finding is that while the embedding and logit spaces remain robust to the perturbations caused by pruning, the final, nonlinear step that converts logits to probability distributions is highly sensitive. Small errors introduced by pruning are dramatically amplified in this probability space. During generative tasks like writing an essay, these amplified errors accumulate step-by-step (a process called autoregression), leading to a complete breakdown in output quality. In contrast, for non-generative tasks like multiple-choice QA or retrieval, the model only needs to make a single, stable prediction from the robust embedding space, allowing pruned models to retain near-original performance.

This work provides the first clear, mechanistic explanation for the inconsistent results seen in model compression. It offers practical guidance for developers: pruned models are reliable for classification and retrieval applications, enabling faster, cheaper deployments. However, for any task requiring multi-step generation, aggressive pruning is likely to fail. The paper, which includes 21 figures and detailed analysis, serves as an essential handbook for engineers optimizing LLMs like GPT-4o or Claude 3 for production, helping them choose the right efficiency technique for the right job.