Exploring the Limits of Pruning: Task-Specific Neurons, Model Collapse, and Recovery in Task-Specific Large Language Models

Researchers led by M.K. Khalidi Siam conducted a systematic pruning study on task-specific LLMs (1.5B and 7B parameters) fine-tuned for mathematical reasoning and code generation. They developed an activation-based selectivity metric to identify neurons with low contribution to the target task. Selective pruning consistently outperformed random pruning, confirming that activation-based selection provides a systematic advantage. Reverse pruning experiments—where highly task-specific neurons were removed first—showed that eliminating just ~10% of these critical neurons led to complete performance collapse, suggesting that essential task information is concentrated in a small fraction of the network. In contrast, pruning 30-35% of less critical neurons reduced accuracy but preserved significant functionality.

The study also measured practical benefits: parameter count and runtime VRAM usage decreased, and inference throughput improved as pruning increased. However, a robustness threshold emerged around 15-20% pruning, beyond which accuracy loss and generation failures rose sharply. Fine-tuning post-pruning proved highly effective, particularly for aggressively pruned models, almost fully recovering performance. These findings provide empirical evidence of neuron specialization in task-specific LLMs, offer clear pruning safety margins (15-20%), and demonstrate that fine-tuning can restore most lost capability. The work has direct implications for efficient model deployment, especially for specialized applications where computational resources are constrained.