This Training-Free Pruning Method Boosts LLM Reasoning by 61% — And It's Not Wanda

A team of researchers (Sheth, Assefa, Huang, Lin, Ge) has introduced Causal Attribution Pruning (CAP), a training-free method that identifies critical attention heads by measuring their causal impact on reasoning tasks. Unlike magnitude-only or activation-based criteria, CAP’s interventional measurement directly captures each head’s functional contribution by estimating the expected performance degradation when the head is masked during forward passes on a small calibration set of reasoning problems. These causal scores are then converted into weight-level importance values for the corresponding projection matrices, enabling fine-grained weight pruning.

Evaluated on Llama-3-8B-Instruct and Mistral-7B-Instruct at 10%, 20%, and 50% sparsity across GSM8K, StrategyQA, and ARC-Challenge, CAP consistently outperforms the existing Wanda method. At 20% sparsity, CAP achieves relative accuracy gains of up to 61% over Wanda on ARC-Challenge for Llama-3. At moderate sparsity (10–20%), CAP improves over Wanda in most model-benchmark configurations, with especially large gains on ARC-Challenge.

However, at 50% sparsity, CAP is limited by coarse MLP attribution, indicating that while attention-head-level causal attribution preserves reasoning better than correlational criteria, more work is needed for high sparsity levels. The paper was accepted at the ICLR 2026 Workshop on LLM Reasoning and is available on arXiv.