CausalGate prunes LLMs via causal importance distillation, boosting speed

Current adaptive inference methods for large language models rely on observational heuristics like hidden-state similarity or activation magnitudes to drop redundant modules. These correlation-based metrics often miss subtle, non-linear computations vital for semantic accuracy. CausalGate replaces these heuristics with an intervention-guided framework: during calibration, it isolates each Attention and MLP sub-layer, zeros out its output, and measures the exact semantic damage via Kullback-Leibler divergence of the final logit distribution. This structural importance hierarchy is then distilled into a global set of static, lightweight scalar gates using an Exponential Moving Average smoothing objective paired with a differentiable pairwise ranking loss—eliminating runtime routing overhead entirely.

Evaluated on TinyLlama-1.1B, Qwen2.5-3B, and Llama-3.1-8B across language modeling and commonsense reasoning benchmarks, CausalGate consistently outperformed prominent dynamic routing and layer-skipping baselines. It translates theoretical compute savings into concrete hardware latency reductions with zero operational overhead—no need for runtime decisions or adaptive controllers. This approach offers a practical path to faster, cheaper LLM inference without sacrificing output quality, making it highly relevant for production deployments.