SymTorch: A Framework for Symbolic Distillation of Deep Neural Networks

Researchers Elizabeth S.Z. Tan, Adil Soubki, and Miles Cranmer have introduced SymTorch, a novel framework designed to automate the symbolic distillation of deep neural networks. This process replaces opaque neural components—or entire networks—with human-readable, closed-form mathematical expressions, a technique with proven value in scientific discovery but previously hampered by significant engineering overhead. SymTorch directly tackles the adoption barrier by wrapping neural modules, collecting their input-output behavior, and leveraging the symbolic regression library PySR to generate interpretable equations. It seamlessly manages the complex logistics of GPU-CPU data transfer, caching, and model serialization, allowing researchers to switch between neural and symbolic forward passes within their existing PyTorch workflows. The team validated the framework across diverse architectures including Graph Neural Networks (GNNs), Physics-Informed Neural Networks (PINNs), and transformers.

The most compelling application demonstrated is the potential for accelerating large language model (LLM) inference. In a proof-of-concept, the researchers replaced standard Multi-Layer Perceptron (MLP) layers within a transformer model with optimized symbolic surrogates. This substitution yielded an 8.3% improvement in inference throughput, albeit with a moderate trade-off in model performance. This breakthrough suggests a new pathway for model optimization that moves beyond traditional pruning or quantization. By distilling learned representations into efficient, interpretable code, SymTorch bridges the gap between black-box deep learning and transparent, equation-based reasoning. The framework's release could catalyze wider use of symbolic regression in AI, leading to more efficient, understandable, and physically-grounded models in scientific computing and beyond.