Neural Network Pruning via QUBO Optimization
A new method combines gradient-aware metrics and activation similarity to prune AI models more effectively than greedy heuristics.
A research team has introduced a novel Hybrid QUBO framework that reframes neural network pruning as a sophisticated combinatorial optimization problem. Traditional pruning methods often rely on greedy heuristics that fail to account for complex interactions between filters. This new approach uses a Quadratic Unconstrained Binary Optimization (QUBO) formulation but crucially enhances it by integrating gradient-aware sensitivity metrics—specifically first-order Taylor and second-order Fisher information—into the linear term of the objective. Simultaneously, it uses data-driven activation similarity in the quadratic term, allowing the model to jointly evaluate individual filter importance and inter-filter redundancy.
The framework employs a dynamic capacity-driven search to strictly enforce target sparsity levels without distorting the optimization landscape. To finalize the pruned network, the researchers implemented a two-stage pipeline featuring a Tensor-Train (TT) Refinement stage. This gradient-free optimizer fine-tunes the solution derived from the QUBO solver directly against the true evaluation metric, ensuring the compressed model retains high performance. Experiments conducted on the SIDD image denoising dataset demonstrated that this Hybrid QUBO method significantly outperforms both conventional greedy Taylor pruning and simpler L1-norm-based QUBO approaches, with the TT Refinement stage providing additional performance gains.
This work highlights a significant shift from heuristic-based compression to principled, optimization-driven methods. By successfully bridging heuristic importance estimation with global combinatorial optimization, the framework paves the way for more scalable, interpretable, and effective neural network compression techniques, which are critical for deploying large models on resource-constrained devices.
- Formulates pruning as a QUBO problem using gradient-aware metrics (Taylor/Fisher) and activation similarity for a holistic filter assessment.
- Uses a two-stage pipeline with a Tensor-Train Refinement stage to fine-tune the QUBO solution against the true evaluation metric.
- Outperformed greedy Taylor pruning and L1-based QUBO on the SIDD image denoising dataset, proving the hybrid method's effectiveness.
Why It Matters
Provides a more principled and effective method for compressing large AI models, enabling efficient deployment on edge devices with less performance loss.