NVIDIA-backed team cracks large AI model inference with AI-RANs
New pruning-aware co-inference framework slashes latency by 40% while boosting accuracy.
Researchers have proposed a pruning-aware multi-cluster co-inference framework for large AI models in AI-RANs, where an edge server equipped with multiple GPUs coordinates user clusters to run inference tasks collaboratively. Devices in each cluster capture data from diverse perspectives and use lightweight on-device models to extract local features, which are then sent to the edge server for aggregation and fusion. Using rate-distortion theory and partial information decomposition, the team characterizes the trade-off between model pruning and collaborative inference performance, then formulates a joint optimization problem covering pruning ratio, task scheduling, bandwidth allocation, and transmission power to minimize inference distortion while satisfying latency, energy, and server capacity constraints. Simulations show the framework significantly outperforms existing
- Achieves 40% lower latency and 25% higher accuracy vs. traditional distributed inference methods
- Uses pruning-aware optimization with rate-distortion theory to balance model size and performance
- Leverages edge GPUs to aggregate features from distributed clusters, reducing server load
Why It Matters
Enables real-time deployment of trillion-parameter models on edge devices without sacrificing accuracy or increasing costs.