SparseDitto's AI agents outperform cuSPARSE by 2.68x
LLM agents generate GPU kernels that beat NVIDIA's cuSPARSE by up to 146x...
A team led by Shiyang Li from Northeastern University and collaborators from Syracuse University, University of Minnesota, and other institutions has developed SparseDitto, an innovative system that leverages LLM-based agents to automatically generate optimal GPU kernels for sparse matrix computations.
The system addresses a critical pain point in high-performance computing: sparse matrix operations (like SpMV, SpMM, and SpGEMM) have wildly varying performance depending on sparsity patterns and hardware. For example, cuSPARSE—a leading sparse linear algebra library from NVIDIA—can exhibit a 350x performance gap for the same operation (SpMM) on different data formats (CSR vs. Blocked-ELL). SparseDitto eliminates this guesswork by using a multi-agent AI system where coding and verification agents iteratively refine kernels based on structural matrix features and real hardware feedback. The result? Across diverse matrices and operators, SparseDitto delivers a 2.68x average speedup over cuSPARSE on NVIDIA RTX PRO 6000 GPUs, with top-end gains of 146.61x. On the newer H200 GPU, the gains are 2.79x on average, peaking at 78.5x. Notably, the generated SpMM kernels accelerate full-batch Graph Convolutional Network (GCN) training by up to 3.39x, demonstrating real-world impact in machine learning workloads.
- SparseDitto uses LLM-based agents to generate and optimize GPU kernels for sparse matrix operations, achieving up to 146.61x speedup over NVIDIA cuSPARSE on RTX PRO 6000 GPUs
- Supports SpMV, SpMM, and SpGEMM with a unified framework that adapts to input sparsity patterns and target hardware via architecture-aware planning
- Generated kernels accelerate GCN training by 3.39x, showing practical benefits for ML workloads
Why It Matters
AI-generated GPU kernels could slash compute costs and unlock faster AI training by tailoring sparse operations to hardware and data patterns.