Gleam: New Framework Boosts GPU Sharing Across LANs by 24x
Turn your home network into a distributed GPU cluster with 1.79x higher throughput
Gleam addresses the network bottleneck in cross-device GPU sharing over local area networks by optimizing CUDA API remoting. The framework reduces bandwidth overhead through automatic model weight caching and mitigates latency from frequent API calls via asynchronous execution. It also includes a runtime task scheduler that dynamically selects optimal remoting pairs between clients and servers, accounting for both network conditions and GPU resource contention under parallel workloads. Dedicated mechanisms ensure CUDA context consistency across distributed executions.
In extensive tests on heterogeneous NVIDIA GPUs with diverse AI workloads, Gleam consistently outperformed state-of-the-art baselines, achieving 1.4x to 24.2x improvements in API remoting efficiency and up to 1.79x higher system throughput. This makes it possible to run compute-intensive AI inference tasks on otherwise underutilized personal devices like laptops and desktops connected via LAN, without requiring specialized hardware. The paper is available on arXiv with 20 pages and 28 figures.
- Gleam uses automatic model weight caching and asynchronous execution to reduce CUDA API remoting bandwidth overhead by up to 24x
- A runtime scheduler dynamically picks optimal server-client pairs based on network and GPU contention, boosting throughput 1.79x
- Works on heterogeneous NVIDIA GPUs and diverse AI workloads; code and data are linked from the arXiv paper
Why It Matters
Democratizes AI inference by turning ordinary LAN-connected devices into a unified, high-performance GPU pool.