ATSInfer boosts local LLM inference 3x on consumer CPU-GPU devices
New tensor-level offloading system makes LLMs run efficiently on laptops and desktops.
Running large language models on consumer devices like laptops or desktops is limited by GPU memory, forcing model offloading to CPU memory. Existing systems rely on coarse layer- or expert-level scheduling, missing the heterogeneity among tensors within a layer and failing to adapt to changing hardware loads. ATSInfer, introduced by Liu et al., attacks this at tensor granularity, moving individual tensors between CPU and GPU as needed. It combines static tensor placement with load-aware dynamic transfers and introduces asynchronous CPU-GPU coordination to overlap data movement with computation.
Evaluated on representative consumer platforms with both dense models and mixture-of-experts architectures, ATSInfer delivers up to 1.94× higher prefill throughput and up to 3.29× higher decode throughput compared to prior systems. It also increases GPU utilization and makes more efficient use of PCIe bandwidth. The results suggest that fine-grained tensor scheduling can substantially improve the user experience of running LLMs locally on personal devices, making AI more accessible without requiring cloud servers or expensive hardware.
- Tensor-level offloading achieves 1.94× prefill and 3.29× decode throughput gains over layer-level scheduling.
- Combines static placement with load-aware dynamic transfers to adapt to fluctuating hardware loads.
- Asynchronous CPU-GPU coordination overlaps data movement with computation, improving PCIe bandwidth usage.
Why It Matters
Enables faster, more usable local LLM inference on laptops, reducing cloud dependency for AI workloads.