New agentic scheduler beats GPU-first policies for AI tools
Profiling 19 AI tools reveals only 11 prefer GPU—blanket scheduling is wasteful.
Researchers Tianxi Lu and Sherief Reda have published a paper on arXiv proposing an agentic CPU-GPU scheduler for heterogeneous AI workloads. They profiled 19 AI tools (e.g., inference, training, data processing) on both GPU and CPU, discovering that only 11 tools are truly GPU-preferred; 4 are ambiguous, 1 is CPU-preferred due to PCIe transfer overhead, and 3 are device-neutral. This data debunks the common practice of assigning all GPU-capable tools to the GPU by default, which can waste VRAM and cause contention.
To solve this, the authors built an agentic scheduler that combines an LLM agent with a runtime monitor. The monitor dynamically observes GPU utilization, VRAM pressure, and swap behavior, feeding the LLM running averages, reprobing hints, and exploration cues—without dictating mappings. The scheduler decides for each tool: immediate GPU, queued GPU, or CPU offload. In 13 scenarios covering serial, parallel, and memory-constrained execution, the agentic scheduler matched brute-force optimal mappings every time, outperforming classical heuristics (HEFT, StarPU) and the all-GPU policy while requiring zero offline training. This work offers a practical, adaptive approach for optimizing AI tool placement on shared GPU/CPU clusters.
- Profiled 19 AI tools: only 11 GPU-preferred, 1 CPU-preferred, rest ambiguous or neutral.
- Agentic scheduler pairs LLM with runtime monitor to dynamically assign tools to GPU immediate, queued GPU, or CPU offload.
- Achieved optimal mapping in all 13 test scenarios, outperforming HEFT, StarPU, and all-GPU policies with no offline training.
Why It Matters
Smarter GPU usage cuts costs and latency for multi-tool AI workloads on shared infrastructure.