New benchmark reveals RL schedulers fail under heavy edge cluster loads

A new paper from Zihang Wang and colleagues at TU Wien introduces a continuous adaptive benchmark for fair comparison of scheduling algorithms on heterogeneous edge clusters. The CMMS (Continuous Multi-Mode Scheduling) problem involves picking both a target node and processing mode (e.g., full vs. reduced inference precision) for each AI workload to satisfy multi-dimensional SLOs on latency, throughput, and output quality. Prior studies evaluated each controller in its own stack with a single workload, making comparisons unfair. This benchmark provides a unified controller interface, a closed-loop workload driver with multiple patterns, and dual-metric SLO scoring that separates raw compliance from steady-state compliance.

Across 424 episodes with six controllers, five cluster configurations, and two load regimes, rankings were strongly configuration-dependent. A deep reinforcement-learning controller that dominated under light workloads lost to a rule-based heuristic by nearly 29 percentage points as load intensified — while incurring roughly 500x the per-decision operational overhead. Separating raw from steady-state SLOs also exposed switching costs that a single aggregate score would hide. This work provides a much-needed tool for evaluating schedulers in real-world edge-cloud deployments where workloads and hardware vary dynamically.