Maestro cuts LLM multi-agent memory by 67% with intelligent scheduling

Large Language Model-based Multi-Agent Systems (LLM-MAS) are gaining traction for complex tasks, but their deployment at scale introduces severe resource challenges. Each user query spawns multiple iterative LLM calls, amplifying GPU memory and compute demands. In resource-constrained cloud settings, these workflows suffer from non-deterministic decode-stage costs, heavy-tailed multi-model requirements causing memory fragmentation, and cross-cluster scheduling trade-offs. Traditional schedulers like EDF fail to handle the unpredictable nature of agent interactions, leading to high KV-cache reservation and SLO violations.

Maestro, accepted at ICDCS 2026, tackles these issues with a workload-aware approach. It explicitly leverages agent semantics by predicting output length and memory usage for each stage. These predictions drive a three-level hierarchical scheduler: at the node level, dynamic multi-model co-location via hierarchical weight caching and elastic memory provisioning; at the cluster level, latency-aware routing to avoid cold starts; and globally, workflow-aware prioritization to minimize head-of-line blocking for interactive tasks. In trace-driven simulations and prototype experiments, Maestro cut KV-reservation HBM by 67.2% and improved high-contention SLO attainment by 23.6 percentage points over EDF, demonstrating significant efficiency gains for LLM-MAS deployments.