Roomie: New AI system cuts GPU colocation SLO violations by 3x

As demand for deep neural network (DNN) inference skyrockets, GPU capacity is increasingly oversubscribed. Cloud and edge operators often colocate multiple models on the same device to maximize utilization, but this frequently causes SLO (service-level objective) violations. The problem? Existing serving systems either ignore or coarsely approximate interference using aggregate resource profiles that miss temporal kernel overlaps. Enter Roomie, a novel orchestration architecture from Youssouph Faye, Francescomaria Faticanti, Shubham Jain, and Francesco Bronzino (arXiv:2607.16784). Roomie decouples offline kernel profiling from online interference prediction. It extracts per-kernel resource configurations via profiling, then uses an occupancy-based analytical model to predict interference—immune to profiler-induced timing distortion. To handle multiple concurrent models, it employs a pairwise greedy heuristic that approximates multi-model interference in polynomial (not exponential) time. An online placement algorithm assigns each incoming model to the GPU that minimizes predicted slowdown.

Experiments across cloud-grade server clusters and embedded edge devices show Roomie reduces SLO violations (inference latency) by up to 3x compared to state-of-the-art solutions. It maintains comparable or superior goodput—the number of requests completed within SLO boundaries. This is a significant advance for anyone running large-scale inference serving, especially as model sizes and request volumes continue to grow. By focusing on kernel-level temporal dynamics rather than aggregate profiles, Roomie offers a practical path to higher GPU utilization without sacrificing performance. The paper is available on arXiv and is pending journal publication.