New guard system ensures SLOs in ML serving even with bad learners

Modern ML serving systems increasingly rely on learned, unbounded components (routers, latency-SLO admitters, admit ladders) to manage quality of service. When these learners make mistakes, assured SLOs can silently break, and underlying Kubernetes layers only add cross-layer surprises. Researchers from the paper "Trusted Floors Under Untrusted Learners" propose a small, trusted guard that wraps the untrusted learner — the learner proposes, the guard disposes. The guard splits the tenant's assured-SLO obligation into two parts: a safety projection (per-request dispatch feasibility and per-class, per-window service floor) that the guard enforces at runtime via reservation and priority dispatch, and an aggregate obligation (tail-latency percentile) that remains statistical. The guard does not require the learner to be correct; it bounds the damage from any arbitrary mistake.

On real 2xV100 hardware, the guard achieved an assured-class miss rate of exactly 0.0 across 10 repetitions (worst upper Wilson CI 0.0053) under two different shed policies that fail in opposite directions of miscalibration. Against a deployed GAIE Flow Control on a serving simulator, a mislabeling router flipped the same assured requests from miss 0.0 to 1.0 without the guard — but the guard reserves by the true class, so the label cannot break its floor. The paper also characterizes when a cheap static screen can be trusted and when it cannot. The authors scope a future agenda including datacenter-scale testing, real-model Flow Control, and a closed worst-case theorem. This approach offers a practical, verifiable safety layer for production ML serving.