Solyx AI Grid Boosts Multi-Site LLM Routing 1.75x with GPU Telemetry

As GPU capacity fragments across multiple datacenters, traditional single-cluster LLM inference routing assumptions fail to account for hardware heterogeneity, network variability, and real-time utilization. Solyx AI Grid, introduced by Aleks Bernhard and Nithin Katla, is a cross-site inference routing control plane that solves this by fusing GPU hardware telemetry (via NVIDIA DCGM), application-layer metrics from vLLM, and active WAN path signals (RTT, jitter) into a per-request placement decision engine. The system uses a 10-signal weighted pressure scorer to route each inference request to the most suitable GPU across sites, balancing throughput, latency SLOs, and hardware capability.

In two empirical campaigns—spanning six H100/H200 SXM GPUs and nine RTX PRO 6000 Blackwell SE GPUs across three US datacenters, eight workload classes, and a 216-cell SLO matrix—Solyx AI Grid delivered 1.56–1.75x throughput improvement at tier-2 SLO over round-robin routing. It reduced capability-mismatch leakage (where requests land on suboptimal hardware) to just 0.43% compared to 32% for standard routers. Failure rerouting was also dramatically faster: p99 of 1,247 ms versus 4,226 ms. Notably, GPU hardware telemetry predicted application-layer SLO breaches by an average of 11.2 seconds, enabling proactive traffic drain before users experience latency impact. This work represents the first public empirical study of live physical multi-site LLM inference routing that jointly leverages hardware telemetry, application metrics, and active WAN path signals.