OmniPilot: AI advisor picks best GPU config with 95% accuracy
New tool predicts LLM throughput within 6.2% error on mixed clusters.
OmniPilot, developed by researchers D. Balamurugan and Thomas W. Bush, tackles the complex problem of deploying large language models on shared heterogeneous GPU clusters (A100, H100, H200). Operators must choose GPU type, tensor-parallel degree, and precision under fluctuating throughput, failure rates, and demand. The system pairs a conformally calibrated quantile cost model covering eight serving targets with an out-of-distribution (OOD) abstention layer. It ranks configurations using an economic utility metric tuned to operator preferences, achieving 6.2% mean absolute percentage error in aggregate throughput predictions and a log-space R² of 0.92 across 460 benchmark runs.
In practice, OmniPilot achieves 95% top-1 accuracy with mean utility regret of just 0.003. When encountering unsupported configurations (OOD), prediction error jumps to 24–46%—but the abstention layer successfully flags all five low-confidence cases. These OOD scenarios are fed back into the training dataset to expand the advisor’s support envelope over time. This approach reduces wasted node-hours and improves launch-success rates by accounting for quantization effects, KV-cache pressure, and variable failure rates across parallelism degrees.
- Predicts aggregate throughput with 6.2% MAPE and 95% top-1 configuration accuracy across A100, H100, H200 hardware.
- Uses conformally calibrated quantile model + OOD abstention layer to avoid low-confidence recommendations (0.003 mean utility regret).
- Automatically expands support envelope by retraining on flagged out-of-distribution scenarios over time.
Why It Matters
Saves hours of trial-and-error on expensive GPU clusters by intelligently selecting configs with measurable confidence.