Clairvoyant AI scheduler cuts LLM response delays by 76%
New scheduler predicts query length in 0.029ms to end HOL blocking
Researcher Aravind Sundaresan has developed Clairvoyant, an open-source scheduler designed to combat Head-of-Line Blocking (HOLB) in serial LLM backends like Ollama. By using a 19-feature ONNX-exported XGBoost classifier, Clairvoyant predicts response lengths in just 0.029ms—four orders of magnitude faster than typical generation times. The system prioritizes short requests without needing exact predictions, achieving 62-96% accuracy in-distribution and 52-66% cross-distribution.
In GPU benchmarks on an RTX 4090, Clairvoyant slashed P50 latency for short requests by 70-76% under maximum queue pressure (100 concurrent requests) and 17% under steady-state Poisson arrivals (ρ=0.74). It’s a drop-in solution requiring no modifications to existing backends, making it ideal for memory-constrained edge or local deployments where continuous batching isn’t feasible.
- Clairvoyant uses an XGBoost model to predict LLM query lengths in 0.029ms using 19 lightweight features
- Reduces short-request latency by 70-76% under high load (100 concurrent requests) without backend changes
- Open-source sidecar proxy compatible with OpenAI-compatible backends like Ollama
Why It Matters
Solves real-world latency issues for local LLM deployments by intelligently prioritizing quick responses under load.