STREAM unifies local, HPC, and cloud LLM inference with sub-second latency

STREAM (Smart Tiered Routing Engine for AI Models) addresses the fragmented LLM inference landscape by introducing a three-tier routing architecture that intelligently distributes queries across local, HPC, and cloud resources. The system uses a local LLM-based complexity judge to determine the appropriate tier for each request, optimizing for cost, privacy, and latency. Its key innovation is a dual-channel HPC streaming architecture: the Globus Compute control plane handles authentication and job dispatch, while a separate WebSocket relay data plane streams tokens. This design achieves a median time-to-first-token (TTFT) of just 0.54 seconds through institutional firewalls—a 21.1x improvement over batch mode's 11.40 seconds—without requiring VPN or firewall rule changes. End-to-end AES-256-GCM encryption ensures the relay operator cannot read token payloads.

Beyond streaming, STREAM introduces tier-aware context summarization to prevent long conversations from forcing simple queries onto expensive tiers, and an HPC-as-API proxy mode that makes HPC inference available as an OpenAI-compatible endpoint. In benchmarks with Llama 3.2 3B across 1,200 queries spanning ten domains, STREAM achieved 85.1% free-tier retention (local or HPC) with measured TTFT of 0.26s local, 0.54s HPC, and 1.68s cloud. This middleware effectively unifies the privacy and cost benefits of local/HPC resources with the ease-of-use of cloud APIs, making it a practical solution for research institutions and enterprises with sensitive data.