LMEdge orchestrates LLM inference across edge clusters with QoS optimization
59K benchmark rows and 57-instance testbed prove it slashes latency while preserving accuracy
Deploying large language models (LLMs) on edge devices promises low-latency, privacy-preserving AI, but managing heterogeneous, resource-constrained hardware is tricky. Researchers from TU Wien and elsewhere introduce LMEdge, a QoS-aware orchestration service that dynamically selects model family, size, quantization level, and execution placement. They formulate the problem as binary integer linear programming (BILP) to minimize response time under accuracy, network, and resource constraints. To scale online scheduling, LMEdge uses five lightweight ML models that predict query-specific latency, accuracy, resource usage, and response size for each possible combination.
A comprehensive benchmarking dataset of over 59,000 rows trains these predictors. The team evaluated LMEdge on a Kubernetes-based edge testbed with 57 instances across diverse query categories. Results show that LMEdge reduces latency, preserves accuracy, improves resource utilization, and increases serving ratio compared to two baselines. The paper (arXiv:2607.17175) also includes 5 figures and 2 tables, and will be presented at EUROPAR. This approach bridges the gap between cloud-grade LLM performance and edge resource constraints.
- LMEdge uses BILP optimization to minimize response time while respecting accuracy, network, and resource constraints on edge devices.
- Five lightweight ML models predict per-query latency, accuracy, resource usage, and response size for each model-size-quantization-device combination.
- Evaluation on a 57-instance Kubernetes edge testbed shows reduced latency, higher serving ratio, and better resource utilization versus baselines.
Why It Matters
Enables low-latency, privacy-preserving LLM inference on resource-constrained edge devices, making AI more accessible and responsive.