E2LLM cuts LLM serving latency 50% in edge/fog environments

Deploying large language models (LLMs) on edge and fog devices is notoriously difficult due to limited memory and compute. Most existing approaches assume a model can fit on a single device, but real-world edge networks consist of heterogeneous devices with varying capabilities. Researchers from the University of Oslo and other institutions propose E2LLM, a framework designed to efficiently serve LLMs in these constrained environments.

Rather than splitting a single model across all available devices, E2LLM replicates the full model across multiple groups of devices (replicas) and applies model parallelism within each replica. Each replica is assigned a specialized role — either PREFILL (optimized for processing input tokens) or DECODER (optimized for generating output tokens) — leveraging inherent differences between the two inference phases. To organize devices, E2LLM uses a genetic algorithm to form clusters that maximize performance, then applies dynamic programming to determine the optimal partitioning strategy within each cluster, minimizing bottlenecks. In experiments, E2LLM reduced average waiting time by over 50% under high-demand conditions compared to the Splitwise baseline, showing robust adaptation to varying workloads with significant differences in input and output token lengths.