Voltron lets edge devices team up to run larger LLMs with 16.5% accuracy boost

A team of researchers has introduced Voltron, a novel framework for running LLM inference across multiple edge devices. Traditional cloud-based inference suffers from high latency and privacy risks. Running LLMs on a single edge device limits model size and accuracy. Voltron solves this by elastically distributing inference across multiple user-owned devices (phones, tablets, laptops) in the edge environment.

Voltron adapts to diverse real-world conditions, dynamically allocating model layers or tensor partitions across available devices. In tests, Voltron achieved up to 16.5% higher accuracy than the best single-device LLM, while maintaining user QoS constraints. This enables privacy-preserving, low-latency AI services on the edge, unlocking use cases like on-device assistants, healthcare, and autonomous systems without relying on cloud servers.