DEFRAG brings RAG to edge devices, slashing LLM costs by 98.4%

Cloud-hosted LLMs are powerful but expensive, suffer from vendor lock-in, and degrade under load. Researchers from UBC, Tsinghua, CUHK, and CityU (Jiaxing Li, Hengzhi Wang, Feng Wang, Chi Xu, Danyang Song, Ruixiao Zhang, Edith C.H. Ngai, Jiangchuan Liu) propose DEFRAG, a decentralized edge collaboration system for retrieval-augmented generation (RAG). DEFRAG addresses two key challenges: knowledge coverage and accuracy gaps seen in edge-deployed small language models (SLMs). For retrieval, it compresses and shares knowledge graphs across edge devices, enabling hybrid retrieval that broadens knowledge coverage without centralizing data. For generation, it uses an optimizer that adaptively selects the best SLM and RAG parameters per query, balancing accuracy and cost in real time.

In tests on a heterogeneous edge testbed with benchmark QA datasets, mobile route stress, non-uniform data placement, and domain-specific workloads, DEFRAG maintained stable service quality and cost efficiency. The results show it reduces cost by up to 98.4% and increases peak throughput by 97.8% compared to centralized services, while significantly narrowing the accuracy gap between SLMs and cloud-based LLMs. The paper (arXiv:2608.00922) demonstrates that decentralized edge collaboration could democratize LLM access, making high-quality AI feasible on resource-constrained devices without sacrificing performance.