Federated Inference for Heterogeneous LLM Communication and Collaboration

A team of researchers from the Singapore University of Technology and Design has introduced a novel framework called FedRefine, detailed in a position paper accepted for the AAAI 2026 Workshop on ML4Wireless. The core challenge they address is the performance and efficiency limitations of running large language models (LLMs) on individual devices. FedRefine proposes a new paradigm where multiple, potentially different LLMs (like GPT-4o and Claude 3.5) can collaborate to perform inference tasks without sharing their private training data or full model parameters.

Instead of exchanging sensitive raw data or entire model weights, the FedRefine framework facilitates collaboration through the communication of Key-Value (KV) caches. These caches are intermediate representations generated during the model's reasoning process. This method allows heterogeneous models to refine each other's outputs and enhance overall performance while adhering to strict privacy requirements and task-specific Quality of Service (QoS) constraints. The authors position this "LLM-native communication" as a way to fully exploit on-device inference capabilities, enabling more powerful and efficient AI applications on edge devices, from smartphones to IoT sensors, where computational resources are limited.