New ProMoE-FL method handles missing data in multimodal federated learning
A prototype-conditioned mixture of experts synthesizes missing modalities on chest X-ray data...
ProMoE-FL addresses a critical challenge in multimodal federated learning: how to handle missing modalities at inference time without relying on a public dataset or naive feature synthesis. The framework creates a global prototype bank that aggregates modality-specific priors across federated clients, making the system client-aware. A mixture-of-experts (MoE) layer is conditioned on these prototypes and modality indices, enabling direction-aware routing to dynamically generate the missing modality features from available ones.
The method was rigorously evaluated on four widely used chest X-ray datasets (MIMIC-CXR, NIH Open-I, PadChest, CheXpert) spanning different institutions and distributions. ProMoE-FL consistently outperformed existing state-of-the-art methods in both homogeneous (similar data distributions) and the more realistic heterogeneous (diverse distributions) federated learning settings. This work has significant implications for medical AI, where it's common for different hospitals to have incomplete imaging data, preventing models from leveraging all available modalities.
- Uses a global prototype bank to capture modality priors across federated clients
- MoE conditioned on prototypes and modality indices enables direction-aware feature synthesis
- Outperforms SOTA on four chest X-ray datasets in both homogeneous and heterogeneous settings
Why It Matters
Enables robust multimodal AI in federated healthcare settings where imaging data is frequently incomplete.