Entropy-regularized gates boost sparse model discovery in scarce-data FL
New method prevents early commitment to sparse support, improving test performance by up to 20%.
Deep Dive
Researchers propose entropy-regularized probabilistic gates for federated learning with scarce data and high dimensionality. The method samples from competing sparse configurations during training, avoiding early commitment to sparse support. Experiments show consistent improvements in test accuracy and sparsity recovery over federated iterative hard thresholding and pruning after dense FedAvg training, especially under data and participation heterogeneity.
Key Points
- Entropy regularization on gate distributions prevents early commitment to sparse subnets, enabling exploration of competing sparse configurations
- Outperforms Fed-IHT and pruned FedAvg by up to 15% in test accuracy under high data heterogeneity and partial client participation
- Works in the small-sample high-dimensional regime (d >> N) without requiring additional communication rounds or data sharing
Why It Matters
Enables efficient FL with sparse models when client data is scarce, reducing communication costs and improving generalization.