New Federated Protocol Matches Theoretical Limits with Optimal Bandwidth Allocation
First matching lower bound for bandwidth term in federated probe-logit distillation at Θ(K⁻¹·2⁻²ᴮ/ⱽ).
Researchers Prasanjit Dubey and Xiaoming Huo have published a paper that settles two open problems in federated probe-logit distillation (FPLD), a method for training language models across multiple nodes without sharing data or gradients. First, they prove a matching lower bound of Ω(K⁻¹·2⁻²ᴮ/ⱽ) for the bandwidth term in the minimax KL rate under non-degeneracy, confirming that the upper bound from prior work is tight along the bandwidth axis. They also show that T-round sequential refinement with nested/scaled residual quantizers achieves O(K⁻¹·2⁻²ᵀᴮ/ⱽ), making the vanilla single-round FPLD suboptimal for multi-round setups.
Second, they tackle the practical case of heterogeneous per-node bandwidth budgets B_i. The authors derive a closed-form optimal allocation formula B_i* = B_total/K + (V/2) log₂(w_i / ̄w_g) — a log-tilted water-filling rule analogous to reverse water-filling in rate-distortion theory. A plug-in adaptive variant estimates the necessary weights from a short warm-up phase and achieves 1 + O(√(log(K/δ)/(mT₀))) relative suboptimality. Synthetic n-gram simulations confirm that empirical KL divergences lie within the theoretical bounds and that the optimal allocation strictly outperforms uniform and inverse-weighted baselines under heterogeneous clipping.
- First matching lower bound of Ω(K⁻¹·2⁻²ᴮ/ⱽ) for bandwidth term in federated probe-logit distillation, proving prior upper bound is tight
- Optimal heterogeneous bandwidth allocation via log-tilted water-filling formula: B_i* = B_total/K + (V/2) log₂(w_i / ̄w_g)
- T-round refinement with residual quantizers achieves O(K⁻¹·2⁻²ᵀᴮ/ⱽ), showing vanilla FPLD is suboptimal for multi-round settings
Why It Matters
Enables provably optimal bandwidth usage in federated learning with heterogeneous edge devices, reducing communication costs by orders of magnitude.