TallyTrain slashes federated learning bandwidth by 1000x

Federated learning faces two bandwidth bottlenecks: model size limits merge frequency, and class count makes soft-label distillation prohibitively expensive for large vocabularies. TallyTrain, developed by Radhakrishna Achanta and Will Reed, collapses the class-count axis by transmitting only each peer's argmax class index—just log2(C) bits per probe. Under non-IID training, this hard-label approach often outperforms soft-label distillation because majority voting filters confidently wrong predictions from under-trained peers, whereas soft-label averaging amplifies that noise. Across standard benchmarks, TallyTrain matches or beats soft-label distillation at up to 1000x less communication.

To also address the model-size bottleneck, the authors propose a bandwidth-bridge variant that composes cheap hard-label consensus with sparse parameter merges. This hybrid approach Pareto-dominates every tested operating point of FedAvg, FedProx, and FedDF. The method is especially promising for edge devices and large-scale deployment where communication overhead is the primary constraint. With 27 pages of analysis, 7 figures, and 12 tables, TallyTrain offers a practical pathway to scalable federated learning without sacrificing accuracy.